v0.5.0

0.5.0 (2025-10-22)

Features

• Add 'Modifie' status and add 'links' column to DILA's datasets (4a46daa)
• add additional downloader classes for CNIL, Constit, Dole, and Legi (8b2cbcc)
• add base extractor classes and extraction functionality (8b2cbcc)
• add ConstitProcessor and enhance processing logic for constitutional files (eed7238)
• add daily data extraction workflow and enhance download script with date handling (eed7238)
• add daily data extraction workflow and enhance download script with date handling (3a509bd)
• Add data.gouv.fr's datasets catalog to the pipeline (24cb8fb)
• Add data.gouv.fr's datasets catalog to the pipeline (5c0d5dc)
• add DOLEProcessor and enhance extraction logic for DOLE files (eed7238)
• Add hash and doc id columns to datasets (#43) (4f9628e)
• add Legifrance schema, models, and table with CRUD operations and tests (8b2cbcc)
• add max_download parameter to download_all methods and update tests (8b2cbcc)
• add process records management with CRUD operations and tests (8b2cbcc)
• add SheetsBaseExtractor for handling sheet extraction and implement related tests (eed7238)
• added new datas (directories, CONSTIT and DOLE) and upgraded global code (9e61698)
• Airflow Implementation (#29) (03f8ea3)
• Ajout de la base de données des textes de la CNIL et amélioration des scripts (472e7da)
• Automated parquet files upload to the Hugging Face repository (1ab007d)
• Automated parquet files upload to the Hugging Face repository (0006482)
• automation of data pipeline, pgsql and config files backup and … (1725762)
• automation of data pipeline, pgsql and config files backup and adding short ci-cd (718227f)
• Better chunks embedding (#23) (8d0205c)
• DOLE data added (853e69e)
• DOLE data added (61ca73f)
• DOLE data added to the pipeline (25b63c6)
• enhance argument parsing for download and extraction scripts, adding user ID and date handling (eed7238)
• enhance argument parsing for download and extraction scripts, adding user ID and date handling (3a509bd)
• enhance BaseProcessor with process_all method for batch processing and add output directory creation (8b2cbcc)
• enhance data extraction and processing for local administrations directory (eed7238)
• enhance data extraction and processing for local administrations directory (3a509bd)
• enhance DilaDownloader and BaseExtractor with regex support and improve extraction logic (8b2cbcc)
• enhance Sheets processing with improved file handling and XML parsing (eed7238)
• enhancement of data names and content (bf79fbd)
• Exported parquet files now splitted by doc_id (feabb01)
• implement DirectoryDownloader with URL fetching and file management (8b2cbcc)
• implement extraction and processing for CNIL and Legiarti datasets (eed7238)
• implement extraction and processing for CNIL and Legiarti datasets (3a509bd)
• implement extraction records management with CRUD operations and tests (8b2cbcc)
• implement SheetsDownloader and TravailDownloader for handling sheet downloads (eed7238)
• implement SheetsProcessor and add process_sheets utility for handling sheet data (eed7238)
• Logging to Tchap (cb5b14e)
• optimize parquet compression for better storage efficiency during export (d2584ae)
• processed folders will now be removed automatically after processing (e87dbfb)
• Sheets and Directories are downloaded only if there is an available updated version (a080d49)
• Split the LEGI table in 'Code' types and categories (b85b23d)
• Split the LEGI table in 'Code' types and categories (655a368)
• Starting chunk_index count from 1 instead of 0 (608efcf)
• supporting service_public and travail_emploi datas (0ffdceb)
• supporting service_public and travail_emploi datas (86bbfb0)
• Take into account deletes updates from DILA folders data (897e7d5)
• Take into account deletes updates from DILA folders data (1835211)
• update download name to 'service_public_part' and enhance error handling for unsupported types (eed7238)
• update download name to 'service_public_part' and enhance error handling for unsupported types (3a509bd)
• upload splitted parquet files (#38) (d6ee797)

Bug Fixes

• .env variables CD (790f505)
• Adding 'text' column to CONSTIT and CNIL tables (e4a2aab)
• Adding env variables in CD (d11cb7f)
• Adding mediatech's folder in container permission (6f7aba8)
• airflow folder (093a2d9)
• better cd ([58221d7](5822...

v0.4.0

0.4.0 (2025-10-20)

Features

• add daily data extraction workflow and enhance download script with date handling (dfe5f0d)
• enhance argument parsing for download and extraction scripts, adding user ID and date handling (dfe5f0d)
• enhance data extraction and processing for local administrations directory (dfe5f0d)
• implement extraction and processing for CNIL and Legiarti datasets (dfe5f0d)
• update download name to 'service_public_part' and enhance error handling for unsupported types (dfe5f0d)

Bug Fixes

• update default end date to current date in argument parser (dfe5f0d)

v0.3.0

0.3.0 (2025-10-19)

Features

• Add 'Modifie' status and add 'links' column to DILA's datasets (4a46daa)
• add additional downloader classes for CNIL, Constit, Dole, and Legi (8b2cbcc)
• add base extractor classes and extraction functionality (8b2cbcc)
• Add data.gouv.fr's datasets catalog to the pipeline (24cb8fb)
• Add data.gouv.fr's datasets catalog to the pipeline (5c0d5dc)
• Add hash and doc id columns to datasets (#43) (4f9628e)
• add Legifrance schema, models, and table with CRUD operations and tests (8b2cbcc)
• add max_download parameter to download_all methods and update tests (8b2cbcc)
• add process records management with CRUD operations and tests (8b2cbcc)
• added new datas (directories, CONSTIT and DOLE) and upgraded global code (9e61698)
• Airflow Implementation (#29) (03f8ea3)
• Ajout de la base de données des textes de la CNIL et amélioration des scripts (472e7da)
• Automated parquet files upload to the Hugging Face repository (1ab007d)
• Automated parquet files upload to the Hugging Face repository (0006482)
• automation of data pipeline, pgsql and config files backup and … (1725762)
• automation of data pipeline, pgsql and config files backup and adding short ci-cd (718227f)
• Better chunks embedding (#23) (8d0205c)
• DOLE data added (853e69e)
• DOLE data added (61ca73f)
• DOLE data added to the pipeline (25b63c6)
• enhance BaseProcessor with process_all method for batch processing and add output directory creation (8b2cbcc)
• enhance DilaDownloader and BaseExtractor with regex support and improve extraction logic (8b2cbcc)
• enhancement of data names and content (bf79fbd)
• Exported parquet files now splitted by doc_id (feabb01)
• implement DirectoryDownloader with URL fetching and file management (8b2cbcc)
• implement extraction records management with CRUD operations and tests (8b2cbcc)
• Logging to Tchap (cb5b14e)
• optimize parquet compression for better storage efficiency during export (d2584ae)
• processed folders will now be removed automatically after processing (e87dbfb)
• Sheets and Directories are downloaded only if there is an available updated version (a080d49)
• Split the LEGI table in 'Code' types and categories (b85b23d)
• Split the LEGI table in 'Code' types and categories (655a368)
• Starting chunk_index count from 1 instead of 0 (608efcf)
• supporting service_public and travail_emploi datas (0ffdceb)
• supporting service_public and travail_emploi datas (86bbfb0)
• Take into account deletes updates from DILA folders data (897e7d5)
• Take into account deletes updates from DILA folders data (1835211)
• upload splitted parquet files (#38) (d6ee797)

Bug Fixes

• .env variables CD (790f505)
• Adding 'text' column to CONSTIT and CNIL tables (e4a2aab)
• Adding env variables in CD (d11cb7f)
• Adding mediatech's folder in container permission (6f7aba8)
• airflow folder (093a2d9)
• better cd (58221d7)
• better cd (b2df4cf)
• Better DOLE's data processing for chunk_number and text column (07e4252)
• Better HF files storage management (2990ea7)
• CD and lockfile mismatch (6512b32)
• Container deployment (0ad2c84)
• dataset's name error for upload (5fb0054)
• directories will no longer be deleted just after beeing downloaded (14b952f)
• Dockerimage build (cb140cc)
• Empty data_history.json is now accepted (4f968a5)
• ensure downloaded files are unpacked and removed only if they exist (8b2cbcc)
• Exporting env variables (e5ccdb5)
• HF upload date may have not be saved to log data history file after upload (65c8df6)
• JWT_TOKEN automatically generated (e677b1a)
• Loading env variables (01d5b93)
• Lock file creation (db64ed2)
• Log data history file not updated in real time (e37f83d)
• Make scripts executable and fixing cd (9fd73c3)
• Missing newline in cron config file (264b616)
• Open file via with (cdd2db1)
• Parquet file is deleted even if the HF dataset is already up to date (1962f95)
• Raise error when there is one ([201aa11](https://github.com/mitch...

v0.2.0

0.2.0 (2025-10-19)

Features

• Add 'Modifie' status and add 'links' column to DILA's datasets (4a46daa)
• add additional downloader classes for CNIL, Constit, Dole, and Legi (13b9203)
• add base extractor classes and extraction functionality (13b9203)
• Add data.gouv.fr's datasets catalog to the pipeline (24cb8fb)
• Add data.gouv.fr's datasets catalog to the pipeline (5c0d5dc)
• Add hash and doc id columns to datasets (#43) (4f9628e)
• add Legifrance schema, models, and table with CRUD operations and tests (023f462)
• add max_download parameter to download_all methods and update tests (023f462)
• add process records management with CRUD operations and tests (023f462)
• added new datas (directories, CONSTIT and DOLE) and upgraded global code (9e61698)
• Airflow Implementation (#29) (03f8ea3)
• Ajout de la base de données des textes de la CNIL et amélioration des scripts (472e7da)
• Automated parquet files upload to the Hugging Face repository (1ab007d)
• Automated parquet files upload to the Hugging Face repository (0006482)
• automation of data pipeline, pgsql and config files backup and … (1725762)
• automation of data pipeline, pgsql and config files backup and adding short ci-cd (718227f)
• Better chunks embedding (#23) (8d0205c)
• DOLE data added (853e69e)
• DOLE data added (61ca73f)
• DOLE data added to the pipeline (25b63c6)
• enhance BaseProcessor with process_all method for batch processing and add output directory creation (13b9203)
• enhance DilaDownloader and BaseExtractor with regex support and improve extraction logic (13b9203)
• enhancement of data names and content (bf79fbd)
• Exported parquet files now splitted by doc_id (feabb01)
• implement DirectoryDownloader with URL fetching and file management (13b9203)
• implement extraction records management with CRUD operations and tests (023f462)
• Logging to Tchap (cb5b14e)
• optimize parquet compression for better storage efficiency during export (d2584ae)
• processed folders will now be removed automatically after processing (e87dbfb)
• Sheets and Directories are downloaded only if there is an available updated version (a080d49)
• Split the LEGI table in 'Code' types and categories (b85b23d)
• Split the LEGI table in 'Code' types and categories (655a368)
• Starting chunk_index count from 1 instead of 0 (608efcf)
• supporting service_public and travail_emploi datas (0ffdceb)
• supporting service_public and travail_emploi datas (86bbfb0)
• Take into account deletes updates from DILA folders data (897e7d5)
• Take into account deletes updates from DILA folders data (1835211)
• upload splitted parquet files (#38) (d6ee797)

Bug Fixes

• .env variables CD (790f505)
• Adding 'text' column to CONSTIT and CNIL tables (e4a2aab)
• Adding env variables in CD (d11cb7f)
• Adding mediatech's folder in container permission (6f7aba8)
• airflow folder (093a2d9)
• better cd (58221d7)
• better cd (b2df4cf)
• Better DOLE's data processing for chunk_number and text column (07e4252)
• Better HF files storage management (2990ea7)
• CD and lockfile mismatch (6512b32)
• Container deployment (0ad2c84)
• dataset's name error for upload (5fb0054)
• directories will no longer be deleted just after beeing downloaded (14b952f)
• Dockerimage build (cb140cc)
• Empty data_history.json is now accepted (4f968a5)
• ensure downloaded files are unpacked and removed only if they exist (023f462)
• Exporting env variables (e5ccdb5)
• HF upload date may have not be saved to log data history file after upload (65c8df6)
• JWT_TOKEN automatically generated (e677b1a)
• Loading env variables (01d5b93)
• Lock file creation (db64ed2)
• Log data history file not updated in real time (e37f83d)
• Make scripts executable and fixing cd (9fd73c3)
• Missing newline in cron config file (264b616)
• Open file via with (cdd2db1)
• Parquet file is deleted even if the HF dataset is already up to date (1962f95)
• Raise error when there is one ([201aa11](https://github.com/mitch...