Releases v0.5.0
Compare
Sorry, something went wrong.
No results found
0.5.0 (2025-10-22)Features
Add 'Modifie' status and add 'links' column to DILA's datasets (4a46daa )
add additional downloader classes for CNIL, Constit, Dole, and Legi (8b2cbcc )
add base extractor classes and extraction functionality (8b2cbcc )
add ConstitProcessor and enhance processing logic for constitutional files (eed7238 )
add daily data extraction workflow and enhance download script with date handling (eed7238 )
add daily data extraction workflow and enhance download script with date handling (3a509bd )
Add data.gouv.fr's datasets catalog to the pipeline (24cb8fb )
Add data.gouv.fr's datasets catalog to the pipeline (5c0d5dc )
add DOLEProcessor and enhance extraction logic for DOLE files (eed7238 )
Add hash and doc id columns to datasets (#43 ) (4f9628e )
add Legifrance schema, models, and table with CRUD operations and tests (8b2cbcc )
add max_download parameter to download_all methods and update tests (8b2cbcc )
add process records management with CRUD operations and tests (8b2cbcc )
add SheetsBaseExtractor for handling sheet extraction and implement related tests (eed7238 )
added new datas (directories, CONSTIT and DOLE) and upgraded global code (9e61698 )
Airflow Implementation (#29 ) (03f8ea3 )
Ajout de la base de données des textes de la CNIL et amélioration des scripts (472e7da )
Automated parquet files upload to the Hugging Face repository (1ab007d )
Automated parquet files upload to the Hugging Face repository (0006482 )
automation of data pipeline, pgsql and config files backup and … (1725762 )
automation of data pipeline, pgsql and config files backup and adding short ci-cd (718227f )
Better chunks embedding (#23 ) (8d0205c )
DOLE data added (853e69e )
DOLE data added (61ca73f )
DOLE data added to the pipeline (25b63c6 )
enhance argument parsing for download and extraction scripts, adding user ID and date handling (eed7238 )
enhance argument parsing for download and extraction scripts, adding user ID and date handling (3a509bd )
enhance BaseProcessor with process_all method for batch processing and add output directory creation (8b2cbcc )
enhance data extraction and processing for local administrations directory (eed7238 )
enhance data extraction and processing for local administrations directory (3a509bd )
enhance DilaDownloader and BaseExtractor with regex support and improve extraction logic (8b2cbcc )
enhance Sheets processing with improved file handling and XML parsing (eed7238 )
enhancement of data names and content (bf79fbd )
Exported parquet files now splitted by doc_id (feabb01 )
implement DirectoryDownloader with URL fetching and file management (8b2cbcc )
implement extraction and processing for CNIL and Legiarti datasets (eed7238 )
implement extraction and processing for CNIL and Legiarti datasets (3a509bd )
implement extraction records management with CRUD operations and tests (8b2cbcc )
implement SheetsDownloader and TravailDownloader for handling sheet downloads (eed7238 )
implement SheetsProcessor and add process_sheets utility for handling sheet data (eed7238 )
Logging to Tchap (cb5b14e )
optimize parquet compression for better storage efficiency during export (d2584ae )
processed folders will now be removed automatically after processing (e87dbfb )
Sheets and Directories are downloaded only if there is an available updated version (a080d49 )
Split the LEGI table in 'Code' types and categories (b85b23d )
Split the LEGI table in 'Code' types and categories (655a368 )
Starting chunk_index count from 1 instead of 0 (608efcf )
supporting service_public and travail_emploi datas (0ffdceb )
supporting service_public and travail_emploi datas (86bbfb0 )
Take into account deletes updates from DILA folders data (897e7d5 )
Take into account deletes updates from DILA folders data (1835211 )
update download name to 'service_public_part' and enhance error handling for unsupported types (eed7238 )
update download name to 'service_public_part' and enhance error handling for unsupported types (3a509bd )
upload splitted parquet files (#38 ) (d6ee797 )
Bug Fixes
.env variables CD (790f505 )
Adding 'text' column to CONSTIT and CNIL tables (e4a2aab )
Adding env variables in CD (d11cb7f )
Adding mediatech's folder in container permission (6f7aba8 )
airflow folder (093a2d9 )
better cd (58221d7 )
better cd (b2df4cf )
Better DOLE's data processing for chunk_number and text column (07e4252 )
Better HF files storage management (2990ea7 )
CD and lockfile mismatch (6512b32 )
Container deployment (0ad2c84 )
dataset's name error for upload (5fb0054 )
directories will no longer be deleted just after beeing downloaded (14b952f )
Dockerimage build (cb140cc )
Empty data_history.json is now accepted (4f968a5 )
ensure downloaded files are unpacked and removed only if they exist (8b2cbcc )
Exporting env variables (e5ccdb5 )
HF upload date may have not be saved to log data history file after upload (65c8df6 )
JWT_TOKEN automatically generated (e677b1a )
Loading env variables (01d5b93 )
Lock file creation (db64ed2 )
Log data history file not updated in real time (e37f83d )
Make scripts executable and fixing cd (9fd73c3 )
Missing newline in cron config file (264b616 )
Open file via with (cdd2db1 )
Parquet file is deleted even if the HF dataset is already up to date (1962f95 )
Raise error when there is one (201aa11 )
Reading the environment variables first (bdba41e )
Removing hours from logs names (scripts) (6c1a402 )
Restore script's env variables (68bee33 )
Splitting CD steps (0430d11 )
update default end date to current date in argument parser (eed7238 )
update default end date to current date in argument parser (3a509bd )
update max_download parameter in test for directory downloader (8b2cbcc )
Upgrading to docker-compose-v2 (d451d10 )
wrong datas format for the Postgres database (696f745 )
wrong deployment_packages.sh path in cd (814c715 )
wrong format for legi's 'numero' column' (0dd5910 )
Wrong lock file path in CD (edb1f5c )
wrong python name in cd (dd39d0d )
wrong python name in cd (da8be70 )
Wrong SCRIPT_DIR (13dd0b0 )
Wrong SCRIPT_DIR (bad49b3 )
Wrong virtual env folder name (baa27e7 )
Documentation
You can’t perform that action at this time.
