Repository navigation
v0.2.0
Pre-release
Pre-release
0.2.0 (2025-10-19)
Features
- Add 'Modifie' status and add 'links' column to DILA's datasets (4a46daa)
- add additional downloader classes for CNIL, Constit, Dole, and Legi (13b9203)
- add base extractor classes and extraction functionality (13b9203)
- Add data.gouv.fr's datasets catalog to the pipeline (24cb8fb)
- Add data.gouv.fr's datasets catalog to the pipeline (5c0d5dc)
- Add hash and doc id columns to datasets (#43) (4f9628e)
- add Legifrance schema, models, and table with CRUD operations and tests (023f462)
- add max_download parameter to download_all methods and update tests (023f462)
- add process records management with CRUD operations and tests (023f462)
- added new datas (directories, CONSTIT and DOLE) and upgraded global code (9e61698)
- Airflow Implementation (#29) (03f8ea3)
- Ajout de la base de données des textes de la CNIL et amélioration des scripts (472e7da)
- Automated parquet files upload to the Hugging Face repository (1ab007d)
- Automated parquet files upload to the Hugging Face repository (0006482)
- automation of data pipeline, pgsql and config files backup and … (1725762)
- automation of data pipeline, pgsql and config files backup and adding short ci-cd (718227f)
- Better chunks embedding (#23) (8d0205c)
- DOLE data added (853e69e)
- DOLE data added (61ca73f)
- DOLE data added to the pipeline (25b63c6)
- enhance BaseProcessor with process_all method for batch processing and add output directory creation (13b9203)
- enhance DilaDownloader and BaseExtractor with regex support and improve extraction logic (13b9203)
- enhancement of data names and content (bf79fbd)
- Exported parquet files now splitted by doc_id (feabb01)
- implement DirectoryDownloader with URL fetching and file management (13b9203)
- implement extraction records management with CRUD operations and tests (023f462)
- Logging to Tchap (cb5b14e)
- optimize parquet compression for better storage efficiency during export (d2584ae)
- processed folders will now be removed automatically after processing (e87dbfb)
- Sheets and Directories are downloaded only if there is an available updated version (a080d49)
- Split the LEGI table in 'Code' types and categories (b85b23d)
- Split the LEGI table in 'Code' types and categories (655a368)
- Starting chunk_index count from 1 instead of 0 (608efcf)
- supporting service_public and travail_emploi datas (0ffdceb)
- supporting service_public and travail_emploi datas (86bbfb0)
- Take into account deletes updates from DILA folders data (897e7d5)
- Take into account deletes updates from DILA folders data (1835211)
- upload splitted parquet files (#38) (d6ee797)
Bug Fixes
- .env variables CD (790f505)
- Adding 'text' column to CONSTIT and CNIL tables (e4a2aab)
- Adding env variables in CD (d11cb7f)
- Adding mediatech's folder in container permission (6f7aba8)
- airflow folder (093a2d9)
- better cd (58221d7)
- better cd (b2df4cf)
- Better DOLE's data processing for chunk_number and text column (07e4252)
- Better HF files storage management (2990ea7)
- CD and lockfile mismatch (6512b32)
- Container deployment (0ad2c84)
- dataset's name error for upload (5fb0054)
- directories will no longer be deleted just after beeing downloaded (14b952f)
- Dockerimage build (cb140cc)
- Empty data_history.json is now accepted (4f968a5)
- ensure downloaded files are unpacked and removed only if they exist (023f462)
- Exporting env variables (e5ccdb5)
- HF upload date may have not be saved to log data history file after upload (65c8df6)
- JWT_TOKEN automatically generated (e677b1a)
- Loading env variables (01d5b93)
- Lock file creation (db64ed2)
- Log data history file not updated in real time (e37f83d)
- Make scripts executable and fixing cd (9fd73c3)
- Missing newline in cron config file (264b616)
- Open file via with (cdd2db1)
- Parquet file is deleted even if the HF dataset is already up to date (1962f95)
- Raise error when there is one (201aa11)
- Reading the environment variables first (bdba41e)
- Removing hours from logs names (scripts) (6c1a402)
- Restore script's env variables (68bee33)
- Splitting CD steps (0430d11)
- update CI/CD workflow to use ubuntu-24.04 runner (d4f9b00)
- update max_download parameter in test for directory downloader (023f462)
- Upgrading to docker-compose-v2 (d451d10)
- wrong datas format for the Postgres database (696f745)
- wrong deployment_packages.sh path in cd (814c715)
- wrong format for legi's 'numero' column' (0dd5910)
- Wrong lock file path in CD (edb1f5c)
- wrong python name in cd (dd39d0d)
- wrong python name in cd (da8be70)
- Wrong SCRIPT_DIR (13dd0b0)
- Wrong SCRIPT_DIR (bad49b3)
- Wrong virtual env folder name (baa27e7)