Repository navigation
Releases
Releases · lbulgarelli/datakiln
datakiln 0.3.0.5
Full Changelog: v0.3.0.4...v0.3.0.5
datakiln 0.3.0.4
Full Changelog: v0.3.0.3...v0.3.0.4
datakiln 0.3.0.3
Full Changelog: v0.3.0.2...v0.3.0.3
datakiln 0.3.0.2
datakiln build now downloads a data file that is missing and that DVC imported from a URL (dvc import-url), so a pipeline no longer needs a separate dvc update.
- If the repository has a default DVC remote, a missing file is pulled from it, and an error there stops the build, as before.
- Otherwise the file is downloaded again from the URL it was imported from, unless the file at the URL changed since the import. In that case the build stops with an error and the DVC file is not touched, because the tables are built with the checksum recorded in it.
- A file that is already there is never fetched.
Install: pip install datakiln
datakiln 0.3.0.1
A small follow-up to 0.3.0. The package itself does not change; this release improves how it is presented and tested.
- The PyPI page now has keywords, classifiers and links to the source, the documentation, the issue tracker and the releases.
- The tests run on every supported Python version (3.10 to 3.13) in CI, and the runners are pinned to Ubuntu 24.04.
- The README describes the
DB_ISOLATION_LEVELsetting and the--paralleloption. - Added
CONTRIBUTING.md,CODE_OF_CONDUCT.md,SECURITY.md, and templates for issues and pull requests. - The publish workflow compares the tag with the version as Python normalizes it.
Install: pip install datakiln
v0.3.0
The package is now called datakiln, and it has one command.
Install
pip install https://github.com/lbulgarelli/datakiln/releases/download/v0.3.0/datakiln-0.3.0-py3-none-any.whl
or from the tag:
pip install git+https://github.com/lbulgarelli/datakiln.git@v0.3.0
Breaking changes
- The package was
data-utilsand its moduledata_utils. Both aredatakilnnow: install it as shown below, importdatakiln, and run the tests of a dataset withpytest --pyargs datakiln.test. - The commands
build_database,build_metadata,build_dbms,build_redcapandshow_graphare replaced bydatakiln build,datakiln metadata,datakiln extract dbms,datakiln extract redcapanddatakiln graph.build_tableis removed:datakiln build --include <table>builds one table with what it needs. - The extractions take the same options:
--data/--no-dataand--metadata/--no-metadata. REDCap's--jsonis now--metadata. - The tables of a dataset are ordered by what they reference, so a two-digit prefix in the name of a metadata file is no longer left out when its data file is looked up.
- Python 3.10 or newer is required, as the package now declares.
Changes since v0.2.0
- Keep local files out of the source archive
- bump version to 0.3.0
- Describe the datakiln command in the README
- Run everything from one datakiln command
- Remove the build_table command
- Stop reading the order of tables from file names
- Declare the Python version the package needs
- Stop printing the REDCap validation types
- Remove the GitLab pipeline
- Add the badges and write the name in lowercase
- Build the documentation and publish it
- Run the tests and the lint on GitHub
- List the files of a dataset in order
- Rename the package to datakiln
- Describe what the package is for and how to use it
Files
datakiln-0.3.0-py3-none-any.whl:a1c252fe4e602884f5c30895b32537feabf26e2e3a448a48e1737323bbbfae02datakiln-0.3.0.tar.gz:8a64ae07b19f2fa7bc36d2d0488fc7d8d7ba70cdd46550493705435fc70f570e
v0.2.0
Changes since v0.1.89:
- Draw the graph with the ELK layout
- Leave out the arrows that other arrows imply
- Build the dependency graph on networkx
- Cast the columns of a foreign key to their own types
- Read the text that holds code as SQL, or by its tokens
- Read the queries that a script gives as text
- Draw the Mermaid arrows as curves
- Draw the graph only with Mermaid
- Track the calls of functions that sqlglot knows
- Support CREATE AGGREGATE in derive scripts
- Write the graph as a page or a document
- Show the graph of the tables and scripts
- bump version to 0.2.0
- Mark the checksum of a script as not for security
- Grant read only on the schemas of the repo
- Pull the data of a table only when it is loaded
- Load the data files of a folder that is not a DVC repository
- Name the check of a table that changed needs_update
- Rebuild the tables too with --rebuild-all
- Rebuild the tables that reference a rebuilt table
- Delete the files the extraction leaves behind
- Build only the selected tables and scripts with --include
- Forget the scripts that use what was rebuilt
- Link the nodes of the dependency graph to each other
- Write every log message as Kind 'name': Sentence.
- Drop the objects removed from the repo with --prune
- Run only the derive scripts that changed
- Record tables and scripts as objects of one kind
- Give scripts a checksum and the statement that drops them
- Do not fail loading a table with no rows to move
- ... and 9 more commits
v0.1.89
Changes since v0.1.88:
- ignore public schema when granting permissions
- Skip sqlparse formatting for large extraction queries using execution options
- precompile regex patterns to improve chunk processing speed
- shorten index names to prevent exceeding 63 chars
- support value-based filters without reference tables
- Centralize logging setup and add --stdout option across all CLI scripts
- add docstrings to oracleclob functions
- limit sqlalchemy version to avoid import error
- fix io linting issues
- bump version to 0.1.89
- Fix hatchling wheel build configuration
- Implement nested retry structure to handle transient database errors during streaming exports
- Refactor logging to use module-level loggers and output to file
- Add abort event to stop threads gracefully on SIGINT
- Use separate connection for batch inserts
- Unify chunk generation into a single OR-bounded query to prevent database-side duplication
- Refactor TempTableStrategy to use Python streaming and simultaneous CSV generation
- Implement SIGINT/SIGTERM handlers and fast-fail orchestration
- Add robust scope validation for FilterConfig joins
- Refactor QueryDirector for strategy-agnostic rendering and fix TempTableStrategy chunking
- allow specifying chunk size in extraction metadata
- fix ORA-01460: use custom _OracleCLOB type to bind JSON seed list as DB_TYPE_CLOB
- fix oracle float compilation error from pandas to_sql
- add --data/--no-data flag to build_dbms to optionally skip data extraction
v0.1.88
Changes since v0.1.87:
- pass only strategy to FilterConfig.to_exists
- fix comparing drop/create object in new sqlglot version
- Suppress DUO130 lint warnings on non-cryptographic md5 usages
- remove unused import
- Use strategy.cohorts to resolve join-chain temp tables and include them as task dependencies
- thread task metadata through cohort set retrieval to build join chains
- improve handling/creation of temporary files
- add comments to clarify logic
- pull cohort data from correct connection when using the temp table strategy
- give preference to temp files
- Refactor DBMS extraction pipeline helpers, database connection event listeners, and untrack .vscode folder
- update tests with new constraint functions
- ignore some linting
- rename variable to avoid overriding python function
- improve comment on statement handling
- build table with updated constraint functions
- bump version to 0.1.88
- remove null characters before loading file
- update constraint test
- add parameter to skip validation
- update log messages
- change order of constraint enforcement
- increase default batch size to 1mm
- enhance .sql validation and allow for multiple create index
- resolve linting issue in test
- specify aliases when joining fk
- create indexes in tests
- move batch_size to function parameter
- invert type casting for foreign key conditions to enable index use
- insert to final schema in batches
- ... and 6 more commits
v0.1.87
Changes since v0.1.86:
- bump pipeline to 0.1.4
- update url in readme files
- remove unused argument
- bump version to 0.1.87
- fix tests to work with recursive dependencies
- retrieve metadata from views as well
- compare table name to lowercase file name
- fix cohort duplication issue