v0.1.14: Fortran Times in the CDL

What's Changed

• dev => main by @ialarmedalien in #346
• Add property-based tests to the spec tests by @ialarmedalien in #347
• Add preprocessors to the pipeline infra by @ialarmedalien in #348
• Adding ncbi_datasets pipeline with filtering by @ialarmedalien in #349
• Converters reorganisation by @ialarmedalien in #350
• Iceberg ahead! by @ialarmedalien in #351
• Converting input_dir to accept s3 directories as well as local dirs by @ialarmedalien in #352
• Fix unnecessary pyarrow => jsonl conversion if output format is jsonl by @ialarmedalien in #353
• No cruft by @ialarmedalien in #355
• Refactor lots of duplicated code in the converters hierarchy by @ialarmedalien in #356
• Refactor a load more duplicated code by @ialarmedalien in #358
• Spec-tacular new code rules, errors, and pipeline stuff by @ialarmedalien in #359
• Implement CTS-safe file globs by @ialarmedalien in #360
• Zensical by @ialarmedalien in #361
• Dev => main by @ialarmedalien in #362

Full Changelog: v0.1.13...v0.1.14

v0.1.13: lucky for all cdm-data-loaders fans!

What's Changed

• Workflow run tests by @ialarmedalien in #291
• Move uniprot/uniref pipelines to uniprot_cdm/uniref_cdm namespace by @ialarmedalien in #292
• New xsd/pyarrow XML data loader by @ialarmedalien in #293
• v0.1.12: new uniprot / uniref XML pipelines by @ialarmedalien in #294
• Develop => main for v0.1.12 by @ialarmedalien in #295
• Refactor: remove lots of repeated rebar by @ialarmedalien in #296
• First drafts of various schema - AllTheBacteria, GAF, GFF, GTDB, NCBI by @ialarmedalien in #300
• Corrections to JSONSchema to pyarrow converter, plus addition of test… by @ialarmedalien in #304
• Refactor super slow tests by @ialarmedalien in #305
• Add in JSONSchema to pyarrow data conversion. by @ialarmedalien in #306
• Bump github/codeql-action/upload-sarif from ae098fa9fe6541a837c2271de7f9718bcfd06aa0 to 3306ba4df796b6b5059316f3af8bdea1e2138c9b in the actions group across 1 directory by @dependabot[bot] in #298
• Report fixes by @ialarmedalien in #307
• Include filenames by @ialarmedalien in #308
• Bump tornado from 6.5.10 to 6.6 in the uv group across 1 directory by @dependabot[bot] in #309
• More deduplication by @ialarmedalien in #313
• More refactoring by @ialarmedalien in #314
• Even more junk dna removal by @ialarmedalien in #315
• table names and pipelines by @ialarmedalien in #316
• First step for making input dir compatible with s3 and other remote fs by @ialarmedalien in #317
• end brief flirtation with xml2db by @ialarmedalien in #318
• Fix up potential issues caused by running code in parallel threads by @ialarmedalien in #330
• Adding in transformer capabilities to generic pipelines by @ialarmedalien in #331
• Splitting up uniref pipeline into 3 easy pieces by @ialarmedalien in #332
• Alter docker tests to run in docker build pipeline. by @ialarmedalien in #333
• Fix segfault in duckdb by @ialarmedalien in #335
• Minor typing changes by @ialarmedalien in #336
• First draft of pydantic => pyarrow converter by @ialarmedalien in #337
• Remove unneeded linter comments. Update AGENTS.md and pyproject.toml. by @ialarmedalien in #338
• Bump the actions group across 1 directory with 4 updates by @dependabot[bot] in #334
• Moving docs tests to a separate action so they do not affect real tests by @ialarmedalien in #341
• Bump github/codeql-action/upload-sarif from 4f900f43bc6bfc406aac5f37e402badfc55f9342 to 0911ee96d2d6c8def97122c36c5cadd93829a4b4 in the actions group across 1 directory by @dependabot[bot] in #340
• Updating module deps by @ialarmedalien in #342
• Xml level parser by @ialarmedalien in #343
• Moving specs to top level directory by @ialarmedalien in #344
• Version bump by @ialarmedalien in #345

Full Changelog: v0.1.11...v0.1.13

v0.1.12: XSD to pyarrow, here we come!

What's Changed

• Workflow run tests by @ialarmedalien in #291
• Move uniprot/uniref pipelines to uniprot_cdm/uniref_cdm namespace by @ialarmedalien in #292
• New xsd/pyarrow XML data loader by @ialarmedalien in #293
• v0.1.12: new uniprot / uniref XML pipelines by @ialarmedalien in #294
• Develop => main for v0.1.12 by @ialarmedalien in #295

Full Changelog: v0.1.11...v0.1.12

Brave new importers

What's Changed

• Develop => main by @ialarmedalien in #194
• Rename functions/variables to remove references to delta lake format by @ialarmedalien in #196
• Allow bucket-only ls with s3 CLI by @mattldawson in #195
• Renaming spark_delta => spark by @ialarmedalien in #197
• Use berdl_notebook_utils on spark_notebook image by @ialarmedalien in #199
• Refactor NCBI FTP Promote into CLI Tool by @mattldawson in #201
• Update CEPH health check by @mattldawson in #202
• Add xsv-validate.sh to Dockerfile by @mattldawson in #203
• Moving the settings objects to cdm_data_loaders.core by @ialarmedalien in #204
• Add xSV validation module by @mattldawson in #205
• Add PDB manifest pipeline by @mattldawson in #206
• file transfer / download / upload reorg, part I by @ialarmedalien in #208
• Move utils.s3 to utils.file_transfer.s3.* by @ialarmedalien in #209
• Move the CLI commands to a separate module by @ialarmedalien in #211
• Add progress module to make tqdm-enhanced functions a little nicer by @ialarmedalien in #212
• Checksums module by @ialarmedalien in #213
• Adding transfer config by @ialarmedalien in #214
• Adding in missing tests and an interface to upload_fileobj by @ialarmedalien in #215
• Adding JSON schema for various different file types by @ialarmedalien in #216
• Minor reorg: by @ialarmedalien in #217
• Adding new xsv validator mode plus support, with tests. by @ialarmedalien in #220
• adding fn to generate a first pass schema if not available by @ialarmedalien in #221
• Adding JSONschema to pyspark generator by @ialarmedalien in #222
• Adding schema dereferencer by @ialarmedalien in #223
• Adding script to filter ncbi updates by date and IDs of interest by @ialarmedalien in #224
• rclone instructions by @mattldawson in #225
• Moving XML reader functions into their own module and adding in an xm… by @ialarmedalien in #226
• Adding tests for xml head parser by @ialarmedalien in #227
• Move ftp client to file transfer utils directory by @ialarmedalien in #228
• Updating pipelines that use XML parsers to reflect the new code by @ialarmedalien in #229
• Editing standard settings model config to autopopulate the dlt_config by @ialarmedalien in #230
• Adding in alias checks and trimming fat from pipelines. by @ialarmedalien in #231
• make input/output names symmetrical by @ialarmedalien in #232
• Adding CLI shortcuts by @ialarmedalien in #233
• Updating interface for qsv validation to include custom metaschema by @ialarmedalien in #234
• splitting out tests into unit and integration/e2e by @ialarmedalien in #235
• Extract e2e by @ialarmedalien in #236
• Move functions out of run_cli into run_pipeline. Fix up tests so that… by @ialarmedalien in #237
• fix caller cli arg leakage into cli arg parsing tests by @ialarmedalien in #238
• switching class-based tests to function-based by @ialarmedalien in #239
• Adding generic JSONLines and XML to Dict pipelines by @ialarmedalien in #240
• switch out tests that mock dlt with real lightweight dlt pipeline tests by @ialarmedalien in #241
• Minor file reorg by @ialarmedalien in #242
• Adding arrow buffers for saving structured data by @ialarmedalien in #243
• Misc minor edits and fixes by @ialarmedalien in #244
• Adding more thorough end to end tests for the xml pipeline by @ialarmedalien in #245
• Tests for xmltodict pipeline against reference dataset by @ialarmedalien in #246
• Adding in XSD capabilities by @ialarmedalien in #247
• Adding pyiceberg to jsonschema client for even more data parsing fun! by @ialarmedalien in #248
• Removing unneeded fixtures and adding new function to the DictBuffer API by @ialarmedalien in #249
• Adding a final pipeline step to save LoadInfo into a JSONL db/file by @ialarmedalien in #250
• Minor config file changes by @ialarmedalien in #252
• Fix logger init errors, reorganize files under the converters tree by @ialarmedalien in #256
• Converter cleanup by @ialarmedalien in #255
• Refactor/converters cleanup by @ialarmedalien in #259
• Remove old / superseded code by @ialarmedalien in #260
• Adding grouped emitter for jsonschema output by @ialarmedalien in #261
• Adding a JSONSchema to pyarrow converter by @ialarmedalien in #262
• Update dependabot.yml by @ialarmedalien in #265
• Pipelines again by @ialarmedalien in #267
• Minor formatting updates by @ialarmedalien in #268
• Bump the actions group across 1 directory with 5 updates by @dependabot[bot] in #269
• Integrating integration tests into pipeline dir by @ialarmedalien in #270
• Move dlt config, edit entry point by @ialarmedalien in #271
• Logger updates by @ialarmedalien in #272
• Bump github/codeql-action/upload-sarif from f0489abddd4e5e9dff53ed28a45b1d6f88978a1b to ae098fa9fe6541a837c2271de7f9718bcfd06aa0 in the actions group across 1 directory by @dependabot[bot] in #273
• Housekeeping by @ialarmedalien in #276
• Convert the xsv import pipeline to using the filesystem resource by @ialarmedalien in #277
• Pipeline cli by @ialarmedalien in #278
• Field reorg by @ialarmedalien in #279
• Remove destination by @ialarmedalien in #281
• update dockerfile and entry point by @ialarmedalien in #275
• Adding extra codecov coverage uploads by @ialarmedalien in #282
• Removing commented-out code by @ialarmedalien in #283
• Adding pipeline report file to output by @ialarmedalien in #284
• Bump version from 0.1.10 to 0.1.11 by @ialarmedalien in #286
• remove infernal spark-based annoyances by @ialarmedalien in #287
• develop => main by @ialarmedalien in #285

Full Changelog: v0.1.10...v0.1.11

v0.1.10

What's Changed

• BER to KBase Lakehouse by @ialarmedalien in #190
• Switch to name logger; rename/moved file batcher by @ialarmedalien in #189
• updating slack messaging to be a bit more circumspect by @ialarmedalien in #191
• Add in root logger configuration by @ialarmedalien in #192
• Version bump to v0.1.10 by @ialarmedalien in #193

Full Changelog: v0.1.9...v0.1.10

v0.1.9

What's Changed

• Adding kernel creation instructions by @ialarmedalien in #143
• add in to(schema) to ensure schema comments are saved by @ialarmedalien in #144
• Add AI Covenant checklist item to PR template by @mattldawson in #155
• Update s3 module to remove dependency on berdl utils and refs to minio by @ialarmedalien in #158
• NCBI FTP Integration by @mattldawson in #133
• Update Jupyter Instructions by @mattldawson in #169
• Add qsv to docker image for xSV linting/validation/etc. by @ialarmedalien in #172
• Replace MinIO with CEPH for testing by @mattldawson in #171
• Use xml_file_splitter binary instead of docker container by @ialarmedalien in #174
• Add decompression of gz file util by @ialarmedalien in #178
• Update CEPH-backed integration tests by @mattldawson in #177
• NCBI pipeline updates by @ialarmedalien in #183
• NCBI FTP Pipeline Clean-Up by @mattldawson in #184
• Dependency updates for 06-2026 by @ialarmedalien in #164
• Develop => main by @ialarmedalien in #186

New Contributors

• @mattldawson made their first contribution in #155

Full Changelog: v0.1.8...v0.1.9

v0.1.8

What's Changed

• dev to main by @ialarmedalien in #131
• Remove old cassettes, minor refactor of pipeline, add extra tests by @ialarmedalien in #132
• File download pipeline for AllTheBacteria by @ialarmedalien in #134
• Add in settings dump, CLI interface, and bump version to 0.1.7 by @ialarmedalien in #135
• Develop => main for 0.1.7 by @ialarmedalien in #136
• ATB Uploader improvements by @ialarmedalien in #138
• Bump lxml from 6.0.4 to 6.1.0 in the uv group across 1 directory by @dependabot[bot] in #137
• add file handler logging ext by @ialarmedalien in #139
• Set Trivy scan timeout to 15 minutes by @ialarmedalien in #140
• Version bump and a couple of minor fixes by @ialarmedalien in #141
• Develop => main by @ialarmedalien in #142

Full Changelog: v0.1.6...v0.1.8

v0.1.6

What's Changed

• Adding in a bit more error handling for pipelines and slack reporting where available by @ialarmedalien in #125
• reenable s3 tests by @ialarmedalien in #127
• Add more error handling for NCBI API responses by @ialarmedalien in #128
• Dev to main by @ialarmedalien in #129
• version bump to 0.1.6 by @ialarmedalien in #130

Full Changelog: v0.1.5...v0.1.6

v0.1.5

What's Changed

• Bump requests from 2.32.5 to 2.33.0 in the uv group across 1 directory by @dependabot[bot] in #107
• Bump ruff from 0.15.4 to 0.15.8 by @dependabot[bot] in #113
• Adding batch size param to ncbi api by @ialarmedalien in #122
• Version bump by @ialarmedalien in #123

Full Changelog: v0.1.4...v0.1.5

v0.1.4

What's Changed

• Main => develop by @ialarmedalien in #112
• Updating UniProt parser to use pydantic interface and consolidating s… by @ialarmedalien in #118
• Updates to core.py to allow more flexible supply of args to the pipeline by @ialarmedalien in #119
• Update dependencies and bump version by @ialarmedalien in #120
• Dev to main by @ialarmedalien in #121

Full Changelog: v0.1.3...v0.1.4