Brave new importers

What's Changed

• Develop => main by @ialarmedalien in #194
• Rename functions/variables to remove references to delta lake format by @ialarmedalien in #196
• Allow bucket-only ls with s3 CLI by @mattldawson in #195
• Renaming spark_delta => spark by @ialarmedalien in #197
• Use berdl_notebook_utils on spark_notebook image by @ialarmedalien in #199
• Refactor NCBI FTP Promote into CLI Tool by @mattldawson in #201
• Update CEPH health check by @mattldawson in #202
• Add xsv-validate.sh to Dockerfile by @mattldawson in #203
• Moving the settings objects to cdm_data_loaders.core by @ialarmedalien in #204
• Add xSV validation module by @mattldawson in #205
• Add PDB manifest pipeline by @mattldawson in #206
• file transfer / download / upload reorg, part I by @ialarmedalien in #208
• Move utils.s3 to utils.file_transfer.s3.* by @ialarmedalien in #209
• Move the CLI commands to a separate module by @ialarmedalien in #211
• Add progress module to make tqdm-enhanced functions a little nicer by @ialarmedalien in #212
• Checksums module by @ialarmedalien in #213
• Adding transfer config by @ialarmedalien in #214
• Adding in missing tests and an interface to upload_fileobj by @ialarmedalien in #215
• Adding JSON schema for various different file types by @ialarmedalien in #216
• Minor reorg: by @ialarmedalien in #217
• Adding new xsv validator mode plus support, with tests. by @ialarmedalien in #220
• adding fn to generate a first pass schema if not available by @ialarmedalien in #221
• Adding JSONschema to pyspark generator by @ialarmedalien in #222
• Adding schema dereferencer by @ialarmedalien in #223
• Adding script to filter ncbi updates by date and IDs of interest by @ialarmedalien in #224
• rclone instructions by @mattldawson in #225
• Moving XML reader functions into their own module and adding in an xm… by @ialarmedalien in #226
• Adding tests for xml head parser by @ialarmedalien in #227
• Move ftp client to file transfer utils directory by @ialarmedalien in #228
• Updating pipelines that use XML parsers to reflect the new code by @ialarmedalien in #229
• Editing standard settings model config to autopopulate the dlt_config by @ialarmedalien in #230
• Adding in alias checks and trimming fat from pipelines. by @ialarmedalien in #231
• make input/output names symmetrical by @ialarmedalien in #232
• Adding CLI shortcuts by @ialarmedalien in #233
• Updating interface for qsv validation to include custom metaschema by @ialarmedalien in #234
• splitting out tests into unit and integration/e2e by @ialarmedalien in #235
• Extract e2e by @ialarmedalien in #236
• Move functions out of run_cli into run_pipeline. Fix up tests so that… by @ialarmedalien in #237
• fix caller cli arg leakage into cli arg parsing tests by @ialarmedalien in #238
• switching class-based tests to function-based by @ialarmedalien in #239
• Adding generic JSONLines and XML to Dict pipelines by @ialarmedalien in #240
• switch out tests that mock dlt with real lightweight dlt pipeline tests by @ialarmedalien in #241
• Minor file reorg by @ialarmedalien in #242
• Adding arrow buffers for saving structured data by @ialarmedalien in #243
• Misc minor edits and fixes by @ialarmedalien in #244
• Adding more thorough end to end tests for the xml pipeline by @ialarmedalien in #245
• Tests for xmltodict pipeline against reference dataset by @ialarmedalien in #246
• Adding in XSD capabilities by @ialarmedalien in #247
• Adding pyiceberg to jsonschema client for even more data parsing fun! by @ialarmedalien in #248
• Removing unneeded fixtures and adding new function to the DictBuffer API by @ialarmedalien in #249
• Adding a final pipeline step to save LoadInfo into a JSONL db/file by @ialarmedalien in #250
• Minor config file changes by @ialarmedalien in #252
• Fix logger init errors, reorganize files under the converters tree by @ialarmedalien in #256
• Converter cleanup by @ialarmedalien in #255
• Refactor/converters cleanup by @ialarmedalien in #259
• Remove old / superseded code by @ialarmedalien in #260
• Adding grouped emitter for jsonschema output by @ialarmedalien in #261
• Adding a JSONSchema to pyarrow converter by @ialarmedalien in #262
• Update dependabot.yml by @ialarmedalien in #265
• Pipelines again by @ialarmedalien in #267
• Minor formatting updates by @ialarmedalien in #268
• Bump the actions group across 1 directory with 5 updates by @dependabot[bot] in #269
• Integrating integration tests into pipeline dir by @ialarmedalien in #270
• Move dlt config, edit entry point by @ialarmedalien in #271
• Logger updates by @ialarmedalien in #272
• Bump github/codeql-action/upload-sarif from f0489abddd4e5e9dff53ed28a45b1d6f88978a1b to ae098fa9fe6541a837c2271de7f9718bcfd06aa0 in the actions group across 1 directory by @dependabot[bot] in #273
• Housekeeping by @ialarmedalien in #276
• Convert the xsv import pipeline to using the filesystem resource by @ialarmedalien in #277
• Pipeline cli by @ialarmedalien in #278
• Field reorg by @ialarmedalien in #279
• Remove destination by @ialarmedalien in #281
• update dockerfile and entry point by @ialarmedalien in #275
• Adding extra codecov coverage uploads by @ialarmedalien in #282
• Removing commented-out code by @ialarmedalien in #283
• Adding pipeline report file to output by @ialarmedalien in #284
• Bump version from 0.1.10 to 0.1.11 by @ialarmedalien in #286
• remove infernal spark-based annoyances by @ialarmedalien in #287
• develop => main by @ialarmedalien in #285

Full Changelog: v0.1.10...v0.1.11