v0.9.1

What's Changed

• Filesystem: List and read through every registered scheme, including r2://,
  oss://, hdfs://, smb://, ftp://, dbfs://, oci:// and webhdfs://,
  by resolving a file's modification date from the listing the filesystem client
  returns.

Details

• fix(deps): update dependency ty to v0.0.66 by @renovate[bot] in #275
• fix(filesystem): list files through every registered scheme by @hampsterx in #274

Full Changelog: v0.9.0...v0.9.1

v0.9.0

What's Changed

• Filesystem: Enabled loading multiple workbook worksheets as tables.
  Thanks, @hampsterx.
• Filesystem: Enabled loading remote SQLite and DuckDB databases.
  Thanks, @hampsterx.
• Filesystem: Refreshed S3 directory listings between repeated loads so
  file-level incremental runs detect newly uploaded objects.
• Connectors: Added Azure Storage destination support for connection-string
  authentication and custom endpoints.
• Connectors: Reject mixed Azure Storage credential modes instead of silently
  ignoring extra fields in source and destination URIs.
• Filesystem: Fail ingestion when a concrete source path matches no file while
  keeping unmatched glob selections valid. Thanks, @hampsterx.
• Filesystem: Added rsync source connector. Thanks, @oferchen.

Details

• fix(deps): update dependency twine to v7 by @renovate[bot] in #257
• test(filesystem): update HDFS mock for fsspec by @hampsterx in #266
• fix(deps): update dependency ty to v0.0.65 by @renovate[bot] in #264
• fix(deps): update dependency mysql-connector-python to v26 by @renovate[bot] in #263
• fix(filesystem): fail missing concrete sources by @hampsterx in #265
• Add rsync source connector by @amotl in #269
• feat(filesystem): load workbook worksheets as tables by @hampsterx in #271
• test(filesystem): expand remote emulator integration coverage by @hampsterx in #273
• feat(sql): load remote SQLite and DuckDB databases by @hampsterx in #272

Full Changelog: v0.8.0...v0.9.0

v0.8.0

What's Changed

• Core: Added real SCD2 loading when selecting the scd2 strategy.
  Thanks, @hampsterx.
• Filesystem: Added file-level incremental loading.
  Thanks, @hampsterx.
• Filesystem: Migrated local filesystem access to Apache Arrow.
• Filesystem: Added support for reading from Databricks, Dropbox,
  FTP, Google Drive, HDFS, OCI, OneDrive, OSS, R2, SharePoint,
  SMB, WebDAV, and WebHDFS.
• Core: Applied a few bits of internal package refactoring.

Details

• Project: Migrate to src layout by @amotl in #227
• fix(deps): update dependency ty to v0.0.60 by @renovate[bot] in #228
• Refactoring: Initialize dlt_filesystem package inline by @amotl in #229
• Refactoring: Consolidate general filesystem information by @amotl in #230
• fix(incremental): perform a real SCD2 load for the scd2 strategy by @hampsterx in #235
• fix(deps): update dependency ty to v0.0.61 by @renovate[bot] in #237
• Filesystem: Migrate local filesystem access to Apache Arrow by @amotl in #238
• feat(filesystem): add file-level incremental loading by @hampsterx in #239
• fix(filesystem): always remove temporary pipeline state by @hampsterx in #243
• chore(deps): update astral-sh/setup-uv action to v9 by @renovate[bot] in #245
• fix(deps): update dependency ty to v0.0.62 by @renovate[bot] in #246
• Documentation: Fix broken links to Google Ads by @amotl in #248
• fix(deps): update dependency ty to v0.0.63 by @renovate[bot] in #250
• fix(deps): update dependency testcontainers to >=4.9.1,<4.16 by @renovate[bot] in #252
• Sandbox: Update to ruff 0.16.0 with old default configuration by @amotl in #253
• Filesystem integration tests by @amotl in #249
• Dependencies: Fix security warnings by updating flagged packages by @amotl in #255
• Filesystem: Add read support for more fsspec-based filesystems by @amotl in #254

Full Changelog: v0.7.0...v0.8.0

v0.7.0

What's Changed

• Connectors: Added Azure Blob Storage (az://) and Azure Data Lake Storage
  Gen2 (adls://, abfss://) source and destination adapters, backed by
  adlfs. Supports account-key, SAS-token, and service-principal auth.
  Thanks, @hampsterx.
• Connectors: Fixed remote filesystem destinations (Azure Blob, GCS, S3)
  aborting at the end of a load because the required post_load hook was
  missing from the shared base class. Thanks, @hampsterx.
• Filesystem: Added #key=value reader hint channel to source URIs
  to propagate options to source implementations. Thanks, @hampsterx.
• Filesystem: Added readers for CBOR, MessagePack, XML and YAML formats.
  Thanks, @hampsterx.
• Filesystem: Added readers for Excel workbook (XLSX) and OpenDocument
  spreadsheet (ODS) formats.
• Filesystem: Fixed and documented semantics of append/replace strategy
  and append/full-refresh behaviour. Thanks, @hampsterx.
• Dependencies: Updated to dlt 1.29

Details

• feat(filesystem): add Azure Blob Storage + ADLS Gen2 adapters by @hampsterx in #182
• feat(filesystem): add #key=value reader-hint channel to source URIs by @hampsterx in #183
• Chore: Fix refactoring leftover about reader hints on AzureSource by @amotl in #185
• chore(deps): update astral-sh/setup-uv action to v8.3.1 by @renovate[bot] in #184
• chore(deps): update astral-sh/setup-uv action to v8.3.2 by @renovate[bot] in #189
• fix(deps): update dependency ty to v0.0.57 by @renovate[bot] in #190
• fix(deps): update dependency ty to v0.0.58 by @renovate[bot] in #193
• fix(deps): update dependency pyarrow to v25 by @renovate[bot] in #194
• docs: improve core and target docstrings by @hampsterx in #197
• feat(filesystem): add MessagePack and CBOR read support by @hampsterx in #196
• Tests: Refactor test cases to use generic db helpers by @amotl in #198
• docs(incremental): document filesystem append/full-refresh behaviour by @hampsterx in #199
• fix(deps): update dependency ty to v0.0.59 by @renovate[bot] in #203
• feat(filesystem): honour explicit append/replace strategy by @hampsterx in #204
• Filesystem: Add reader for Excel XLSX files by @amotl in #123
• Filesystem: Propagate reader hints to Polars readers, with type casting by @amotl in #216
• Filesystem: Harmonize CSV reading with Excel reading by @coderabbitai[bot] in #217
• fix(deps): update dependency dlt to <1.30 by @renovate[bot] in #207
• fix(deps): update dependency typer to <0.28 by @renovate[bot] in #221
• Documentation: Improve filesystem / file format overview by @amotl in #225
• Filesystem: Add reader for OpenDocument spreadsheet (ODS) files by @amotl in #222
• feat(filesystem): add XML and YAML read support by @hampsterx in #205

New Contributors

• @coderabbitai[bot] made their first contribution in #217

Full Changelog: v0.6.0...v0.7.0

v0.6.0

What's Changed

• Connectors: Added file:// source and destination adapters that
  read and write from/to local CSV / JSONL / Parquet files.
  Thanks, @hampsterx.
• Connectors: Refactored filesystem internals
• Connectors: Migrated filesystem CSV reader from pandas to Polars
• Filesystem readers: Added support for BSON. Thanks, @hampsterx.

Details

• feat(source): add file:// local filesystem source by @hampsterx in #135
• CI: Stop collapsing walkthrough comment by @amotl in #155
• Documentation: s/RabbitMQ/AMQP/ by @amotl in #154
• Tests: Refactor and bundle stream tests, deduplicate code by @amotl in #158
• connector(filesystem): Refactor and bundle blobstorage, sftp, and utils by @amotl in #156
• Linter: Configure stricter settings for Ruff, and format code by @amotl in #164
• Connectors: Migrate filesystem CSV reader from pandas to Polars by @amotl in #165
• chore(deps): update astral-sh/setup-uv action to v8.3.0 by @renovate[bot] in #167
• CodeRabbit: Toggle review_status and poem by @amotl in #170
• CodeRabbit: Use label to designate PR reviews by @amotl in #171
• connector(filesystem): Refactor resource_for_reader args into data class by @amotl in #169
• feat(target): add file:// local filesystem destination by @hampsterx in #166
• Tests: Make table arguments optional in invoke_ingest_command by @hampsterx in #172
• connector(filesystem): Refactoring and naming things by @amotl in #173
• feat(filesystem): add BSON read support by @hampsterx in #174
• Documentation: Improve supported pipeline elements by @amotl in #175
• docs(linkcheck): harden against transient external-link flakes by @hampsterx in #176

Full Changelog: v0.5.0...v0.6.0

v0.5.0

What's Changed

• Connectors: Updated to Asana client v5
• Core: Started using standard Python logger, removed --quiet option
• API: Made option --source-table optional to prepare for streaming sources
• Connectors: Added an mq-bridge source for streaming brokers
  (Kafka/NATS/AMQP/MQTT/ZeroMQ/IBM MQ/AWS SQS), via the <transport>+mqb://
  URI scheme, with dotted query keys (e.g. tls.required=true) for nested
  config such as TLS. Thanks, @marcomq.
• Maintenance: Started using orjson across the board
• Connectors: Centralize file-format to reader mapping across all sources.
  Thanks, @hampsterx.

Details

• fix(deps): update dependency ty to v0.0.54 by @renovate[bot] in #108
• fix(deps): update dependency asana to v5 by @renovate[bot] in #49
• fix(deps): update dependency pyrate-limiter to v4 by @renovate[bot] in #50
• CI: Run test families in parallel by @amotl in #98
• Tests: Touch all source- and target-connectors by @amotl in #113
• Core: Refactor API details to model by @amotl in #114
• fix(deps): update dependency ty to v0.0.55 by @renovate[bot] in #115
• Core: Start using standard Python logger, remove --quiet option by @amotl in #116
• API: Option --source-table becomes optional for streaming sources by @amotl in #119
• Update .coderabbit.yaml by @amotl in #121
• Maintenance: Use orjson across the board by @amotl in #122
• fix(deps): update dependency trino to v0.338.0 by @renovate[bot] in #128
• fix(deps): update dependency confluent-kafka to <2.16 by @renovate[bot] in #129
• Tests: Run Couchbase container on Docker-assigned ports by @hampsterx in #127
• Tests: Isolate container boot failures to their dependent tests by @hampsterx in #124
• Add mq-bridge by @marcomq in #125
• Sources: Centralize file-format to reader mapping by @hampsterx in #126
• fix(deps): update dependency ty to v0.0.56 by @renovate[bot] in #130
• fix(deps): update dependency oracledb to v4 by @renovate[bot] in #133

New Contributors

• @marcomq made their first contribution in #125

Full Changelog: v0.4.0...v0.5.0

v0.4.0

What's Changed

• Maintenance: Refactored module namespace. If you are using omniload
  as a library, this introduces many breaking changes. However, the new
  layout is much more ergonomic.
• Core: Added lazy-loading adapter module registry, to speed up startup times.

Details

• Chore: Add CodeRabbit configuration by @hampsterx in #69
• fix(deps): update dependency ty to v0.0.53 by @renovate[bot] in #87
• Refactor towards modular code layout. Naming things. by @amotl in #86
• Add integration marker and a Docker-free unit test lane (poe test-fast) by @hampsterx in #85
• Refactoring aftermath and adjustments by @amotl in #88
• Core: Lazy-loading adapter module registry, to speed up startup times by @amotl in #91
• Run the Docker-free unit lane on macOS and Windows by @hampsterx in #90

Full Changelog: v0.3.0...v0.4.0

v0.3.0

What's Changed

• Feature: Added embeddable run_ingest() Python API. Thanks, @hampsterx.
• Dependencies: Updated to dlt v1.28
• Dependencies: Updated to clickhouse-connect v1

Details

• Tests: Update OCI images for database containers to most recent versions by @amotl in #61
• Dependencies: Update dlt from v1.21 to v1.28 by @amotl in #60
• Tests: Migrate DynamoDB emulator from LocalStack to Floci by @amotl in #63
• Documentation: Fix broken examples and inaccuracies in getting-started by @hampsterx in #65
• Document all test credential env vars in test.env.template by @hampsterx in #66
• Surface container startup failures instead of swallowing them by @hampsterx in #67
• docs: clarify what dlt is and omniload's role in the About section by @hampsterx in #70
• Feature: Add embeddable run_ingest() Python API by @hampsterx in #68
• Chore: Satisfy link checker about LinkedIn by @amotl in #76
• Documentation Update README.md by @amotl in #72
• fix(deps): update dependency clickhouse-connect to v1 by @renovate[bot] in #35
• fix(deps): update dependency ty to v0.0.52 by @renovate[bot] in #74
• Tests: Retry flaky container boot and dump logs on startup failure by @hampsterx in #73
• Tests: Refactor tests into individual files and dedicated tests folder by @amotl in #79

New Contributors

• @hampsterx made their first contribution in #65

Full Changelog: v0.2.0...v0.3.0

v0.2.0

What's Changed

• Kafka: Expanded default_msg_processor into a miniature decoding unit
• dlt: Migrated from ensure_pendulum_datetime to ensure_pendulum_datetime_utc

Details

• fix(deps): update dependency typer to <0.27 by @renovate[bot] in #30
• fix(deps): update dependency google-analytics-data to <0.24 by @renovate[bot] in #29
• Chore: Update backlog by @amotl in #31
• Documentation copy editing — fix admonitions by @amotl in #32
• fix(deps): update dependency confluent-kafka to <2.15 by @renovate[bot] in #28
• fix(deps): update dependency testcontainers to <4.15 by @renovate[bot] in #27
• fix(deps): update dependency mypy to <1.21 by @renovate[bot] in #21
• fix(deps): update dependency asana to v5 by @renovate[bot] in #34
• fix(deps): update dependency elasticsearch to v9 by @renovate[bot] in #36
• fix(deps): update dependency google-ads to v31 by @renovate[bot] in #38
• fix(deps): update dependency pyairtable to v3 by @renovate[bot] in #40
• fix(deps): update dependency smartsheet-python-sdk to v4 by @renovate[bot] in #42
• fix(deps): update dependency stripe to v15 by @renovate[bot] in #43
• fix(deps): update dependency facebook-business to v25 by @renovate[bot] in #37
• fix(deps): update dependency pyrate-limiter to v4 by @renovate[bot] in #41
• fix(deps): update dependency mypy to v2 by @renovate[bot] in #39
• fix(deps): update dependency click to <8.5 by @renovate[bot] in #44
• Chore: Migrate from mypy to ty by @amotl in #14
• fix(deps): update dependency ty to v0.0.51 by @renovate[bot] in #48
• Kafka: Expand default_msg_processor into a miniature decoding unit by @amotl in #51
• dlt: Migrate to ensure_pendulum_datetime_utc by @amotl in #54

Full Changelog: v0.1.0...v0.2.0

v0.1.0

What's Changed

• Dependencies: Permitted installation of SQLAlchemy 2.0
• Dependencies: Relaxed package requirements across the board
• Documentation: Migrated from VitePress to Sphinx
• Runtime: Used tqdm progress bar instead of spinner

Details

• Dependencies: Permit installation of SQLAlchemy 2.0 by @amotl in #11
• Tests: Run integration tests against MS SQL Server only on Linux by @amotl in #12
• Dependencies: Relax package requirements across the board by @amotl in #13
• Documentation: Migrate from VitePress to Sphinx and Read the Docs by @amotl in #15
• Remove VitePress leftovers by @amotl in #16
• No spinner by @amotl in #17
• Improve documentation by @amotl in #18
• chore(deps): update astral-sh/setup-uv action to v8.2.0 by @renovate[bot] in #24
• chore(deps): update actions/checkout action to v7 by @renovate[bot] in #23
• chore(deps): update codecov/codecov-action action to v7 by @renovate[bot] in #25
• fix(deps): update dependency ruff to <0.16 by @renovate[bot] in #22

New Contributors

• @renovate[bot] made their first contribution in #24

Full Changelog: v0.0.0...v0.1.0