Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

when data source is empty and write disposition is replace, the destination table is not truncated #263

Closed
rudolfix opened this issue Apr 13, 2023 · 0 comments
Assignees
Labels
bug Something isn't working

Comments

@rudolfix
Copy link
Collaborator

rudolfix commented Apr 13, 2023

Description
When a resource has write disposition set to replace and it yields no data, dlt skips the loading process for that resource. Instead it should load "empty" file with the result of truncating destination table.

Steps to reproduce

info = dlt.run([1,2,3], table_name="tableeee", destination="duckdb")
# will not truncate `tableeee` table in duckdb
info = dlt.run([], table_name="tableeee", destination="duckdb")

Proposed fix

  1. we must detect the following situation: given resource with write disposition replace was extracted in extract step but produced no data items. a table schema for such resource already exists (it is not a first extract). in that case we must somehow truncate the target table(s). we could generate an empty file and handle this situation downstream.
  2. we must truncate all tables that are generated by the resource above. for that you will need store resource name that generated given table in Schema #210 implemented
  3. replace write disposition must always use staging dataset. #271 must be implemented first

Tests
We must test thoroughly

  • condition (1) in proposed fix: many cases with new replace resources, resources simply skipped in future loads etc.
  • resource generating several tables
  • child tables must be truncated as well (this is another bug btw.)
@rudolfix rudolfix added the bug Something isn't working label Apr 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants