Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PLAN] Reconsider auto_create_table option because it is not intuitive #98

Closed
sonots opened this issue May 11, 2019 · 1 comment · Fixed by #105
Closed

[PLAN] Reconsider auto_create_table option because it is not intuitive #98

sonots opened this issue May 11, 2019 · 1 comment · Fixed by #105

Comments

@sonots
Copy link
Member

sonots commented May 11, 2019

PROBLEM

auto_create_table: true option is not intuitive because, tables are automatically created in serveral modes such as replace even if auto_create_table: false (default).

This is because it is mandatory to create a new table with mode: replace because it works like

  • create a temp table (HERE)
  • load data into a temp table
  • copy the temp table to the target table atomically. (or HERE)
  • delete the temp table

BACKGROUND REASON

This plugin has behaved like this from old days, when this plugin was java implementation.

I guess (I am not the original author) this is because

  • auto_create_table: false is the default value
  • raising error with mode: replace under a default configuration feels bad

IDEA

Change the default value to something like auto_create_table: 'auto'?

And, then, raise an error with contradictory configuration such as

  • mode: replace and auto_create_table: false
  • mode: delete_in_advance and auto_create_table: false
  • TODO: write completed list

ANOTHER PROBLEM

For partitioned table #58, mode: replace copy tables into a partition rather than a table.
Thus, it does not create a table. It requires that a table already exists.

auto_create_table: true or false works to create a table this case as its word describes.
It is not mandatory to create a new table with mode: replace for a partitioned table.

CONCLUSION

So...?

@sonots sonots changed the title [PLAN] auto_create_table option is not intuitive [PLAN] Reconsider auto_create_table option because it is not intuitive May 11, 2019
@sonots
Copy link
Member Author

sonots commented Aug 10, 2019

Let me change as following simply:

  • auto_create_table: false is available only for mode: append_direct
  • Other modes requires auto_create_table: true
  • Change the default value of auto_create_table to true.
  • Drop support of auto_create_table: false for creating a partition (with partitioning decorator)

This is based on my assumption that

  • there are few people using mode: append_direct
  • there are few people creating a table with partitioning decorator (because we can use field partitioning now) together with auto_create_table: false
  • Most of people using replace mode with the previous behavior which always creates a temp table automatically

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging a pull request may close this issue.

1 participant