Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for MAXFILESIZE option in UNLOAD #123

Closed
atrigent opened this issue Jul 25, 2017 · 6 comments
Closed

Add support for MAXFILESIZE option in UNLOAD #123

atrigent opened this issue Jul 25, 2017 · 6 comments

Comments

@atrigent
Copy link

The UNLOAD command now has a very useful MAXFILESIZE option to control the maximum size of the produced files. The UnloadFromSelect construct in the dialect should have support for that.

http://docs.aws.amazon.com/redshift/latest/dg/r_UNLOAD.html

@jleclanche
Copy link
Contributor

I wanted to implement this but then, what would the API look like? The syntax is max-size [MB | GB]. So we have two values to choose.

Choices:

  • UnloadFromSelect(max_file_size_mb=123), UnloadFromSelect(max_file_size_gb=6.2) (mutually-exclusive)
  • UnloadFromSelect(max_file_size=(123, "MB")), UnloadFromSelect(max_file_size=(6.2, "GB")) (tuple int or float with string)
  • UnloadFromSelect(max_file_size=(123, Size.MB)), UnloadFromSelect(max_file_size=(6.2, Size.GB)) (tuple int or float with enum)
  • UnloadFromSelect(max_file_size=123), UnloadFromSelect(max_file_size=6.2 * 1024) (only accept megabytes, auto-calculate when to show GB instead [if we even need to? Does the syntax allow for "6348 MB"?])

@graingert
Copy link
Member

what sort of MB are these? MiB or MB?

@jleclanche
Copy link
Contributor

Docs don't specify. Probably MiB, but would have to test.

@graingert
Copy link
Member

once someone has found out, I think this should accept bytes and convert to "MB"

@jleclanche
Copy link
Contributor

I don't think a bytes-level API is ideal, makes it a little impractical to use :/

@graingert
Copy link
Member

graingert commented Oct 1, 2017

@jleclanche bytes is perfect, then people can multiply using our own flags, or their own:

class Size(enum.IntEnum):
    MiB = 2**20
    MB = 10**6
    GiB = 2**30
    GB =  10**9

UnloadFromSelect(max_file_size=123 * Size.MB) // can use our flags
UnloadFromSelect(max_file_size=123_000_000) // can use fancy new _ numbers
UnloadFromSelect(max_file_size=123 * 1024 * 1024) // can use their own multiplies

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants