Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fs] hailtop.fs makes it impossible to explicitly configure the GCS requester pays project in code #13567

Closed
danking opened this issue Sep 6, 2023 · 3 comments · Fixed by #13795
Assignees

Comments

@danking
Copy link
Collaborator

danking commented Sep 6, 2023

What happened?

See: https://hail.zulipchat.com/#narrow/stream/128581-Cloud-support/topic/requester.20pays.20in.20batch

We need a way to configure the RouterFS in a Python file. Maybe hfs.init? I'm not sure.

Version

0.2.120

Relevant log output

No response

@danking
Copy link
Collaborator Author

danking commented Oct 11, 2023

Where do I write requester pays information?

import hailtop.fs as hfs
with hfs.open('gs://bucket/path/to/file') as fobj:
    ....

@danking
Copy link
Collaborator Author

danking commented Oct 11, 2023

Probably need an hfs.init?

@danking
Copy link
Collaborator Author

danking commented Oct 11, 2023

Options:

  1. hfs.init. Pros: applies to all uses of open/close/etc. Cons: global state.
  2. hfs.open(..., requester_pays_configuration=...). Pros: allows use-site overriding. Cons: user has to specify at every open.
  3. both? Pros: easy way to set a default, and allows use-site overrides. Cons: committing ourselves to another global state.

Let's do option (2) for now.

iris-garden added a commit to iris-garden/hail that referenced this issue Oct 11, 2023
iris-garden added a commit to iris-garden/hail that referenced this issue Oct 12, 2023
iris-garden added a commit to iris-garden/hail that referenced this issue Oct 12, 2023
iris-garden added a commit to iris-garden/hail that referenced this issue Oct 12, 2023
iris-garden added a commit to iris-garden/hail that referenced this issue Oct 12, 2023
@danking danking added the fs label Oct 23, 2023
iris-garden added a commit to iris-garden/hail that referenced this issue Oct 26, 2023
iris-garden added a commit to iris-garden/hail that referenced this issue Oct 26, 2023
iris-garden added a commit to iris-garden/hail that referenced this issue Oct 26, 2023
iris-garden added a commit to iris-garden/hail that referenced this issue Nov 8, 2023
iris-garden added a commit to iris-garden/hail that referenced this issue Nov 8, 2023
iris-garden added a commit to iris-garden/hail that referenced this issue Nov 9, 2023
iris-garden added a commit to iris-garden/hail that referenced this issue Nov 9, 2023
iris-garden added a commit to iris-garden/hail that referenced this issue Nov 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants