[batch] adds requester pays config to `hailtop.fs.open` #13795

iris-garden · 2023-10-11T20:33:24Z

daniel-goldstein

Looking good, just a few organizational comments

daniel-goldstein · 2023-10-19T18:26:44Z

hail/python/hailtop/fs/fs_utils.py

+    ...     requester_pays_config=('my-project', ['my-bucket', 'bucket-2'])
+    ... ) as f:
+    ...     for line in f:
+    ...         print(line.strip())


I feel like this should only support requester_pays_project, since you can't open from more than one bucket.

AFAICT the only method where it would be possible to specify multiple buckets would be copy. It's perhaps worth asking though if anyone would ever want to specify two different requester pays projects when copying between two buckets. Seems kind of weird and like it's leading to some more annoying bookkeeping than if we just had a Dict[str, FS].

good point; checked in with @danking and he said it's not inconceivable that someone would want to put together one big requester pays config and just pass the same object into multiple invocations of open, so hopefully this latest change reduces the bookkeeping enough that it's not too big of an issue to support the Tuple[str, List[str]] case of the type

daniel-goldstein · 2023-10-19T18:35:53Z

hail/python/hailtop/utils/gcs_requester_pays.py

+    hashable_config = (
+        requester_pays_config
+        if requester_pays_config is None or isinstance(requester_pays_config, str)
+        else f"{requester_pays_config[0]}:{','.join(requester_pays_config[1])}"


I think this will produce different keys for ("project", ["bucket1", "bucket2"]) vs ("project", ["bucket2", "bucket1"]). If doing this interpolation is stemming from lists not being hashable, you can construct a Tuple[str, frozenset] which should be hashable.

ooo good catch, updated to use a frozenset

daniel-goldstein · 2023-10-19T18:45:10Z

hail/python/hailtop/utils/gcs_requester_pays.py

+    except KeyError:
+        requester_pays_fses[hashable_config] = cls(
+            gcs_kwargs={"gcs_requester_pays_configuration": requester_pays_config}
+        )


Maybe its the introduction of the hashable stuff, but the bookkeeping around the requester_pays_fses is starting to feel a little messy. I wonder if instead of keeping a dict around + using this utility method, we could instead subclass defaultddict?

i definitely like the approach of subclassing DefaultDict better haha, updated to do that

_

daniel-goldstein

Sorry for the delay on this, this looks great! Just a couple more organizational nits.

daniel-goldstein · 2023-10-31T21:36:29Z

hail/python/hailtop/batch/backend.py

@@ -160,7 +157,12 @@ def __init__(self,
            flags += f' -v {gsa_key_file}:/gsa-key/key.json'

        self._extra_docker_run_flags = flags
-        self.__fs = RouterAsyncFS()
+        self.__requester_pays_fses = gcs_requester_pays_fs_cache(fs_constructor=RouterAsyncFS)


Instead of having this field in both subclasses and requiring an abstract property, you can pass a GCSRequesterPaysFSCache to super().__init__

good call, updated!

daniel-goldstein · 2023-10-31T21:38:25Z

hail/python/hailtop/utils/gcs_requester_pays.py

+
+
+class GCSRequesterPaysFSCache(DefaultDict[MaybeGCSRequesterPaysConfiguration, FS]):
+    def __init__(self, *args: Any, fs_constructor: Type[FS], default_kwargs: Dict[str, Any] = {}, **kwargs: Any) -> None:


Is the only use the *args here the default factory in the gcs_requester_pays_fs_cache function? If so can we restrict to just that argument and add more as necessary down the line? Makes for nicer type checking.

On a similar note, I don't see **kwargs used. Can we remove it until we need it?

daniel-goldstein · 2023-10-31T21:43:51Z

hail/python/hailtop/utils/gcs_requester_pays.py

+class GCSRequesterPaysFSCache(DefaultDict[MaybeGCSRequesterPaysConfiguration, FS]):
+    def __init__(self, *args: Any, fs_constructor: Type[FS], default_kwargs: Dict[str, Any] = {}, **kwargs: Any) -> None:
+        self._fs_constructor = fs_constructor
+        self[None] = fs_constructor(**default_kwargs)


I feel like a cache shouldn't eagerly construct elements. Can we delete this and let the normal behavior deal with this case?

daniel-goldstein · 2023-10-31T21:49:30Z

hail/python/test/hailtop/utils/test_gcs_requester_pays.py

+        kwargs_2 = {"gcs_requester_pays_configuration": config_2}
+        fses = gcs_requester_pays_fs_cache(cls)
+        assert attrgetter(kwargs_field)(fses[None]) == {}
+        assert attrgetter(kwargs_field)(fses[config_1]) == kwargs_1


Interesting, I've never seen attrgetter before. I think I would prefer though adding a _gcs_kwargs property to both RouterFS and RouterAsyncFS and accessing that property directly, so that if I mess something up the linters will tell me that these attr accesses will fail without me having to run this test.

daniel-goldstein · 2023-10-31T21:50:18Z

hail/python/test/hailtop/utils/test_gcs_requester_pays.py

+        assert attrgetter(kwargs_field)(default_kwargs_fses[None]) == kwargs_1
+
+    test_with_cls(RouterFS, "afs._gcs_kwargs")
+    test_with_cls(RouterAsyncFS, "_gcs_kwargs")


I feel like we can instead make test_with_cls the actual test method and use pytest.mark.parametrize with the two classes.

daniel-goldstein

One last thing. We should no longer need to apply nest_asyncio in this constructor.

daniel-goldstein · 2023-11-20T20:28:44Z

hail/python/hailtop/batch/backend.py

-        self._requester_pays_fses: Dict[GCSRequesterPaysConfiguration, RouterAsyncFS] = {}
+    def __init__(self, requester_pays_fses: GCSRequesterPaysFSCache):
+        import nest_asyncio  # pylint: disable=import-outside-toplevel
+        nest_asyncio.apply()


I think this nest_asyncio was deleted in main. Is this still necessary?

ooo good catch, thank you!

Closes hail-is#13567.

_

iris-garden force-pushed the batch/fs-requester-pays branch 4 times, most recently from b4ece33 to da24ff8 Compare October 12, 2023 15:33

iris-garden assigned daniel-goldstein Oct 12, 2023

daniel-goldstein previously requested changes Oct 19, 2023

View reviewed changes

iris-garden force-pushed the batch/fs-requester-pays branch 3 times, most recently from 3943d78 to e823430 Compare October 26, 2023 20:46

daniel-goldstein requested changes Oct 31, 2023

View reviewed changes

iris-garden force-pushed the batch/fs-requester-pays branch 4 times, most recently from bc81b1a to c3e8e04 Compare November 9, 2023 21:36

daniel-goldstein previously requested changes Nov 20, 2023

View reviewed changes

[batch] adds requester pays config to hailtop.fs.open

ad8e221

Closes hail-is#13567.

iris-garden force-pushed the batch/fs-requester-pays branch from c3e8e04 to 287220a Compare November 21, 2023 16:12

removes bespoke constructor function and fixes types

287220a

daniel-goldstein approved these changes Nov 28, 2023

View reviewed changes

Merge branch 'main' into batch/fs-requester-pays

b07e60e

danking merged commit cf293b9 into hail-is:main Jan 3, 2024
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[batch] adds requester pays config to `hailtop.fs.open` #13795

[batch] adds requester pays config to `hailtop.fs.open` #13795

iris-garden commented Oct 11, 2023

daniel-goldstein left a comment

daniel-goldstein Oct 19, 2023

daniel-goldstein Oct 19, 2023

iris-garden Oct 26, 2023

daniel-goldstein Oct 19, 2023 •

edited

iris-garden Oct 26, 2023

daniel-goldstein Oct 19, 2023

iris-garden Oct 26, 2023

daniel-goldstein left a comment

daniel-goldstein Oct 31, 2023

iris-garden Nov 8, 2023

daniel-goldstein Oct 31, 2023

daniel-goldstein Oct 31, 2023

iris-garden Nov 8, 2023

daniel-goldstein Oct 31, 2023

iris-garden Nov 8, 2023

daniel-goldstein Oct 31, 2023

iris-garden Nov 8, 2023

daniel-goldstein Oct 31, 2023

iris-garden Nov 8, 2023

daniel-goldstein left a comment

daniel-goldstein Nov 20, 2023

iris-garden Nov 21, 2023



		class GCSRequesterPaysFSCache(DefaultDict[MaybeGCSRequesterPaysConfiguration, FS]):
		def __init__(self, args: Any, fs_constructor: Type[FS], default_kwargs: Dict[str, Any] = {}, *kwargs: Any) -> None:

[batch] adds requester pays config to hailtop.fs.open #13795

[batch] adds requester pays config to hailtop.fs.open #13795

Conversation

iris-garden commented Oct 11, 2023

daniel-goldstein left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daniel-goldstein Oct 19, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daniel-goldstein left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daniel-goldstein left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

[batch] adds requester pays config to `hailtop.fs.open` #13795

[batch] adds requester pays config to `hailtop.fs.open` #13795

daniel-goldstein Oct 19, 2023 •

edited