[hailctl] describe with requester pays project id fails

[chrisvittal]

What happened?

When running hailctl describe -u, we attempt to initialize aiohttp.ClientSession with passthrough args. One of these args is GCS specific and should not be used for the aiohttp.ClientSession.

Version

b598a78

Relevant log output

$ hailctl describe -ubroad-mpg-gnomad gs://gnomad/v4.0/raw/exomes/gnomad_v4.0.vds/variant_data
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/cvittal/src/hail/hail/python/hailtop/hailctl/describe.py:104 in describe                  │
│                                                                                                  │
│   101 │   '''                                                                                    │
│   102 │   Describe the MatrixTable or Table at path FILE.                                        │
│   103 │   '''                                                                                    │
│ ❱ 104 │   asyncio.get_event_loop().run_until_complete(async_describe(file, requester_pays_proj   │
│   105                                                                                            │
│   106                                                                                            │
│   107 async def async_describe(                                                                  │
│                                                                                                  │
│ /opt/homebrew/Cellar/python@3.10/3.10.13/Frameworks/Python.framework/Versions/3.10/lib/python3.1 │
│ 0/asyncio/base_events.py:649 in run_until_complete                                               │
│                                                                                                  │
│    646 │   │   if not future.done():                                                             │
│    647 │   │   │   raise RuntimeError('Event loop stopped before Future completed.')             │
│    648 │   │                                                                                     │
│ ❱  649 │   │   return future.result()                                                            │
│    650 │                                                                                         │
│    651 │   def stop(self):                                                                       │
│    652 │   │   """Stop running the event loop.                                                   │
│                                                                                                  │
│ /Users/cvittal/src/hail/hail/python/hailtop/hailctl/describe.py:119 in async_describe            │
│                                                                                                  │
│   116 │   │   gcs_kwargs['project'] = requester_pays_project_id                                  │
│   117 │                                                                                          │
│   118 │   async with aio_contextlib.closing(RouterAsyncFS(gcs_kwargs=gcs_kwargs)) as fs:         │
│ ❱ 119 │   │   j = orjson.loads(decompress(await fs.read(path.join(file, 'metadata.json.gz')),    │
│   120 │   │                                                                                      │
│   121 │   │   # Get the file schema                                                              │
│   122 │   │   file_schema = parse_schema(j[next(k for k in j.keys() if k.endswith('type'))])     │
│                                                                                                  │
│ /Users/cvittal/src/hail/hail/python/hailtop/aiotools/fs/fs.py:281 in read                        │
│                                                                                                  │
│   278 │   │   │   pass                                                                           │
│   279 │                                                                                          │
│   280 │   async def read(self, url: str) -> bytes:                                               │
│ ❱ 281 │   │   async with await self.open(url) as f:                                              │
│   282 │   │   │   return await f.read()                                                          │
│   283 │                                                                                          │
│   284 │   async def read_from(self, url: str, start: int) -> bytes:                              │
│                                                                                                  │
│ /Users/cvittal/src/hail/hail/python/hailtop/aiotools/router_fs.py:75 in open                     │
│                                                                                                  │
│    72 │   │   return self._load_fs(uri)                                                          │
│    73 │                                                                                          │
│    74 │   async def open(self, url: str) -> ReadableStream:                                      │
│ ❱  75 │   │   fs = self._get_fs(url)                                                             │
│    76 │   │   return await fs.open(url)                                                          │
│    77 │                                                                                          │
│    78 │   async def _open_from(self, url: str, start: int, *, length: Optional[int] = None) ->   │
│                                                                                                  │
│ /Users/cvittal/src/hail/hail/python/hailtop/aiotools/router_fs.py:72 in _get_fs                  │
│                                                                                                  │
│    69 │   │   for fs in self._filesystems:                                                       │
│    70 │   │   │   if fs.valid_url(uri):                                                          │
│    71 │   │   │   │   return fs                                                                  │
│ ❱  72 │   │   return self._load_fs(uri)                                                          │
│    73 │                                                                                          │
│    74 │   async def open(self, url: str) -> ReadableStream:                                      │
│    75 │   │   fs = self._get_fs(url)                                                             │
│                                                                                                  │
│ /Users/cvittal/src/hail/hail/python/hailtop/aiotools/router_fs.py:54 in _load_fs                 │
│                                                                                                  │
│    51 │   │   if LocalAsyncFS.valid_url(uri):                                                    │
│    52 │   │   │   fs = LocalAsyncFS(**self._local_kwargs)                                        │
│    53 │   │   elif aiogoogle.GoogleStorageAsyncFS.valid_url(uri):                                │
│ ❱  54 │   │   │   fs = aiogoogle.GoogleStorageAsyncFS(                                           │
│    55 │   │   │   │   **self._gcs_kwargs,                                                        │
│    56 │   │   │   │   bucket_allow_list = self._gcs_bucket_allow_list.copy()                     │
│    57 │   │   │   )                                                                              │
│                                                                                                  │
│ /Users/cvittal/src/hail/hail/python/hailtop/aiocloud/aiogoogle/client/storage_client.py:602 in   │
│ __init__                                                                                         │
│                                                                                                  │
│   599 │   │   │   │    bucket_allow_list: Optional[List[str]] = None,                            │
│   600 │   │   │   │    **kwargs):                                                                │
│   601 │   │   if not storage_client:                                                             │
│ ❱ 602 │   │   │   storage_client = GoogleStorageClient(**kwargs)                                 │
│   603 │   │   self._storage_client = storage_client                                              │
│   604 │   │   if bucket_allow_list is None:                                                      │
│   605 │   │   │   bucket_allow_list = []                                                         │
│                                                                                                  │
│ /Users/cvittal/src/hail/hail/python/hailtop/aiocloud/aiogoogle/client/storage_client.py:309 in   │
│ __init__                                                                                         │
│                                                                                                  │
│   306 │   │   if 'timeout' not in kwargs and 'http_session' not in kwargs:                       │
│   307 │   │   │   # Around May 2022, GCS started timing out a lot with our default 5s timeout    │
│   308 │   │   │   kwargs['timeout'] = aiohttp.ClientTimeout(total=20)                            │
│ ❱ 309 │   │   super().__init__('https://storage.googleapis.com/storage/v1', **kwargs)            │
│   310 │   │   self._gcs_requester_pays_configuration = get_gcs_requester_pays_configuration(     │
│   311 │   │   │   gcs_requester_pays_configuration=gcs_requester_pays_configuration              │
│   312 │   │   )                                                                                  │
│                                                                                                  │
│ /Users/cvittal/src/hail/hail/python/hailtop/aiocloud/aiogoogle/client/base_client.py:15 in       │
│ __init__                                                                                         │
│                                                                                                  │
│   12 │   def __init__(self, base_url: str, *, session: Optional[GoogleSession] = None,           │
│   13 │   │   │   │    rate_limit: Optional[RateLimit] = None, **kwargs):                         │
│   14 │   │   if session is None:                                                                 │
│ ❱ 15 │   │   │   session = GoogleSession(**kwargs)                                               │
│   16 │   │   super().__init__(base_url, session, rate_limit=rate_limit)                          │
│   17                                                                                             │
│                                                                                                  │
│ /Users/cvittal/src/hail/hail/python/hailtop/aiocloud/aiogoogle/session.py:18 in __init__         │
│                                                                                                  │
│   15 │   │   │   │   credentials = GoogleCredentials.from_file(credentials_file)                 │
│   16 │   │   │   else:                                                                           │
│   17 │   │   │   │   credentials = GoogleCredentials.default_credentials()                       │
│ ❱ 18 │   │   super().__init__(credentials=credentials, params=params, **kwargs)                  │
│   19                                                                                             │
│                                                                                                  │
│ /Users/cvittal/src/hail/hail/python/hailtop/aiocloud/common/session.py:80 in __init__            │
│                                                                                                  │
│    77 │   │   │   self._http_session = http_session                                              │
│    78 │   │   else:                                                                              │
│    79 │   │   │   self._owns_http_session = True                                                 │
│ ❱  80 │   │   │   self._http_session = httpx.ClientSession(**kwargs)                             │
│    81 │   │   self._credentials = credentials                                                    │
│    82 │                                                                                          │
│    83 │   async def request(self, method: str, url: str, **kwargs) -> aiohttp.ClientResponse:    │
│                                                                                                  │
│ /Users/cvittal/src/hail/hail/python/hailtop/httpx.py:110 in __init__                             │
│                                                                                                  │
│   107 │   │   │   timeout = aiohttp.ClientTimeout(total=5)                                       │
│   108 │   │                                                                                      │
│   109 │   │   self.raise_for_status = raise_for_status                                           │
│ ❱ 110 │   │   self.client_session = aiohttp.ClientSession(                                       │
│   111 │   │   │   *args,                                                                         │
│   112 │   │   │   timeout=timeout,                                                               │
│   113 │   │   │   raise_for_status=False,                                                        │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: ClientSession.__init__() got an unexpected keyword argument 'project'
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x1041b6980>