[hailtop] Add hailtop.fs for user-level RouterFS functions by daniel-goldstein · Pull Request #12731 · hail-is/hail

daniel-goldstein · 2023-02-24T16:53:11Z

CHANGELOG: Introduce hailtop.fs that makes public a filesystem module that works for local fs, gs, s3 and abs. This is now used as the Backend.fs for hail query but can be used standalone for Hail Batch users by import hailtop.fs as hfs.

Still have to do the docs but a couple questions remain:

I create a hidden singleton RouterFS object so that is used by functions in hailtop.fs. Should this singleton also be used by the Hail Query backends when they are initialized? How do we propagate configuration information such as requester_pays_bucket to the FS?

danking · 2023-04-06T14:10:33Z

            )

-        async_fs = RouterAsyncFS('file')
+        if not isinstance(gcs_requester_pays_configuration, str):


It can also be None, right?

I was initially going to throw this into another PR and then stack this on top. Do you want me to just keep it in here though?

Looks fixed as of now!

Hmm. Actually it looks like this is still failing service backend tests that explicitly request a handful of requester pays buckets, e.g.

test_requester_pays_with_project_more_than_one_partition

Ah, OK, I follow what your point is. You'll need to change RouterAsyncFS to intelligently use or not use the project based on the buckets and you hoped to do that in a follow up PR?

I think, unfortunately, we have to pull that into this one because folks, at least in principle, could be relying on that behavior.

In particular, IIRC, you get an error if you include a project on a non-requester-pays bucket, right?

daniel-goldstein · 2023-04-06T21:30:47Z

So I think I'd appreciate a review on this. Would especially appreciate feedback about the question I wrote in the PR body as well as what to do about documentation and testing:

We have pretty expansive FS testing, but not for these new shim functions. Should we convert some of our tests to use these functions instead of the FS objects themselves?
We don't have hailtop docs, and afaik this is the first module outside of hailtop.batch that would be public. Where should its docs go?

ready for actual review

danking · 2023-04-12T18:17:00Z

-                 project: Optional[str] = None,
                 **kwargs):
        if not storage_client:
-            if project is not None:


@daniel-goldstein you gotta fix copy.py line 118 as well.

@danking What do you mean here? I changed copy.py line 118 to use gcs_requester_pays_configuration, which I thought was correct.

I must've been looking at an old test

daniel-goldstein · 2023-04-13T14:11:50Z

Huh, quite confused as to why the local backend test hung in GCP and not azure. As far as I can tell the test that hung is test/hail/utils/test_utils.py::Tests::test_hadoop_ls_glob_2 but it passed in azure and locally. Any ideas?

danking · 2023-04-13T14:21:45Z

Seems to have starved itself of memory? https://batch.hail.is/batches/7260182/jobs/69

danking · 2023-04-13T14:28:42Z

Next thing I would try is running the full split. Maybe we need to run that in a memory limited way?

HAIL_RUN_IMAGE_SPLITS=14 \
HAIL_RUN_IMAGE_SPLIT_INDEX=2 \
HAIL_CLOUD=gcp \
HAIL_TEST_STORAGE_URI=/tmp/ \
HAIL_TEST_RESOURCES_DIR=./../src/test/resources/ \
HAIL_DOCTEST_DATA_DIR=./hail/docs/data \
HAIL_QUERY_BACKEND=local python3 -m pytest -Werror:::hail -Werror:::hailtop --log-cli-level=INFO -s -r A -vv --instafail --durations=50 --ignore=test/hailtop/ test

danking · 2023-04-13T14:29:45Z

Also not the most sophisticated thing but littering the test and the things it calls with print statements, making sure the tests print in real-time, and kicking off another test is something else I'd try.

danking · 2023-04-13T14:30:13Z

Though it does seem like something upstream of hadoop_ls_glob_2 has just sucked all the memory out of the container.

danking · 2023-04-13T14:32:36Z

Also ask Tim about limiting RAM available to Hail & Java, we should probably keep at least a gig dedicated to Python. Maybe that will cause memory errors to appear closer to where they belong.

danking · 2023-04-13T14:36:59Z

One more idea, can you add this https://pypi.org/project/pytest-timestamper/ package?

daniel-goldstein · 2023-04-18T21:17:07Z

Hm, I added timestamper and merged main and the tests passed… getting retested now though

danking

LGTM. This does remind me that we don't currently generate docs for hail top anywhere. We probably need to sit down and rethink the structure of the docs a bit.

danking · 2023-04-21T21:40:53Z

All the tests are passing but something is hanging.

danking · 2023-04-21T21:41:02Z

Like, pytest is hanging.

danking · 2023-04-21T21:43:13Z

CPU use is flat after that test completes successfully.

danking · 2023-04-21T21:45:23Z

Next thing to try is running the split on your laptop to reproduce

danking · 2023-04-21T21:48:17Z

So, there was a test run that got cancelled (probably main branch changed) but which passed the service backend tests.

It confirms that this most recent run ran all the tests, but it has some fishy looking error outputs:

[gw2] PASSED [2023-04-21 18:31:07] test/hail/table/test_table.py::Tests::test_from_pandas_missing_and_nans Exception ignored in: <_io.FileIO name=0 mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name=0 mode='r' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name=0 mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name=0 mode='r' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name=0 mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name=0 mode='r' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name=0 mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name=0 mode='r' encoding='UTF-8'>

Are we not cleaning up files somewhere and that's somehow hanging the system?

danking · 2023-04-21T21:56:14Z

I pushed a commit that will error on resource warning. Hopefully we can figure out where we're leaking, fix the leaks, and stop the hangs.

daniel-goldstein force-pushed the hail-fs branch 3 times, most recently from 1e3aad7 to fd34227 Compare February 27, 2023 16:52

daniel-goldstein force-pushed the hail-fs branch from 8048bd4 to 48cad87 Compare March 24, 2023 18:26

daniel-goldstein force-pushed the hail-fs branch 3 times, most recently from 98aeaa8 to 46c6128 Compare April 5, 2023 21:49

danking previously requested changes Apr 6, 2023

View reviewed changes

daniel-goldstein force-pushed the hail-fs branch from 3fd1a28 to aae1582 Compare April 6, 2023 21:24

daniel-goldstein marked this pull request as ready for review April 6, 2023 21:27

daniel-goldstein assigned danking Apr 6, 2023

danking reviewed Apr 12, 2023

View reviewed changes

daniel-goldstein added 11 commits April 12, 2023 20:37

pass requester pays project to service backend fs

61002a0

[hailtop] Add hailtop.fs for user-level RouterFS functions

6d05a19

fix

c5c7309

fix

f533803

have the fs dynamically choose which userProject to use on every request

91702e0

requester_pays_project is actually an optional

41b4b6f

missed it in get object metadata

88e9df1

repetitive but gets the job done

68c8b78

a bit less redundant

ba5a931

fix

fa4f12e

uh it does work now with service backend

654a826

daniel-goldstein force-pushed the hail-fs branch from 19ec594 to 654a826 Compare April 13, 2023 00:38

daniel-goldstein added 3 commits April 14, 2023 13:56

add timestamper

650282a

update pinned

e96fcac

Merge branch 'main' into hail-fs

a5bff7d

danking approved these changes Apr 19, 2023

View reviewed changes

daniel-goldstein added 2 commits April 21, 2023 13:32

Merge branch 'main' into hail-fs

f6e4638

fix after merge main

3c807b1

-Werror::ResourceWarning

b10fe96

danking merged commit 53f7bac into hail-is:main Apr 21, 2023

Uh oh!

Conversation

daniel-goldstein commented Feb 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danking Apr 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daniel-goldstein commented Apr 6, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daniel-goldstein commented Apr 13, 2023

Uh oh!

danking commented Apr 13, 2023

Uh oh!

danking commented Apr 13, 2023

Uh oh!

danking commented Apr 13, 2023

Uh oh!

danking commented Apr 13, 2023

Uh oh!

danking commented Apr 13, 2023

Uh oh!

danking commented Apr 13, 2023

Uh oh!

daniel-goldstein commented Apr 18, 2023

Uh oh!

danking left a comment

Choose a reason for hiding this comment

Uh oh!

danking commented Apr 21, 2023

Uh oh!

danking commented Apr 21, 2023

Uh oh!

danking commented Apr 21, 2023

Uh oh!

danking commented Apr 21, 2023

Uh oh!

danking commented Apr 21, 2023

Uh oh!

danking commented Apr 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

daniel-goldstein commented Feb 24, 2023 •

edited

Loading

danking Apr 7, 2023 •

edited

Loading