[copy] finally eliminate all stalls from the copy tool #11778

danking · 2022-04-19T21:39:44Z

No changelog because the copy tool is not properly public, though
maybe it can be public now.

The main problem was that we were saturating our network bandwidth
with data we did not need. Why? As far as I can tell, release on
an aiohttp.ClientResponse is insufficient to stop the receipt of
more bytes.

I realized one proper fix was to simply tell the cloud storage vendors
how much data we wanted to receive. That is the change to open_from
which I have made pervasively to all clouds.

I also changed release to close because that seems more correct.
I do not understand why we only released it before.

jigold

Rebase needed as well.

jigold · 2022-04-20T17:11:26Z

hail/python/hailtop/aiocloud/aioaws/fs.py

        bucket, name = self._get_bucket_name(url)
+        range_str = f'bytes={start}-'
+        if length is not None:
+            range_str += str(start + length)


Can we double check the definitions of the range are correct for all clouds? I vaguely recall Google might have different semantics and the end point is inclusive.

https://cloud.google.com/storage/docs/samples/storage-download-byte-range#storage_download_byte_range-python

Yeah, you're totally right it's inclusive. I'll fix.

https://cloud.google.com/storage/docs/xml-api/reference-headers#range

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range

https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html#API_GetObject_RequestSyntax

https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35

jigold · 2022-04-20T17:14:14Z

hail/python/hailtop/aiocloud/aioazure/fs.py

@@ -156,7 +157,7 @@ async def read(self, n: int = -1) -> bytes:

        if n == -1:
            try:
-                downloader = await self._client.download_blob(offset=self._offset)
+                downloader = await self._client.download_blob(offset=self._offset, length=self._length)


I can't remember exactly, but I might have run into issues with Azure when the file is empty and you ask for 0 bytes.

Totally right. I can add a helpful error message for azure.

HttpResponseError: The range specified is invalid for the current size of the resource. RequestId:314c0805-301e-001f-64f6-547b9b000000 Time:2022-04-20T20:38:34.1301964Z ErrorCode:InvalidRange

Hrm. that's annoying to have one give an error, so I'll just have it return an empty readable.

jigold · 2022-04-20T17:17:56Z

hail/python/hailtop/httpx.py

@@ -38,8 +38,8 @@ class ClientResponse:
    def __init__(self, client_response: aiohttp.ClientResponse):
        self.client_response = client_response

-    def release(self) -> None:
-        return self.client_response.release()
+    async def release(self) -> None:


What is the reason for this change? I looked at aiohttp.ClientResponse and release is not async.

release is async: https://docs.aiohttp.org/en/stable/client_reference.html#aiohttp.ClientResponse.release

I returns "noop" which is a generator which makes the function async.

jigold · 2022-04-20T17:18:14Z

hail/python/test/hailtop/inter_cloud/test_fs.py

+    async with await fs.create(file) as f:
+        await f.write(b'abcde')
+
+    async with await fs.open_from(file, 2, 2) as f:


Can you add a test with an empty file?

great comments, all addressed, I added some relevant tests

danking · 2022-04-20T22:00:17Z

Actually, I decided to just properly implement truncated files in local fs too. I can't use .truncate because I'm fairly certain that literally truncates the file on disk.

jigold

There's also CI errors for check_services etc.

jigold · 2022-04-26T18:17:02Z

hail/python/test/hailtop/inter_cloud/test_fs.py

+
+    await fs.write(file, b'')
+
+    assert await fs.read_range(file, 0, 0, end_inclusive=False) == b''


What happens if end_inclusive=True? Error?

That will try to read the zeroth byte from an empty file and will trigger an error. I've added a new test for that case, test_read_range_end_inclusive_empty_file_should_error. In Google, you get

hailtop.httpx.ClientResponseError: 416, message='Requested range not satisfiable'

In local, you get:

hailtop.aiotools.fs.exceptions.UnexpectedEOFError

Hmm. I suppose we should catch and convert these all to UnexpectedEOF errors

OK, all now UnexpectedEOFError and a test for that.

jigold · 2022-04-26T18:20:18Z

hail/python/hailtop/aiotools/local_fs.py

+        return self.bio.isatty()
+
+    def read(self, n: int = -1):
+        assert self.n <= self.limit


I find self.n confusing. Can we name it self.offset?

jigold · 2022-04-26T18:21:03Z

hail/python/hailtop/aiotools/local_fs.py

+            n = self.limit - self.n
+        else:
+            n = min(self.limit - self.n, n)
+        b = self.bio.read(self.limit - self.n)


shouldn't this be b = self.bio.read(n)?

jigold · 2022-05-11T20:35:53Z

Bump.

x

danking · 2022-05-13T17:21:34Z

Let's try to get this merged today, I don't have meetings so I can respond quickly to changes

jigold · 2022-05-13T18:02:02Z

hail/python/hailtop/aiotools/fs/fs.py

@@ -108,7 +108,8 @@ async def open(self, url: str) -> ReadableStream:

    async def open_from(self, url: str, start: int, *, length: Optional[int] = None) -> ReadableStream:
        if length == 0:
-            isfile, isdir = await asyncio.gather(self.isfile(url), self.isdir(url))
+            assert not url.endswith('/')


Was this not working correctly before? I'm not sure I like this change. What happens if the file ends in a slash?

Ah I see. We can't distinguish an empty file that's a directory or a file?

This is because isdir fails an assert if the url doesn't end with a slash. I kind of hate that we assert that, but it seems to be a pervasive assumption. If that's the assumption, assuming we never do open_from('gs://bucket/foo/') also seems reasonable.

x

No changelog because the copy tool is not properly public, though maybe it can be public now. The main problem was that we were saturating our network bandwidth with data we did not need. Why? As far as I can tell, `release` on an `aiohttp.ClientResponse` is insufficient to stop the receipt of more bytes. I realized one proper fix was to simply tell the cloud storage vendors how much data we wanted to receive. That is the change to `open_from` which I have made pervasively to all clouds. I also changed `release` to `close` because that seems more correct. I do not understand why we only `release`d it before.

danking · 2022-05-16T15:51:23Z

The assertion causes a bunch of tests to fail which expect to get an IsADirectoryError. I removed the assertion and just construct two urls: file_url and dir_url which don't (resp do) end in a slash.

danking assigned jigold Apr 19, 2022

jigold previously requested changes Apr 20, 2022

View reviewed changes

danking force-pushed the fix-copy branch from d0de66e to c881265 Compare April 20, 2022 21:43

jigold previously requested changes Apr 26, 2022

View reviewed changes

danking mentioned this pull request May 13, 2022

[release] 0.2.95 #11834

Merged

jigold previously requested changes May 13, 2022

View reviewed changes

jigold approved these changes May 13, 2022

View reviewed changes

Daniel King added 17 commits May 16, 2022 09:55

address comments

2e9c755

fix for azure and test

f92bb4a

fixes

2f7ed7e

fix

dbaba35

fixes for local too

fbb6822

fix length=0;

2ebb60a

fix

41fc894

fix gcs open_from name

8144e90

fixes

3827f6f

add missing abstract methgod

535443c

fix pylint

06123f4

fix

e2829ee

fixes

b253bab

n to offset

c8ab84c

fixes

67572b2

notimplemented abstract methods

b44b7a5

fix

75f938f

danking force-pushed the fix-copy branch from f696ef4 to 75f938f Compare May 16, 2022 13:56

Daniel King added 2 commits May 16, 2022 11:48

fix

b7ad67b

fix

6569a40

danking merged commit 0481bae into hail-is:main May 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[copy] finally eliminate all stalls from the copy tool #11778

[copy] finally eliminate all stalls from the copy tool #11778

danking commented Apr 19, 2022

jigold left a comment

jigold Apr 20, 2022

danking Apr 20, 2022

jigold Apr 20, 2022

danking Apr 20, 2022

danking Apr 20, 2022

jigold Apr 20, 2022

danking Apr 20, 2022

jigold Apr 20, 2022

danking Apr 20, 2022

danking commented Apr 20, 2022

jigold left a comment

jigold Apr 26, 2022

danking May 13, 2022

danking May 13, 2022

danking May 13, 2022

jigold Apr 26, 2022

danking May 13, 2022

jigold Apr 26, 2022

danking May 13, 2022

jigold commented May 11, 2022

danking commented May 13, 2022

jigold May 13, 2022

danking May 13, 2022

danking commented May 16, 2022


		await fs.write(file, b'')

		assert await fs.read_range(file, 0, 0, end_inclusive=False) == b''

[copy] finally eliminate all stalls from the copy tool #11778

[copy] finally eliminate all stalls from the copy tool #11778

Conversation

danking commented Apr 19, 2022

jigold left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danking commented Apr 20, 2022

jigold left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jigold commented May 11, 2022

danking commented May 13, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danking commented May 16, 2022