GCSFUSE hanging while trying to write large files (~6GB) #1541

tarun360 · 2023-12-08T07:10:52Z

Describe the issue
I am trying to cp a relatively large single file, around 6GB, through GCSFUSE, and it just hangs forever. I am able to use gsutil/gcloud-storage-rsync to copy the same file, so I am sure there's no bandwidth problem here. I have also ensured that there's enough space in --temp-dir=/tmp to write a 6GB file. Doing ls from separate terminal is also stuck.

I don't see this issue with smaller files around 200MB.

I think this issue is similar to #715.

This is greatly limiting our use of GCSFUSE, hence requesting to please have look urgently.

To Collect more Debug logs

Relevant GCSFUSE logs while doing cp:

{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002906        connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5228199936, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":109021661}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000028f0        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":109514277}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000028f2        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":109968981}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002908        connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5229248512, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":110087270}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000290a        connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5230297088, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":110350364}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000028f4        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":110457680}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000290c        connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5231345664, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":110798379}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000028f8        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":112676976}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000028f6        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":112803534}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000028fa        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":112841686}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000028fc        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":112869553}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000290e        connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5232394240, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":112983238}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002910        connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5233442816, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":113224758}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002912        connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5234491392, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":113465212}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002914        connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5235539968, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":113721204}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000028fe        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":114112281}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002900        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":114194980}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002916        connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5236588544, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":114386752}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002918        connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5237637120, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":114636800}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002902        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":114689493}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000291a        connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5238685696, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":114994381}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002904        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":115484631}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000291c        connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5239734272, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":115805222}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002906        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":116074374}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000291e        connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5240782848, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":116348745}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002908        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":116654494}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002920        connection.go:416] \u003c- WriteFile (inode 4, PID 0, handle 0, offset 5241831424, 1048576 bytes)\n","timestampSeconds":1702017866,"timestampNanos":116973897}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000290a        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":117296953}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000290c        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":117931049}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002914        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":118536042}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000290e        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":119144253}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002910        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":119719868}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002912        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":120347024}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002916        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":120978491}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002918        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":121618115}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000291a        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":122236082}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000291c        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":122920430}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002920        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":123971391}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x0000291e        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702017866,"timestampNanos":124041422}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002922        connection.go:416] \u003c- FlushFile (inode 4, PID 1809725)\n","timestampSeconds":1702017866,"timestampNanos":124142730}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"gcs: Req              0x9: \u003c- StatObject(\".bfuse/proj/scratch/gupttaru/file0.txt\")\n","timestampSeconds":1702017866,"timestampNanos":127842104}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"gcs: Req              0x9: -\u003e StatObject(\".bfuse/proj/scratch/gupttaru/file0.txt\") (140.90789ms): OK\n","timestampSeconds":1702017866,"timestampNanos":268764912}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"gcs: Req              0xa: \u003c- CreateObject(\".bfuse/proj/scratch/gupttaru/file0.txt\")\n","timestampSeconds":1702017866,"timestampNanos":268833249}

The logs stop being generated at this point, i.e. at CreateObject, even though the cp command hasn't been completed. On the bucket, I just a see an object with 0 bytes present.

Doing ls from a separate terminal is also stuck, and the last GCSFUSE log statement is the following:

{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x00002954        connection.go:416] \u003c- LookUpInode (parent 3, name \"file0.txt\", PID 1820773)\n","timestampSeconds":1702018841,"timestampNanos":720568546}

System (please complete the following information):

OS: RHEL8 (version 8.9)
Platform: On-premise machine.
Version: gcsfuse version 1.1.0 (Go version go1.20.4)

The text was updated successfully, but these errors were encountered:

ashmeenkaur · 2023-12-08T10:09:16Z

Hi @tarun360,
When dealing with large files, GCSFuse may require a significant amount of time to complete the upload. For instance, writing a 6GB file can take around 15 minutes. During this process, ls calls may appear to be stuck, but other read/write operations within the directory should remain functional.

Could you please confirm if GCSFuse is truly stuck or simply experiencing a longer upload duration due to the file size?

Thanks!

tarun360 · 2023-12-08T13:37:00Z

@ashmeenkaur Thanks for the reply.

Could you please confirm if GCSFuse is truly stuck or simply experiencing a longer upload duration due to the file size?

Indeed, I think its truly stuck. I left it overnight to check how much time it would take, however, even after 12 hours it was still stuck.

writing a 6GB file can take around 15 minutes.

Uploading the same file via gsutil takes less than 1 minute (without using any type of parallelism like parallel composite uploads). Is it expected, that a file which takes less than 1 minute to upload by gsutil CLI, will take 15 minutes on writing via GCSFUSE? Could you please provide more details on this, as I wasn't aware of this performance difference, and I couldn't find any reference to this in the documentation. Likewise, are reads from GCSFUSE expected to be equally slow as compared to downloading files via gsutil?

raj-prince · 2023-12-10T20:26:58Z

@tarun360 we don't have clear performance comparison with gsutil right now.

Coming to the issue, we are not able to reproduce this at our end. It would be easier for us to debug if you could provide more information like bucket_name. This will help us in getting the GCS server logs.

I would suggest:
(1) Try reproducing the issue with the latest version of gcsfuse.
(2) Please provide bucket_name via creating an internal customer support ticket.

Regards,
Prince Kumar.

raj-prince · 2023-12-11T19:18:14Z

@tarun360 Gentle reminder!

Could you please create a support ticket - https://support.cloud.google.com/portal/ ?

Thanks!

tarun360 · 2023-12-12T06:34:44Z

Hi @raj-prince , thanks for the reply.

I will update here after logging a support ticket. The project in which the bucket is doesn't allow logging a support ticket, so I need to get it changed via cloud admin. I am trying to get this expedited.

tarun360 · 2023-12-13T12:24:50Z

Hi @raj-prince , @ashmeenkaur

I tried the same operation today, and for some reason the behavior is different. Instead of getting stuck, it throws some timeout error:

WriteFile (inode 4, PID 0, handle 0, offset 5232394240, 1048576 bytes)\n","timestampSeconds":1702467722,"timestampNanos":6189 85658}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027e2        connection.go:416] \u003c-  WriteFile (inode 4, PID 0, handle 0, offset 5233442816, 1048576 bytes)\n","timestampSeconds":1702467722,"timestampNanos":6194 70316}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027e0        connection.go:498] -\u003e  OK ()\n","timestampSeconds":1702467722,"timestampNanos":619510992}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027e4        connection.go:416] \u003c-  WriteFile (inode 4, PID 0, handle 0, offset 5234491392, 1048576 bytes)\n","timestampSeconds":1702467722,"timestampNanos":6201 00741}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027e2        connection.go:498] -\u003e  OK ()\n","timestampSeconds":1702467722,"timestampNanos":620195750}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027e4        connection.go:498] -\u003e  OK ()\n","timestampSeconds":1702467722,"timestampNanos":620580663}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027e6        connection.go:416] \u003c-  WriteFile (inode 4, PID 0, handle 0, offset 5235539968, 1048576 bytes)\n","timestampSeconds":1702467722,"timestampNanos":6207 65955}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027e8        connection.go:416] \u003c-  WriteFile (inode 4, PID 0, handle 0, offset 5236588544, 1048576 bytes)\n","timestampSeconds":1702467722,"timestampNanos":6212 44026}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027e6        connection.go:498] -\u003e  OK ()\n","timestampSeconds":1702467722,"timestampNanos":621300854}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027ea        connection.go:416] \u003c-  WriteFile (inode 4, PID 0, handle 0, offset 5237637120, 1048576 bytes)\n","timestampSeconds":1702467722,"timestampNanos":6218 60159}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027e8        connection.go:498] -\u003e  OK ()\n","timestampSeconds":1702467722,"timestampNanos":621883111}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027ea        connection.go:498] -\u003e  OK ()\n","timestampSeconds":1702467722,"timestampNanos":622295194}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027ec        connection.go:416] \u003c-  WriteFile (inode 4, PID 0, handle 0, offset 5238685696, 1048576 bytes)\n","timestampSeconds":1702467722,"timestampNanos":6223 87506}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027ec        connection.go:498] -\u003e  OK ()\n","timestampSeconds":1702467722,"timestampNanos":622860002}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027ee        connection.go:416] \u003c-  WriteFile (inode 4, PID 0, handle 0, offset 5239734272, 1048576 bytes)\n","timestampSeconds":1702467722,"timestampNanos":6229 93387}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027f0        connection.go:416] \u003c-  WriteFile (inode 4, PID 0, handle 0, offset 5240782848, 1048576 bytes)\n","timestampSeconds":1702467722,"timestampNanos":6234 35009}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027ee        connection.go:498] -\u003e  OK ()\n","timestampSeconds":1702467722,"timestampNanos":623473372}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027f2        connection.go:416] \u003c-  WriteFile (inode 4, PID 0, handle 0, offset 5241831424, 1048576 bytes)\n","timestampSeconds":1702467722,"timestampNanos":6240 33831}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027f4        connection.go:416] \u003c-  SetInodeAttributes (inode 4, PID 2649659, mtime 2023-12-13 06:42:02.60670602 -0500 EST)\n","timestampSeconds":1702467722,"tim estampNanos":624092436}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027f0        connection.go:498] -\u003e  OK ()\n","timestampSeconds":1702467722,"timestampNanos":624123551}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027f2        connection.go:498] -\u003e  OK ()\n","timestampSeconds":1702467722,"timestampNanos":624559145}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027f4        connection.go:498] -\u003e  OK ()\n","timestampSeconds":1702467722,"timestampNanos":624610152}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027f6        connection.go:416] \u003c-  FlushFile (inode 4, PID 2649659)\n","timestampSeconds":1702467722,"timestampNanos":624702607}


{"name":"root","levelname":"INFO","severity":"INFO","message":"Starting a garbage collection run.\n","timestampSeconds":1702468306,"timestampNanos":807542532}
{"name":"root","levelname":"INFO","severity":"INFO","message":"Garbage collection succeeded after deleted 0 objects in 132.741948ms.\n","timestampSeconds":1702468306,"timestampNanos":940426931}
{"name":"root","levelname":"ERROR","severity":"ERROR","message":"FlushFile: connection timed out, FileInode.Sync: SyncObject: Create: CreateObject: error in io.Copy: Post \"https://storage.googleapis.com/upload/storage/v1/b/<TRIMMED BY ME>": write tcp <TRIMMED BY ME>-\u003e<TRIMMED BY ME>: write: connection timed out\n","timestampSeconds":1702468360,"timestampNanos":820447859}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027f6        connection.go:500] -\u003e Error: \"connection timed out\"\n","timestampSeconds":1702468360,"timestampNanos":820604536}
{"name":"root","levelname":"ERROR","severity":"ERROR","message":"fuse: *fuseops.FlushFileOp error: connection timed out\n","timestampSeconds":1702468360,"timestampNanos":820630051}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027f8        connection.go:416] \u003c- ReleaseFileHandle (PID 0)\n","timestampSeconds":1702468360,"timestampNanos":820850891}
{"name":"root","levelname":"DEBUG","severity":"DEBUG","message":"fuse_debug: Op 0x000027f8        connection.go:498] -\u003e OK ()\n","timestampSeconds":1702468360,"timestampNanos":820984966}

$ time cp /path1/file0.txt /path2/file0.txt
cp: failed to close '/path2/file0.txt': Connection timed out

real    10m9.015s
user    0m0.011s
sys     0m5.883s

The above error is again happening ONLY for relatively large files. The one I tried above is ~5GB.

The closest issue I found slightly resembling the above error is this, although this issue is marked as fixed.

Given this new finding, do you have any ideas about what could be going wrong with my setup here?

raj-prince · 2023-12-15T03:34:13Z

All these error signals some issue with internet connection. As we have tried this our side, which is working fine.

Could you please check VM's internet connection and try again?
Also, I haven't received any internal ticket yet. Did you file the ticket?

Thanks,
Prince Kumar.

tarun360 · 2023-12-15T07:31:13Z

Could you please check VM's internet connection and try again?

The network connection is fine since I am able to read/write the same file using gsutil.

Also, I am trying to read/write from a on-prem machine to bucket, not from a cloud VM. The network connection between on-prem machine and GCP seems fine, since I am able to read/write the same file using gsutil.

Also, I haven't received any internal ticket yet. Did you file the ticket?

Not yet. Due to holiday season, approvers are on leave. I'll update here once this is done.

Thanks,
Tarun

tarun360 · 2023-12-15T12:34:04Z

I created Google Cloud Support #48565270: GCSFUSE hanging while trying to write large files (~6GB)

sethiay · 2023-12-15T16:45:55Z

Thanks for reaching out via Google cloud support. Further communication can be done via the support.

ashmeenkaur · 2023-12-19T12:25:23Z

Is this issue always reproducible, @tarun360?

tarun360 · 2023-12-19T13:35:15Z

@ashmeenkaur Yes, its always reproducible.

tarun360 · 2023-12-20T13:24:35Z

Hi, I have added some more details on the support ticket, which I believe could be helpful in debugging this issue. I hope you have access to the support ticket.

The problem seems related to perhaps GCSFUSE and proxy server.

tarun360 · 2023-12-20T15:19:07Z

When creating a storage client in this function, in Lines 48-58 we are adding option.WithHTTPClient, so this means that the environment variable https_proxy will not be respected directly by the StorageClient created in line 71 by default.

For that, perhaps in the CreateHTTPClient function, we probably need something like

transport := &http.Transport{
    Proxy: http.ProxyFromEnvironment,
    // other stuff ...
}

or clone the default transport and set your options in that transport (ref: https://stackoverflow.com/a/51848441/13416549).

I may be wrong here since I don't know the whole code, but am I missing anything in above logic?

tarun360 · 2023-12-21T06:52:42Z

After making the above change, I built gscfuse from source code, and it seems that the above issue I was facing has gone away.

ashmeenkaur · 2023-12-21T07:13:32Z

After making the above change, I built gscfuse from source code, and it seems that the above issue I was facing has gone away.

Thanks for looking into this @tarun360. Did it also solve the 6GB file upload issue or just write: connection timed out issue?

tarun360 · 2023-12-21T07:20:29Z

Did it also solve the 6GB file upload issue or just write: connection timed out issue?

Without the above change, the data upload traffic wasn't being routed via the proxy server (verified by checking the proxy-server logs), so I suppose that might be related to some sort of rate-limiting by the firewalls for public endpoints, which lead to write: connection timed out.

With the above change, I am able to upload the 6GB file, so that probably implies that data is being routed via the proxy server (although I am yet to check proxy logs, but I am pretty sure that is the case since there's no other explaination of why this change would fix this issue).

gargnitingoogle · 2023-12-26T08:22:18Z

@tarun360 the fix for this is in #1562 and is currently planned to release in Jan 2024 release, on Jan 22.

ashmeenkaur · 2024-01-22T11:22:30Z

Hi @tarun360,
GCSFuse v1.4.1 fixes this issue. Please feel free to re-open incase you still run into any failure.

Thanks

tarun360 added p1 P1 question Customer Issue: question about how to use tool labels Dec 8, 2023

ashmeenkaur added the pending customer action label Dec 12, 2023

ashmeenkaur added pending customer action and removed pending customer action labels Dec 12, 2023

github-actions bot removed the pending customer action label Dec 14, 2023

ashmeenkaur added p2 P2 and removed p1 P1 labels Dec 22, 2023

ashmeenkaur mentioned this issue Dec 25, 2023

Http proxy fix #1562

Merged

ashmeenkaur self-assigned this Jan 4, 2024

ashmeenkaur removed question Customer Issue: question about how to use tool p2 P2 labels Jan 4, 2024

ashmeenkaur added next release p2 P2 labels Jan 4, 2024

ashmeenkaur closed this as completed Jan 22, 2024

ashmeenkaur mentioned this issue Jan 22, 2024

No such host while creating mounts #1633

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GCSFUSE hanging while trying to write large files (~6GB) #1541

GCSFUSE hanging while trying to write large files (~6GB) #1541

tarun360 commented Dec 8, 2023 •

edited

ashmeenkaur commented Dec 8, 2023

tarun360 commented Dec 8, 2023 •

edited

raj-prince commented Dec 10, 2023 •

edited

raj-prince commented Dec 11, 2023

tarun360 commented Dec 12, 2023

tarun360 commented Dec 13, 2023

raj-prince commented Dec 15, 2023

tarun360 commented Dec 15, 2023

tarun360 commented Dec 15, 2023

sethiay commented Dec 15, 2023

ashmeenkaur commented Dec 19, 2023

tarun360 commented Dec 19, 2023

tarun360 commented Dec 20, 2023 •

edited

tarun360 commented Dec 20, 2023

tarun360 commented Dec 21, 2023

ashmeenkaur commented Dec 21, 2023

tarun360 commented Dec 21, 2023 •

edited

gargnitingoogle commented Dec 26, 2023 •

edited

ashmeenkaur commented Jan 22, 2024

GCSFUSE hanging while trying to write large files (~6GB) #1541

GCSFUSE hanging while trying to write large files (~6GB) #1541

Comments

tarun360 commented Dec 8, 2023 • edited

ashmeenkaur commented Dec 8, 2023

tarun360 commented Dec 8, 2023 • edited

raj-prince commented Dec 10, 2023 • edited

raj-prince commented Dec 11, 2023

tarun360 commented Dec 12, 2023

tarun360 commented Dec 13, 2023

raj-prince commented Dec 15, 2023

tarun360 commented Dec 15, 2023

tarun360 commented Dec 15, 2023

sethiay commented Dec 15, 2023

ashmeenkaur commented Dec 19, 2023

tarun360 commented Dec 19, 2023

tarun360 commented Dec 20, 2023 • edited

tarun360 commented Dec 20, 2023

tarun360 commented Dec 21, 2023

ashmeenkaur commented Dec 21, 2023

tarun360 commented Dec 21, 2023 • edited

gargnitingoogle commented Dec 26, 2023 • edited

ashmeenkaur commented Jan 22, 2024

tarun360 commented Dec 8, 2023 •

edited

tarun360 commented Dec 8, 2023 •

edited

raj-prince commented Dec 10, 2023 •

edited

tarun360 commented Dec 20, 2023 •

edited

tarun360 commented Dec 21, 2023 •

edited

gargnitingoogle commented Dec 26, 2023 •

edited