[query] rare Google Cloud Storage error

[danking]

What happened?

https://batch.hail.is/batches/8043496/jobs/22010 (wlu, all-by-aou)

It seems that, on close, we encountered an exception. Google searches have not revealed much of use. There's this stack overflow issue where the fix was to use a different part of the API. We're not using createFrom though. I can't even find the source code for JsonResumableSessionFailureScenario.

Version

0.2.124

Relevant log output

2023-09-24 01:58:16.721 JVMEntryway: INFO: is.hail.JVMEntryway received arguments:
2023-09-24 01:58:16.721 JVMEntryway: INFO: 0: /hail-jars/gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar
2023-09-24 01:58:16.721 JVMEntryway: INFO: 1: is.hail.backend.service.Main
2023-09-24 01:58:16.721 JVMEntryway: INFO: 2: /batch/8cca2fb0e9764b6195f85b899fb76986
2023-09-24 01:58:16.721 JVMEntryway: INFO: 3: /batch/8cca2fb0e9764b6195f85b899fb76986/log
2023-09-24 01:58:16.721 JVMEntryway: INFO: 4: gs://hail-query-ger0g/jars/13536b531342a263b24a7165bfeec7bd02723e4b.jar
2023-09-24 01:58:16.721 JVMEntryway: INFO: 5: worker
2023-09-24 01:58:16.721 JVMEntryway: INFO: 6: gs://aou_tmp/parallelizeAndComputeWithIndex/OLkY5pgCTBWt2Yw4iCp6WsR2N5drFQmMiQJa7wSE_ik=
2023-09-24 01:58:16.721 JVMEntryway: INFO: 7: 9571
2023-09-24 01:58:16.721 JVMEntryway: INFO: 8: 12185
2023-09-24 01:58:16.721 JVMEntryway: INFO: Yielding control to the QoB Job.
2023-09-24 01:58:16.722 Worker$: INFO: is.hail.backend.service.Worker 13536b531342a263b24a7165bfeec7bd02723e4b
2023-09-24 01:58:16.722 Worker$: INFO: running job 9571/12185 at root gs://aou_tmp/parallelizeAndComputeWithIndex/OLkY5pgCTBWt2Yw4iCp6WsR2N5drFQmMiQJa7wSE_ik= with scratch directory '/batch/8cca2fb0e9764b6195f85b899fb76986'
2023-09-24 01:58:16.729 GoogleStorageFS$: INFO: Initializing google storage client from service account key
2023-09-24 01:58:17.061 WorkerTimer$: INFO: readInputs took 338.458743 ms.
2023-09-24 01:58:17.061 : INFO: RegionPool: initialized for thread 10: pool-2-thread-2
2023-09-24 01:58:17.096 : INFO: RegionPool: REPORT_THRESHOLD: 265.0K allocated (201.0K blocks / 64.0K chunks), regions.size = 5, 0 current java objects, thread 10: pool-2-thread-2
2023-09-24 01:58:17.707 : INFO: RegionPool: REPORT_THRESHOLD: 521.0K allocated (457.0K blocks / 64.0K chunks), regions.size = 9, 0 current java objects, thread 10: pool-2-thread-2
2023-09-24 01:58:18.609 : INFO: RegionPool: REPORT_THRESHOLD: 1.1M allocated (698.0K blocks / 410.0K chunks), regions.size = 19, 0 current java objects, thread 10: pool-2-thread-2
2023-09-24 01:58:19.984 : INFO: RegionPool: REPORT_THRESHOLD: 2.0M allocated (1.0M blocks / 1010.0K chunks), regions.size = 19, 0 current java objects, thread 10: pool-2-thread-2
2023-09-24 01:58:24.240 : INFO: RegionPool: REPORT_THRESHOLD: 4.3M allocated (2.2M blocks / 2.1M chunks), regions.size = 19, 0 current java objects, thread 10: pool-2-thread-2
2023-09-24 01:58:24.240 GoogleStorageFS$: INFO: createNoCompression: gs://aou_tmp/tmp/hail/icullIwHC8dQXtq8JU2uDW/aggregate_intermediates/-ntpjdAQ9sKaR8lK26cV0p5790a4d87-9035-41ae-afc6-326f710d9a89
2023-09-24 01:58:24.305 GoogleStorageFS$: INFO: close: gs://aou_tmp/tmp/hail/icullIwHC8dQXtq8JU2uDW/aggregate_intermediates/-ntpjdAQ9sKaR8lK26cV0p5790a4d87-9035-41ae-afc6-326f710d9a89
2023-09-24 01:58:51.513 : INFO: TaskReport: stage=0, partition=9571, attempt=0, peakBytes=4507648, peakBytesReadable=4.30 MiB, chunks requested=51, cache hits=0
2023-09-24 01:58:51.513 : INFO: RegionPool: FREE: 4.3M allocated (2.2M blocks / 2.1M chunks), regions.size = 19, 0 current java objects, thread 10: pool-2-thread-2
2023-09-24 01:58:51.515 JVMEntryway: ERROR: QoB Job threw an exception.
java.lang.reflect.InvocationTargetException: null
	at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) ~[?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_382]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_382]
	at is.hail.JVMEntryway$1.run(JVMEntryway.java:119) ~[jvm-entryway.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_382]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_382]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_382]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_382]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_382]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_382]
	at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_382]
Caused by: is.hail.relocated.com.google.cloud.storage.StorageException: Missing Range header in response
	|> PUT https://storage.googleapis.com/upload/storage/v1/b/aou_tmp/o?name=tmp/hail/icullIwHC8dQXtq8JU2uDW/aggregate_intermediates/-ntpjdAQ9sKaR8lK26cV0p5790a4d87-9035-41ae-afc6-326f710d9a89&uploadType=resumable&upload_id=ADPycdtl5JSqwvftT4W190_-ueC032I_oZcwLAlVVMFkqp06W4eY8b-XMwf8DeT7If9I7uIgmI_PLCuFsExsT0aEh2b4FrHtAiUktumQbvgl1U0icw
	|> content-range: bytes */*
	|  
	|< HTTP/1.1 308 Resume Incomplete
	|< content-length: 0
	|< content-type: text/plain; charset=utf-8
	|< x-guploader-uploadid: ADPycdtl5JSqwvftT4W190_-ueC032I_oZcwLAlVVMFkqp06W4eY8b-XMwf8DeT7If9I7uIgmI_PLCuFsExsT0aEh2b4FrHtAiUktumQbvgl1U0icw
	|  
	at is.hail.relocated.com.google.cloud.storage.JsonResumableSessionFailureScenario.toStorageException(JsonResumableSessionFailureScenario.java:185) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.JsonResumableSessionFailureScenario.toStorageException(JsonResumableSessionFailureScenario.java:117) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.JsonResumableSessionFailureScenario.toStorageException(JsonResumableSessionFailureScenario.java:98) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.JsonResumableSessionQueryTask.call(JsonResumableSessionQueryTask.java:100) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.JsonResumableSession.query(JsonResumableSession.java:57) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.JsonResumableSession.lambda$put$0(JsonResumableSession.java:73) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.Retrying.lambda$run$0(Retrying.java:102) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.Retrying.run(Retrying.java:99) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.JsonResumableSession.put(JsonResumableSession.java:68) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.ApiaryUnbufferedWritableByteChannel.internalWrite(ApiaryUnbufferedWritableByteChannel.java:114) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.ApiaryUnbufferedWritableByteChannel.writeAndClose(ApiaryUnbufferedWritableByteChannel.java:65) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.UnbufferedWritableByteChannelSession$UnbufferedWritableByteChannel.writeAndClose(UnbufferedWritableByteChannelSession.java:40) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.DefaultBufferedWritableByteChannel.close(DefaultBufferedWritableByteChannel.java:166) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.StorageByteChannels$SynchronizedBufferedWritableByteChannel.close(StorageByteChannels.java:119) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.StorageException.wrapIOException(StorageException.java:179) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.BaseStorageWriteChannel.close(BaseStorageWriteChannel.java:84) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.io.fs.GoogleStorageFS$$anon$2.$anonfun$close$2(GoogleStorageFS.scala:326) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.io.fs.GoogleStorageFS$$anon$2.doHandlingRequesterPays(GoogleStorageFS.scala:296) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.io.fs.GoogleStorageFS$$anon$2.$anonfun$close$1(GoogleStorageFS.scala:326) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?]
	at is.hail.services.package$.retryTransientErrors(package.scala:182) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.io.fs.GoogleStorageFS$$anon$2.close(GoogleStorageFS.scala:324) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at java.io.FilterOutputStream.close(FilterOutputStream.java:159) ~[?:1.8.0_382]
	at __C1867collect_distributed_array_table_aggregate.apply(Unknown Source) ~[?:?]
	at __C1867collect_distributed_array_table_aggregate.apply(Unknown Source) ~[?:?]
	at is.hail.backend.BackendUtils.$anonfun$collectDArray$16(BackendUtils.scala:91) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.utils.package$.using(package.scala:637) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.annotations.RegionPool.scopedRegion(RegionPool.scala:162) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.backend.BackendUtils.$anonfun$collectDArray$15(BackendUtils.scala:90) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.backend.service.Worker$.$anonfun$main$12(Worker.scala:167) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?]
	at is.hail.services.package$.retryTransientErrors(package.scala:182) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.backend.service.Worker$.$anonfun$main$11(Worker.scala:166) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.backend.service.Worker$.$anonfun$main$11$adapted(Worker.scala:164) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.utils.package$.using(package.scala:637) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.backend.service.Worker$.main(Worker.scala:164) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.backend.service.Main$.main(Main.scala:14) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.backend.service.Main.main(Main.scala) ~[gs:__hail-query-ger0g_jars_13536b531342a263b24a7165bfeec7bd02723e4b.jar.jar:0.0.1-SNAPSHOT]
	... 11 more

[danking]

I observed this in my namespace while testing fixes for a different transient error.
https://internal.hail.is/dking/batch/batches/5/jobs/7034 (this URL will obviously not last long).

The worker actually successfully wrote the correct output to GCS, but it received this error anyway which caused the job to fail. Since QoB checks for the results files first, it didn't even notice that the worker job had failed. That seems not great. We should probably fail if an worker job fails, even if we find output because that output could be corrupt.

(base) dking@wm28c-761 hail % hexdump -C foo                                                                                                                          
00000000  01 82 00 00 00 7e 00 00  00 67 73 3a 2f 2f 31 2d  |.....~...gs://1-|
00000010  64 61 79 2f 74 6d 70 2f  68 61 69 6c 2f 54 53 4f  |day/tmp/hail/TSO|
00000020  66 4f 72 67 5a 55 6d 62  56 69 78 6e 52 69 4b 57  |fOrgZUmbVixnRiKW|
00000030  51 46 42 2f 61 67 67 72  65 67 61 74 65 5f 69 6e  |QFB/aggregate_in|
00000040  74 65 72 6d 65 64 69 61  74 65 73 2f 2d 50 74 33  |termediates/-Pt3|
00000050  67 4e 74 51 57 35 57 6f  42 64 43 54 44 50 51 69  |gNtQW5WoBdCTDPQi|
00000060  77 48 64 61 39 63 32 36  35 66 32 2d 66 62 64 38  |wHda9c265f2-fbd8|
00000070  2d 34 66 31 62 2d 62 63  64 65 2d 66 62 66 32 39  |-4f1b-bcde-fbf29|
00000080  31 38 30 63 33 34 37 00  00 00 00                 |180c347....|
0000008b

code:

In [1]: import hail as hl
   ...: import gnomad.utils.sparse_mt
   ...: 
   ...: 
   ...: tmp_dir = 'gs://danking/tmp/'
   ...: vds_file = 'gs://neale-bge/bge-wave-1.vds'
   ...: out = 'gs://danking/foo.vcf.bgz'
   ...: 
   ...: vds = hl.vds.read_vds(vds_file)
   ...: mt = hl.vds.to_dense_mt(vds)
   ...: t = gnomad.utils.sparse_mt.default_compute_info(mt)
   ...: t = t.annotate(info=t.info.drop('AS_SB_TABLE'))
   ...: t = t.annotate(info = t.info.drop(
   ...:     'AS_QUALapprox', 'AS_VarDP', 'AS_SOR', 'AC_raw', 'AC', 'AS_SB'
   ...: ))
   ...: t = t.drop('AS_lowqual')
   ...: 
   ...: hl.methods.export_vcf(dataset = t, output = out, tabix = True)

worker failure:

2023-09-27 16:43:10.389 JVMEntryway: INFO: is.hail.JVMEntryway received arguments:
2023-09-27 16:43:10.389 JVMEntryway: INFO: 0: /hail-jars/gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar
2023-09-27 16:43:10.389 JVMEntryway: INFO: 1: is.hail.backend.service.Main
2023-09-27 16:43:10.390 JVMEntryway: INFO: 2: /batch/83e7aee9e9244f6884b8a84ea81b4c7a
2023-09-27 16:43:10.390 JVMEntryway: INFO: 3: /batch/83e7aee9e9244f6884b8a84ea81b4c7a/log
2023-09-27 16:43:10.390 JVMEntryway: INFO: 4: gs://hail-test-ezlis/dking/jars/ch4g3zvqceyo/09526a168d57dac1a26f8caa4ab49593931ed2ef.jar
2023-09-27 16:43:10.390 JVMEntryway: INFO: 5: worker
2023-09-27 16:43:10.390 JVMEntryway: INFO: 6: gs://1-day/parallelizeAndComputeWithIndex/al3OJfYZMMNoi9F2jvcAf0jBirVTayRVqro03dnIa1g=
2023-09-27 16:43:10.390 JVMEntryway: INFO: 7: 7028
2023-09-27 16:43:10.390 JVMEntryway: INFO: 8: 9060
2023-09-27 16:43:10.390 JVMEntryway: INFO: Yielding control to the QoB Job.
2023-09-27 16:43:10.393 Worker$: INFO: is.hail.backend.service.Worker 09526a168d57dac1a26f8caa4ab49593931ed2ef
2023-09-27 16:43:10.394 Worker$: INFO: running job 7028/9060 at root gs://1-day/parallelizeAndComputeWithIndex/al3OJfYZMMNoi9F2jvcAf0jBirVTayRVqro03dnIa1g= with scratch directory '/batch/83e7aee9e9244f6884b8a84ea81b4c7a'
2023-09-27 16:43:10.398 GoogleStorageFS$: INFO: Initializing google storage client from service account key
2023-09-27 16:43:10.779 WorkerTimer$: INFO: readInputs took 384.743327 ms.
2023-09-27 16:43:10.779 : INFO: RegionPool: initialized for thread 10: pool-2-thread-2
2023-09-27 16:43:10.787 : INFO: RegionPool: REPORT_THRESHOLD: 2.2M allocated (192.0K blocks / 2.0M chunks), regions.size = 3, 0 current java objects, thread 10: pool-2-thread-2
2023-09-27 16:43:10.794 : INFO: RegionPool: REPORT_THRESHOLD: 2.3M allocated (192.0K blocks / 2.1M chunks), regions.size = 3, 0 current java objects, thread 10: pool-2-thread-2
2023-09-27 16:43:10.794 : INFO: RegionPool: REPORT_THRESHOLD: 2.3M allocated (256.0K blocks / 2.1M chunks), regions.size = 4, 0 current java objects, thread 10: pool-2-thread-2
2023-09-27 16:43:10.794 : INFO: RegionPool: REPORT_THRESHOLD: 2.4M allocated (320.0K blocks / 2.1M chunks), regions.size = 5, 0 current java objects, thread 10: pool-2-thread-2
2023-09-27 16:43:11.286 : INFO: RegionPool: REPORT_THRESHOLD: 28.8M allocated (576.0K blocks / 28.2M chunks), regions.size = 9, 0 current java objects, thread 10: pool-2-thread-2
2023-09-27 16:43:11.657 : INFO: RegionPool: REPORT_THRESHOLD: 30.8M allocated (576.0K blocks / 30.2M chunks), regions.size = 9, 0 current java objects, thread 10: pool-2-thread-2
2023-09-27 16:43:11.683 : INFO: RegionPool: REPORT_THRESHOLD: 32.8M allocated (576.0K blocks / 32.2M chunks), regions.size = 9, 0 current java objects, thread 10: pool-2-thread-2
2023-09-27 16:43:11.709 : INFO: RegionPool: REPORT_THRESHOLD: 32.8M allocated (576.0K blocks / 32.3M chunks), regions.size = 9, 0 current java objects, thread 10: pool-2-thread-2
2023-09-27 16:44:22.426 GoogleStorageFS$: INFO: createNoCompression: gs://1-day/tmp/hail/TSOfOrgZUmbVixnRiKWQFB/aggregate_intermediates/-Pt3gNtQW5WoBdCTDPQiwHda9c265f2-fbd8-4f1b-bcde-fbf29180c347
2023-09-27 16:44:22.495 GoogleStorageFS$: INFO: close: gs://1-day/tmp/hail/TSOfOrgZUmbVixnRiKWQFB/aggregate_intermediates/-Pt3gNtQW5WoBdCTDPQiwHda9c265f2-fbd8-4f1b-bcde-fbf29180c347
2023-09-27 16:44:22.620 GoogleStorageFS$: INFO: closed: gs://1-day/tmp/hail/TSOfOrgZUmbVixnRiKWQFB/aggregate_intermediates/-Pt3gNtQW5WoBdCTDPQiwHda9c265f2-fbd8-4f1b-bcde-fbf29180c347
2023-09-27 16:44:22.621 : INFO: TaskReport: stage=0, partition=7028, attempt=0, peakBytes=62266032, peakBytesReadable=59.38 MiB, chunks requested=72126, cache hits=72121
2023-09-27 16:44:22.622 : INFO: RegionPool: FREE: 59.4M allocated (25.2M blocks / 34.2M chunks), regions.size = 11, 0 current java objects, thread 10: pool-2-thread-2
2023-09-27 16:44:22.623 WorkerTimer$: INFO: executeFunction took 71843.446957 ms.
2023-09-27 16:44:22.623 GoogleStorageFS$: INFO: createNoCompression: gs://1-day/parallelizeAndComputeWithIndex/al3OJfYZMMNoi9F2jvcAf0jBirVTayRVqro03dnIa1g=/result.7028
2023-09-27 16:44:22.668 GoogleStorageFS$: INFO: close: gs://1-day/parallelizeAndComputeWithIndex/al3OJfYZMMNoi9F2jvcAf0jBirVTayRVqro03dnIa1g=/result.7028
2023-09-27 16:44:33.788 JVMEntryway: ERROR: QoB Job threw an exception.
java.lang.reflect.InvocationTargetException: null
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_382]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_382]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_382]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_382]
	at is.hail.JVMEntryway$1.run(JVMEntryway.java:119) ~[jvm-entryway.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_382]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_382]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_382]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_382]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_382]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_382]
	at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_382]
Caused by: is.hail.relocated.com.google.cloud.storage.StorageException: Missing Range header in response
	|> PUT https://storage.googleapis.com/upload/storage/v1/b/1-day/o?name=parallelizeAndComputeWithIndex/al3OJfYZMMNoi9F2jvcAf0jBirVTayRVqro03dnIa1g%3D/result.7028&uploadType=resumable&upload_id=ADPycdv7y3A6GjTh6Kv7vrUu2ap2Kv0peLVWsVTAghIs7RCZk9X3fI1BDkeHag1cd9g3etP2sS4f13bN6iJPU_sbnRnyRE91VPtjUpuYLPqmOq13
	|> content-range: bytes */*
	|  
	|< HTTP/1.1 308 Resume Incomplete
	|< content-length: 0
	|< content-type: text/plain; charset=utf-8
	|< x-guploader-uploadid: ADPycdv7y3A6GjTh6Kv7vrUu2ap2Kv0peLVWsVTAghIs7RCZk9X3fI1BDkeHag1cd9g3etP2sS4f13bN6iJPU_sbnRnyRE91VPtjUpuYLPqmOq13
	|  
	at is.hail.relocated.com.google.cloud.storage.JsonResumableSessionFailureScenario.toStorageException(JsonResumableSessionFailureScenario.java:185) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.JsonResumableSessionFailureScenario.toStorageException(JsonResumableSessionFailureScenario.java:117) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.JsonResumableSessionFailureScenario.toStorageException(JsonResumableSessionFailureScenario.java:98) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.JsonResumableSessionQueryTask.call(JsonResumableSessionQueryTask.java:100) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.JsonResumableSession.query(JsonResumableSession.java:57) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.JsonResumableSession.lambda$put$0(JsonResumableSession.java:73) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.Retrying.lambda$run$0(Retrying.java:102) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.Retrying.run(Retrying.java:99) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.JsonResumableSession.put(JsonResumableSession.java:68) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.ApiaryUnbufferedWritableByteChannel.internalWrite(ApiaryUnbufferedWritableByteChannel.java:114) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.ApiaryUnbufferedWritableByteChannel.writeAndClose(ApiaryUnbufferedWritableByteChannel.java:65) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.UnbufferedWritableByteChannelSession$UnbufferedWritableByteChannel.writeAndClose(UnbufferedWritableByteChannelSession.java:40) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.DefaultBufferedWritableByteChannel.close(DefaultBufferedWritableByteChannel.java:166) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.StorageByteChannels$SynchronizedBufferedWritableByteChannel.close(StorageByteChannels.java:119) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.StorageException.wrapIOException(StorageException.java:179) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.relocated.com.google.cloud.storage.BaseStorageWriteChannel.close(BaseStorageWriteChannel.java:84) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.io.fs.GoogleStorageFS$$anon$2.$anonfun$close$2(GoogleStorageFS.scala:310) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.io.fs.GoogleStorageFS$$anon$2.doHandlingRequesterPays(GoogleStorageFS.scala:280) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.io.fs.GoogleStorageFS$$anon$2.$anonfun$close$1(GoogleStorageFS.scala:310) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?]
	at is.hail.services.package$.retryTransientErrors(package.scala:182) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.io.fs.GoogleStorageFS$$anon$2.close(GoogleStorageFS.scala:308) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at java.io.FilterOutputStream.close(FilterOutputStream.java:159) ~[?:1.8.0_382]
	at is.hail.utils.package$.using(package.scala:677) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.io.fs.FS.writePDOS(FS.scala:441) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.io.fs.FS.writePDOS$(FS.scala:440) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.io.fs.RouterFS.writePDOS(RouterFS.scala:3) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.backend.service.Worker$.$anonfun$main$4(Worker.scala:124) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.backend.service.Worker$.$anonfun$main$4$adapted(Worker.scala:124) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.backend.service.Worker$.$anonfun$main$13(Worker.scala:178) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?]
	at is.hail.services.package$.retryTransientErrors(package.scala:182) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.backend.service.Worker$.main(Worker.scala:177) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.backend.service.Main$.main(Main.scala:14) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	at is.hail.backend.service.Main.main(Main.scala) ~[gs:__hail-test-ezlis_dking_jars_ch4g3zvqceyo_09526a168d57dac1a26f8caa4ab49593931ed2ef.jar.jar:0.0.1-SNAPSHOT]
	... 12 more

[danking]

I asked Google to please give us a bit more clarity on what this means: googleapis/java-storage#2232

[danking]

Ahah! This is a known issue with the version of the GCS client libs we're using.

1. Comment confirming its a known issue
2. PR which fixed it in main
3. Corresponding commit to main
4. Changelog entry specifically mentioning this issue.