Mirroring IT failures in `anvildev` due to 400 status from DRS

[achave11-ucsc]

… due to "Snapshot requires an x-user-project header for DRS access URL requests".

[ERROR] DRSStatusException: ('Unexpected response from https://jade.datarepo-dev.broadinstitute.org/ga4gh/drs/v1/objects/v2_4ffefed7-76a2-4807-8f3b-b4dde17ad3b5/access/gcp-us-central1*74089fe0-c732-40b0-b828-d7e091f0f54b', 400, b'{""msg"":""Snapshot requires an x-user-project header for DRS access URL requests"",""status_code"":400}')

Traceback (most recent call last):
  File ""/var/task/azul/chalice.py"", line 190, in patched_event_source_handler
    return old_handler(self_, event, context)
  File ""/var/task/chalice/app.py"", line 1761, in __call__
    return self.handler(event_obj)
  File ""/var/task/azul/indexer/mirror_controller.py"", line 67, in mirror
    self.mirror(event)
  File ""/var/task/azul/indexer/mirror_controller.py"", line 77, in mirror
    self._handle_events(event, action_handler)
  File ""/var/task/azul/indexer/action_controller.py"", line 50, in _handle_events
    message_handler(action)
  File ""/var/task/azul/indexer/mirror_controller.py"", line 75, in action_handler
    self.service(action.catalog).mirror(action)
  File ""/var/task/azul/indexer/mirror_service.py"", line 561, in mirror
    self._queue_actions(self._mirror(action))
  File ""/var/task/azul/indexer/mirror_service.py"", line 430, in _queue_actions
    return self._queues.send_messages(self._mirror_queue(),
  File ""/var/task/azul/queues.py"", line 210, in send_messages
    for batch in chunked(messages, self.batch_size):
  File ""/opt/python/more_itertools/recipes.py"", line 126, in take
    return list(islice(iterable, n))
  File ""/var/task/azul/indexer/mirror_service.py"", line 622, in _
    self._mirror_file(a.file)
  File ""/var/task/azul/indexer/mirror_service.py"", line 636, in _mirror_file
    file_content = self._download(file)
  File ""/var/task/azul/indexer/mirror_service.py"", line 757, in _download
    url = self._repository_url(file)
  File ""/var/task/azul/indexer/mirror_service.py"", line 752, in _repository_url
    access = object.get(AccessMethod.gs)
  File ""/var/task/azul/drs.py"", line 365, in get
    return self._get(access_method)
  File ""/var/task/azul/drs.py"", line 387, in _get
    return self._get_access(access_id, AccessMethod.https)
  File ""/var/task/azul/drs.py"", line 427, in _get_access
    raise DRSStatusException(url, response)

CloudWatch Logs Insights
region: us-east-1
log-group-names: /aws/lambda/azul-indexer-anvildev-mirror
data-sources:
facets:
start-time: 2026-04-24T02:25:42.888Z
end-time: 2026-04-24T02:27:36.523Z
query-string:

fields @timestamp, @message
| sort @timestamp asc
| limit 10000

---

@timestamp	@message
2026-04-24 02:26:26.777	START RequestId: bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed Version: 109
2026-04-24 02:26:26.780	[INFO] 2026-04-24T02:26:26.779Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.indexer.action_controller Worker handling message SQSFifoMessage(body={'catalog': 'anvil-it', 'operation_id': '529aa38c-b3ed-4826-a928-03a3a8387a66', 'source': {'id': '74089fe0-c732-40b0-b828-d7e091f0f54b', 'spec': 'tdr:bigquery:gcp:datarepo-dev-dd576076:ANVIL_CMG_Sample_1_20230225_ANV5_202512031111', 'prefix': '/1', 'type': 'azul.terra.TDRSourceRef'}, 'prefix': 'a', 'file': {'uuid': '00d3b050-30ec-4eb8-81e3-2a5c2bb9513f', 'name': '10137.merged.matefixed.sorted.markeddups.recal.bam', 'version': '2022-06-01T00:00:00.000000Z', 'drs_uri': 'drs://jade.datarepo-dev.broadinstitute.org/v2_4ffefed7-76a2-4807-8f3b-b4dde17ad3b5', 'size': 1840, 'content_type': None, 'md5': 'a5a2273ce9a6df96e3a0861a12a8a2ba', 'type': 'azul.plugins.metadata.anvil.AnvilFile'}, 'action': 'MirrorFileAction'}, attempts=3, id='cd432568-28c7-4d72-b64c-1d848a2b9453', group_id='f8a3068a-46ab-5a28-8844-892ea2007319', dedup_id='ff18308e-3b27-57cc-a9e8-1fb94efb416a')
2026-04-24 02:26:26.785	[INFO] 2026-04-24T02:26:26.785Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.boto3 s3.HeadObject: Making HEAD request to https://s3.amazonaws.com/edu-ucsc-gi-platform-anvil-dev-mirror-anvildev.us-east-1/_it/info/a5a2273ce9a6df96e3a0861a12a8a2ba.md5.json
2026-04-24 02:26:26.785	[INFO] 2026-04-24T02:26:26.785Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.boto3 s3.HeadObject: … without a request body
2026-04-24 02:26:26.860	[INFO] 2026-04-24T02:26:26.860Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.boto3 s3.HeadObject: … without a response body
2026-04-24 02:26:26.860	[INFO] 2026-04-24T02:26:26.860Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.boto3 s3.HeadObject: Got 404 response
2026-04-24 02:26:26.861	[INFO] 2026-04-24T02:26:26.861Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.boto3 s3.HeadObject: Making HEAD request to https://s3.amazonaws.com/edu-ucsc-gi-platform-anvil-dev-mirror-anvildev.us-east-1/_it/file/a5a2273ce9a6df96e3a0861a12a8a2ba.md5
2026-04-24 02:26:26.861	[INFO] 2026-04-24T02:26:26.861Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.boto3 s3.HeadObject: … without a request body
2026-04-24 02:26:26.869	[INFO] 2026-04-24T02:26:26.869Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.boto3 s3.HeadObject: Got 404 response
2026-04-24 02:26:26.869	[INFO] 2026-04-24T02:26:26.869Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.boto3 s3.HeadObject: … without a response body
2026-04-24 02:26:26.869	[INFO] 2026-04-24T02:26:26.869Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.indexer.mirror_service Mirroring file via standard upload: AnvilFile(uuid='00d3b050-30ec-4eb8-81e3-2a5c2bb9513f', name='10137.merged.matefixed.sorted.markeddups.recal.bam', version='2022-06-01T00:00:00.000000Z', drs_uri='drs://jade.datarepo-dev.broadinstitute.org/v2_4ffefed7-76a2-4807-8f3b-b4dde17ad3b5', size=1840, content_type=None, md5='a5a2273ce9a6df96e3a0861a12a8a2ba')
2026-04-24 02:26:26.870	[INFO] 2026-04-24T02:26:26.870Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.terra … without a request body
2026-04-24 02:26:26.870	[INFO] 2026-04-24T02:26:26.870Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.terra Making GET request to 'https://jade.datarepo-dev.broadinstitute.org/ga4gh/drs/v1/objects/v2_4ffefed7-76a2-4807-8f3b-b4dde17ad3b5'
2026-04-24 02:26:26.870	[INFO] 2026-04-24T02:26:26.870Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.terra … with request headers [('x-goog-api-client', 'cred-type/sa'), ('authorization', 'REDACTED')]
2026-04-24 02:26:28.087	[INFO] 2026-04-24T02:26:28.087Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.terra Got 200 response after 1.216s from GET to https://jade.datarepo-dev.broadinstitute.org/ga4gh/drs/v1/objects/v2_4ffefed7-76a2-4807-8f3b-b4dde17ad3b5
2026-04-24 02:26:28.087	[INFO] 2026-04-24T02:26:28.087Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.terra … with response headers HTTPHeaderDict({'Date': 'Fri, 24 Apr 2026 02:26:28 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'Access-Control-Allow-Headers': 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,Accept,Referer,X-App-Id,Origin', 'Access-Control-Allow-Methods': 'GET,POST,DELETE,PUT,PATCH,OPTIONS,HEAD', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains', 'Cache-Control': 'no-cache,no-store,must-revalidate', 'X-Request-ID': 'aKl3QvXe', 'Content-Type': 'application/json', 'Content-Length': '2495', 'Set-Cookie': 'JSESSIONID=09458FE25A62D72F9F6CD2C8BC1155FF; Path=/; Secure; HttpOnly; SameSite=Lax', 'Vary': 'Origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000'})
2026-04-24 02:26:28.087	[INFO] 2026-04-24T02:26:28.087Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.terra … with a response body of length 2495 starting in b'{"id":"v2_4ffefed7-76a2-4807-8f3b-b4dde17ad3b5","name":"10137_10137.merged.matefixed.sorted.markeddups.recal.bam","self_uri":"drs://jade.datarepo-dev.broadinstitute.org/v2_4ffefed7-76a2-4807-8f3b-b4dde17ad3b5","size":1840,"created_time":"2023-02-24T20:08:34.007Z","updated_time":"2023-02-24T20:08:34.007Z","version":"0","mime_type":"application/octet-stream","checksums":[{"checksum":"c2c3a933","type":"crc32c"},{"checksum":"a5a2273ce9a6df96e3a0861a12a8a2ba","type":"md5"}],"access_methods":[{"type":"gs","access_url":{"url":"gs://fc-73435f45-96f6-4077-8dea-89782c5c1de5/10137/10137.merged.matefixed.sorted.markeddups.recal.bam","headers":null},"access_id":"gcp-us-central1*74089fe0-c732-40b0-b828-d7e091f0f54b","region":"us-central1","authorizations":{"supported_types":["BearerAuth"],"passport_auth_issuers":null,"bearer_auth_issuers":null}},{"type":"https","access_url":{"url":"https://www.googleapis.com/storage/v1/b/fc-73435f45-96f6-4077-8dea-89782c5c1de5/o/10137%2F10137.merged.matefixed.sorted.markeddups.recal.bam...'
2026-04-24 02:26:28.088	[INFO] 2026-04-24T02:26:28.087Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.terra Making GET request to 'https://jade.datarepo-dev.broadinstitute.org/ga4gh/drs/v1/objects/v2_4ffefed7-76a2-4807-8f3b-b4dde17ad3b5/access/gcp-us-central1*74089fe0-c732-40b0-b828-d7e091f0f54b'
2026-04-24 02:26:28.088	[INFO] 2026-04-24T02:26:28.088Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.terra … without a request body
2026-04-24 02:26:28.088	[INFO] 2026-04-24T02:26:28.088Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.terra … with request headers [('x-goog-api-client', 'cred-type/sa'), ('authorization', 'REDACTED')]
2026-04-24 02:26:29.304	[INFO] 2026-04-24T02:26:29.304Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.terra Got 400 response after 1.216s from GET to https://jade.datarepo-dev.broadinstitute.org/ga4gh/drs/v1/objects/v2_4ffefed7-76a2-4807-8f3b-b4dde17ad3b5/access/gcp-us-central1*74089fe0-c732-40b0-b828-d7e091f0f54b
2026-04-24 02:26:29.304	[INFO] 2026-04-24T02:26:29.304Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.terra … with a response body of length 98 being b'{"msg":"Snapshot requires an x-user-project header for DRS access URL requests","status_code":400}'
2026-04-24 02:26:29.304	[INFO] 2026-04-24T02:26:29.304Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.terra … with response headers HTTPHeaderDict({'Date': 'Fri, 24 Apr 2026 02:26:29 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'Access-Control-Allow-Headers': 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,Accept,Referer,X-App-Id,Origin', 'Access-Control-Allow-Methods': 'GET,POST,DELETE,PUT,PATCH,OPTIONS,HEAD', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains', 'Cache-Control': 'no-cache,no-store,must-revalidate', 'X-Request-ID': '6v5NRGMw', 'Content-Type': 'application/json', 'Content-Length': '98', 'Set-Cookie': 'JSESSIONID=2F4CE8C4D3C2C81301338CE62844CF44; Path=/; Secure; HttpOnly; SameSite=Lax', 'Vary': 'Origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000'})
2026-04-24 02:26:29.305	[WARNING] 2026-04-24T02:26:29.304Z bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed azul.indexer.action_controller Worker failed to handle message SQSFifoMessage(body={'catalog': 'anvil-it', 'operation_id': '529aa38c-b3ed-4826-a928-03a3a8387a66', 'source': {'id': '74089fe0-c732-40b0-b828-d7e091f0f54b', 'spec': 'tdr:bigquery:gcp:datarepo-dev-dd576076:ANVIL_CMG_Sample_1_20230225_ANV5_202512031111', 'prefix': '/1', 'type': 'azul.terra.TDRSourceRef'}, 'prefix': 'a', 'file': {'uuid': '00d3b050-30ec-4eb8-81e3-2a5c2bb9513f', 'name': '10137.merged.matefixed.sorted.markeddups.recal.bam', 'version': '2022-06-01T00:00:00.000000Z', 'drs_uri': 'drs://jade.datarepo-dev.broadinstitute.org/v2_4ffefed7-76a2-4807-8f3b-b4dde17ad3b5', 'size': 1840, 'content_type': None, 'md5': 'a5a2273ce9a6df96e3a0861a12a8a2ba', 'type': 'azul.plugins.metadata.anvil.AnvilFile'}, 'action': 'MirrorFileAction'}, attempts=3, id='cd432568-28c7-4d72-b64c-1d848a2b9453', group_id='f8a3068a-46ab-5a28-8844-892ea2007319', dedup_id='ff18308e-3b27-57cc-a9e8-1fb94efb416a') Traceback (most recent call last): File "/var/task/azul/indexer/action_controller.py", line 50, in _handle_events message_handler(action) ~~~~~~~~~~~~~~~^^^^^^^^ File "/var/task/azul/indexer/mirror_controller.py", line 75, in action_handler self.service(action.catalog).mirror(action) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ File "/var/task/azul/indexer/mirror_service.py", line 561, in mirror self._queue_actions(self._mirror(action)) ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ File "/var/task/azul/indexer/mirror_service.py", line 430, in _queue_actions return self._queues.send_messages(self._mirror_queue(), ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ map(MirrorAction.to_sqs, actions), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ rate_limit=rate_limit) ^^^^^^^^^^^^^^^^^^^^^^ File "/var/task/azul/queues.py", line 210, in send_messages for batch in chunked(messages, self.batch_size): ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/python/more_itertools/recipes.py", line 126, in take return list(islice(iterable, n)) File "/var/task/azul/indexer/mirror_service.py", line 622, in _ self._mirror_file(a.file) ~~~~~~~~~~~~~~~~~^^^^^^^^ File "/var/task/azul/indexer/mirror_service.py", line 636, in _mirror_file file_content = self._download(file) File "/var/task/azul/indexer/mirror_service.py", line 757, in _download url = self._repository_url(file) File "/var/task/azul/indexer/mirror_service.py", line 752, in _repository_url access = object.get(AccessMethod.gs) File "/var/task/azul/drs.py", line 365, in get return self._get(access_method) ~~~~~~~~~^^^^^^^^^^^^^^^ File "/var/task/azul/drs.py", line 387, in _get return self._get_access(access_id, AccessMethod.https) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/task/azul/drs.py", line 427, in _get_access raise DRSStatusException(url, response) azul.drs.DRSStatusException: ('Unexpected response from https://jade.datarepo-dev.broadinstitute.org/ga4gh/drs/v1/objects/v2_4ffefed7-76a2-4807-8f3b-b4dde17ad3b5/access/gcp-us-central1*74089fe0-c732-40b0-b828-d7e091f0f54b', 400, b'{"msg":"Snapshot requires an x-user-project header for DRS access URL requests","status_code":400}')
2026-04-24 02:26:29.306	[ERROR] DRSStatusException: ('Unexpected response from https://jade.datarepo-dev.broadinstitute.org/ga4gh/drs/v1/objects/v2_4ffefed7-76a2-4807-8f3b-b4dde17ad3b5/access/gcp-us-central1*74089fe0-c732-40b0-b828-d7e091f0f54b', 400, b'{"msg":"Snapshot requires an x-user-project header for DRS access URL requests","status_code":400}') Traceback (most recent call last):   File "/var/task/azul/chalice.py", line 190, in patched_event_source_handler     return old_handler(self_, event, context)   File "/var/task/chalice/app.py", line 1761, in call     return self.handler(event_obj)   File "/var/task/azul/indexer/mirror_controller.py", line 67, in mirror     self.mirror(event)   File "/var/task/azul/indexer/mirror_controller.py", line 77, in mirror     self._handle_events(event, action_handler)   File "/var/task/azul/indexer/action_controller.py", line 50, in _handle_events     message_handler(action)   File "/var/task/azul/indexer/mirror_controller.py", line 75, in action_handler     self.service(action.catalog).mirror(action)   File "/var/task/azul/indexer/mirror_service.py", line 561, in mirror     self._queue_actions(self._mirror(action))   File "/var/task/azul/indexer/mirror_service.py", line 430, in _queue_actions     return self._queues.send_messages(self._mirror_queue(),   File "/var/task/azul/queues.py", line 210, in send_messages     for batch in chunked(messages, self.batch_size):   File "/opt/python/more_itertools/recipes.py", line 126, in take     return list(islice(iterable, n))   File "/var/task/azul/indexer/mirror_service.py", line 622, in _     self._mirror_file(a.file)   File "/var/task/azul/indexer/mirror_service.py", line 636, in _mirror_file     file_content = self._download(file)   File "/var/task/azul/indexer/mirror_service.py", line 757, in _download     url = self._repository_url(file)   File "/var/task/azul/indexer/mirror_service.py", line 752, in _repository_url     access = object.get(AccessMethod.gs)   File "/var/task/azul/drs.py", line 365, in get     return self._get(access_method)   File "/var/task/azul/drs.py", line 387, in _get     return self._get_access(access_id, AccessMethod.https)   File "/var/task/azul/drs.py", line 427, in _get_access     raise DRSStatusException(url, response)
2026-04-24 02:26:29.309	REPORT RequestId: bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed Duration: 2531.50 ms Billed Duration: 2532 ms Memory Size: 512 MB Max Memory Used: 171 MB
2026-04-24 02:26:29.309	END RequestId: bdd903b0-67c2-5ef9-a8aa-61d4b13c04ed

---

[achave11-ucsc]

From the most recent IT failures in anvildev, it seems this problem may also cause the repository_files IT test to fail via 500 response from Azul. CW logs at the bottom.

2026-04-24 22:15:57,554    INFO MainThread test.integration_test: Beginning sub-test [repository_files] {'catalog': 'anvil-it'}
2026-04-24 22:15:57,555    INFO MainThread test.integration_test: Making GET request to 'https://service.anvil.gi.ucsc.edu/index/files?catalog=anvil-it&filters=%7B%22files.file_format%22%3A+%7B%22is%22%3A+%5B%22.fastq%22%2C+%22.fastq.gz%22%5D%7D%7D&size=1&order=asc&sort=files.file_size'
2026-04-24 22:15:57,555    INFO MainThread test.integration_test: … without a request body
2026-04-24 22:15:57,556    INFO MainThread test.integration_test: … without request headers
2026-04-24 22:15:57,639    INFO MainThread test.integration_test: Got 200 response after 0.083s from GET to https://service.anvil.gi.ucsc.edu/index/files?catalog=anvil-it&filters=%7B%22files.file_format%22%3A+%7B%22is%22%3A+%5B%22.fastq%22%2C+%22.fastq.gz%22%5D%7D%7D&size=1&order=asc&sort=files.file_size
2026-04-24 22:15:57,639    INFO MainThread test.integration_test: … with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '1512', 'Connection': 'keep-alive', 'Date': 'Fri, 24 Apr 2026 22:15:57 GMT', 'X-Amzn-Trace-Id': 'Root=1-69ebeb9d-570db18e0070e8fa19abbcf5;Parent=0d85553b6ee2affd;Sampled=0;Lineage=1:87986af1:0', 'x-amzn-RequestId': '77f3a228-0f6f-44a1-b116-e3046a7207dc', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", 'x-amz-apigw-id': 'cWHArG-BIAMEayw=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Miss from cloudfront', 'Via': '1.1 185e69fae2e25450e587a1fae1f63962.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'IAD61-P3', 'X-Amz-Cf-Id': 'Vlpif7Gk3pzQjFAniGVxQ32bYYrmt8PGVUkNeSF8FPn4yu-U8YPh1w=='})
2026-04-24 22:15:57,639    INFO MainThread test.integration_test: … with a response body of length 1512 starting in b'{"hits":[],"pagination":{"count":0,"total":0,"size":1,"next":null,"previous":null,"pages":0,"sort":"files.file_size","order":"asc"},"termFacets":{"diagnoses.phenotype":{"terms":[],"total":0,"type":"terms"},"biosamples.disease":{"terms":[],"total":0,"type":"terms"},"diagnoses.disease":{"terms":[],"total":0,"type":"terms"},"diagnoses.phenopacket":{"terms":[],"total":0,"type":"terms"},"files.is_supplementary":{"terms":[],"total":0,"type":"terms"},"files.data_modality":{"terms":[],"total":0,"type":"terms"},"donors.reported_ethnicity":{"terms":[],"total":0,"type":"terms"},"datasets.consent_group":{"terms":[],"total":0,"type":"terms"},"activities.assay_type":{"terms":[],"total":0,"type":"terms"},"datasets.title":{"terms":[],"total":0,"type":"terms"},"biosamples.anatomical_site":{"terms":[],"total":0,"type":"terms"},"donors.organism_type":{"terms":[],"total":0,"type":"terms"},"datasets.data_use_permission":{"terms":[],"total":0,"type":"terms"},"donors.phenotypic_sex":{"terms":[],"total":0,"type":"terms"},"activit...'
2026-04-24 22:15:57,641    INFO MainThread test.integration_test: Making GET request to 'https://service.anvil.gi.ucsc.edu/index/files?catalog=anvil-it&filters=%7B%7D&size=1&order=asc&sort=files.file_size'
2026-04-24 22:15:57,641    INFO MainThread test.integration_test: … without a request body
2026-04-24 22:15:57,641    INFO MainThread test.integration_test: … without request headers
2026-04-24 22:15:57,699    INFO MainThread test.integration_test: Got 200 response after 0.058s from GET to https://service.anvil.gi.ucsc.edu/index/files?catalog=anvil-it&filters=%7B%7D&size=1&order=asc&sort=files.file_size
2026-04-24 22:15:57,699    INFO MainThread test.integration_test: … with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '4032', 'Connection': 'keep-alive', 'Date': 'Fri, 24 Apr 2026 22:15:57 GMT', 'X-Amzn-Trace-Id': 'Root=1-69ebeb9d-53650a25668e921746f345a8;Parent=79c5d56e83def8ef;Sampled=0;Lineage=1:87986af1:0', 'x-amzn-RequestId': '8d96dd5a-1b96-4454-a024-48569b6c85a9', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", 'x-amz-apigw-id': 'cWHAsE07oAMEB8A=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Miss from cloudfront', 'Via': '1.1 185e69fae2e25450e587a1fae1f63962.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'IAD61-P3', 'X-Amz-Cf-Id': '8ATcBPGbUJP-u2HSXkvdjSUR_48pqbycrb8ylVI4qurdhNUv4bc81A=='})
2026-04-24 22:15:57,699    INFO MainThread test.integration_test: … with a response body of length 4032 starting in b'{"hits":[{"entryId":"00d3b050-30ec-4eb8-81e3-2a5c2bb9513f","sources":[{"source_prefix":"/1","source_spec":"tdr:bigquery:gcp:datarepo-dev-dd576076:ANVIL_CMG_Sample_1_20230225_ANV5_202512031111","source_id":"74089fe0-c732-40b0-b828-d7e091f0f54b"}],"bundles":[{"bundle_uuid":"1bb209b9-7104-afcc-9ce9-b9a931e23d1a","bundle_version":"2022-06-01T00:00:00.000000Z"}],"activities":[{"activity_type":["Sequencing"],"assay_type":[null],"data_modality":[null]}],"biosamples":[{"anatomical_site":["sample"],"biosample_type":["sample"],"disease":[null],"donor_age_at_collection_unit":[null],"donor_age_at_collection":[{"gte":null,"lte":null}]}],"datasets":[{"dataset_id":["c70c5ba7-7571-8299-776f-b5dfba9ad4f5"],"title":["ANVIL_CMG_Sample_1"]}],"diagnoses":[],"donors":[{"organism_type":[null],"phenotypic_sex":["Female"],"reported_ethnicity":["Hispanic or Latino"],"genetic_ancestry":[null]}],"files":[{"document_id":"00d3b050-30ec-4eb8-81e3-2a5c2bb9513f","source_datarepo_row_ids":["file_inventory:d21a07bf-9ae4-4cfb-8410-f1a148e0dc...'
2026-04-24 22:15:57,701    INFO MainThread test.integration_test: Making GET request to 'https://service.anvil.gi.ucsc.edu/fetch/repository/files/00d3b050-30ec-4eb8-81e3-2a5c2bb9513f?catalog=anvil-it&version=2022-06-01T00%3A00%3A00.000000Z'
2026-04-24 22:15:57,701    INFO MainThread test.integration_test: … without a request body
2026-04-24 22:15:57,702    INFO MainThread test.integration_test: … without request headers
2026-04-24 22:16:00,504    INFO MainThread test.integration_test: Got 500 response after 2.802s from GET to https://service.anvil.gi.ucsc.edu/fetch/repository/files/00d3b050-30ec-4eb8-81e3-2a5c2bb9513f?catalog=anvil-it&version=2022-06-01T00%3A00%3A00.000000Z
2026-04-24 22:16:00,504    INFO MainThread test.integration_test: … with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '128', 'Connection': 'keep-alive', 'Date': 'Fri, 24 Apr 2026 22:16:00 GMT', 'X-Amzn-Trace-Id': 'Root=1-69ebeb9d-3969d1a71d4a94b645c7f145;Parent=5ac7c076ca98415a;Sampled=0;Lineage=1:87986af1:0', 'x-amzn-RequestId': '46581862-9caa-42d2-8ed3-f0d2601c82c1', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", 'x-amz-apigw-id': 'cWHAtErXoAMEgSg=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 185e69fae2e25450e587a1fae1f63962.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'IAD61-P3', 'X-Amz-Cf-Id': 'o_MbIhC8BeZuAcxr5vfXuFFYGf9zQ6Yd554YPa808lb3wbPomjhiHw=='})
2026-04-24 22:16:00,504    INFO MainThread test.integration_test: … with a response body of length 128 being b'{"Code":"InternalServerError","Message":"An internal server error occurred.","RequestId":"fd2dcc6a-b22b-4067-9cb7-3749258e6390"}'
2026-04-24 22:16:00,504    INFO MainThread test.integration_test: Making GET request to 'https://service.anvil.gi.ucsc.edu/fetch/repository/files/00d3b050-30ec-4eb8-81e3-2a5c2bb9513f?catalog=anvil-it&version=2022-06-01T00%3A00%3A00.000000Z'
2026-04-24 22:16:00,504    INFO MainThread test.integration_test: … without a request body
2026-04-24 22:16:00,505    INFO MainThread test.integration_test: … without request headers
2026-04-24 22:16:02,858    INFO MainThread test.integration_test: Got 500 response after 2.354s from GET to https://service.anvil.gi.ucsc.edu/fetch/repository/files/00d3b050-30ec-4eb8-81e3-2a5c2bb9513f?catalog=anvil-it&version=2022-06-01T00%3A00%3A00.000000Z
2026-04-24 22:16:02,858    INFO MainThread test.integration_test: … with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '128', 'Connection': 'keep-alive', 'Date': 'Fri, 24 Apr 2026 22:16:02 GMT', 'X-Amzn-Trace-Id': 'Root=1-69ebeba0-71315ba8197970cc7d9b0af5;Parent=0643f30b4cd4357f;Sampled=0;Lineage=1:87986af1:0', 'x-amzn-RequestId': 'dcd38262-d3a6-40a9-9102-9a6098eb6308', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", 'x-amz-apigw-id': 'cWHBJEmHIAMEelQ=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 185e69fae2e25450e587a1fae1f63962.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'IAD61-P3', 'X-Amz-Cf-Id': 'k8r2pdDZLLxj8Og94YBElRIyLCMiBQeWvfaOW7FboR5G89hZGfOrOg=='})
2026-04-24 22:16:02,858    INFO MainThread test.integration_test: … with a response body of length 128 being b'{"Code":"InternalServerError","Message":"An internal server error occurred.","RequestId":"3088362a-a930-4c2e-baf2-8faac380837e"}'
2026-04-24 22:16:02,858    INFO MainThread test.integration_test: Making GET request to 'https://service.anvil.gi.ucsc.edu/fetch/repository/files/00d3b050-30ec-4eb8-81e3-2a5c2bb9513f?catalog=anvil-it&version=2022-06-01T00%3A00%3A00.000000Z'
2026-04-24 22:16:02,858    INFO MainThread test.integration_test: … without a request body
2026-04-24 22:16:02,859    INFO MainThread test.integration_test: … without request headers
2026-04-24 22:16:04,642    INFO MainThread test.integration_test: Got 500 response after 1.783s from GET to https://service.anvil.gi.ucsc.edu/fetch/repository/files/00d3b050-30ec-4eb8-81e3-2a5c2bb9513f?catalog=anvil-it&version=2022-06-01T00%3A00%3A00.000000Z
2026-04-24 22:16:04,642    INFO MainThread test.integration_test: … with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '128', 'Connection': 'keep-alive', 'Date': 'Fri, 24 Apr 2026 22:16:04 GMT', 'X-Amzn-Trace-Id': 'Root=1-69ebeba2-3d69ec490a88ad8a51eeb8ca;Parent=787cafe95f4b1879;Sampled=0;Lineage=1:87986af1:0', 'x-amzn-RequestId': '745031df-9002-4e41-8212-3d9462beeaa7', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", 'x-amz-apigw-id': 'cWHBgHQ-IAMEncw=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 185e69fae2e25450e587a1fae1f63962.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'IAD61-P3', 'X-Amz-Cf-Id': '-NdYdMDqASNSBLJha8ieiIpK9NA8lTh8X4x7lMRus-xr2swwsMOiTg=='})
2026-04-24 22:16:04,642    INFO MainThread test.integration_test: … with a response body of length 128 being b'{"Code":"InternalServerError","Message":"An internal server error occurred.","RequestId":"607bb778-223c-48f0-b0c9-5ec77511c90d"}'
2026-04-24 22:16:04,642    INFO MainThread test.integration_test: Making GET request to 'https://service.anvil.gi.ucsc.edu/fetch/repository/files/00d3b050-30ec-4eb8-81e3-2a5c2bb9513f?catalog=anvil-it&version=2022-06-01T00%3A00%3A00.000000Z'
2026-04-24 22:16:04,642    INFO MainThread test.integration_test: … without a request body
2026-04-24 22:16:04,643    INFO MainThread test.integration_test: … without request headers
2026-04-24 22:16:06,065    INFO MainThread test.integration_test: Got 500 response after 1.422s from GET to https://service.anvil.gi.ucsc.edu/fetch/repository/files/00d3b050-30ec-4eb8-81e3-2a5c2bb9513f?catalog=anvil-it&version=2022-06-01T00%3A00%3A00.000000Z
2026-04-24 22:16:06,073    INFO MainThread test.integration_test: … with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '128', 'Connection': 'keep-alive', 'Date': 'Fri, 24 Apr 2026 22:16:06 GMT', 'X-Amzn-Trace-Id': 'Root=1-69ebeba4-7c3214181fea73bb381b889b;Parent=64e71e4033b23e37;Sampled=0;Lineage=1:87986af1:0', 'x-amzn-RequestId': '26dc8d4c-db66-450f-a33d-24eee2a3c170', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", 'x-amz-apigw-id': 'cWHByEw-oAMEtFA=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 185e69fae2e25450e587a1fae1f63962.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'IAD61-P3', 'X-Amz-Cf-Id': 'yBHgsIqWastIerplSogPvh3o9p3dWvWDLSX-5DS6-5CHCqjxg-Jyyw=='})
2026-04-24 22:16:06,073    INFO MainThread test.integration_test: … with a response body of length 128 being b'{"Code":"InternalServerError","Message":"An internal server error occurred.","RequestId":"4addce63-5aa3-4b90-800e-901340eb8030"}'
2026-04-24 22:16:06,073    INFO MainThread test.integration_test: Making GET request to 'https://service.anvil.gi.ucsc.edu/fetch/repository/files/00d3b050-30ec-4eb8-81e3-2a5c2bb9513f?catalog=anvil-it&version=2022-06-01T00%3A00%3A00.000000Z'
2026-04-24 22:16:06,073    INFO MainThread test.integration_test: … without a request body
2026-04-24 22:16:06,074    INFO MainThread test.integration_test: … without request headers
2026-04-24 22:16:07,437    INFO MainThread test.integration_test: Got 500 response after 1.364s from GET to https://service.anvil.gi.ucsc.edu/fetch/repository/files/00d3b050-30ec-4eb8-81e3-2a5c2bb9513f?catalog=anvil-it&version=2022-06-01T00%3A00%3A00.000000Z
2026-04-24 22:16:07,437    INFO MainThread test.integration_test: … with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '128', 'Connection': 'keep-alive', 'Date': 'Fri, 24 Apr 2026 22:16:07 GMT', 'X-Amzn-Trace-Id': 'Root=1-69ebeba6-4fd55a3d4b74f41169833747;Parent=5ca8f7adb4e5e904;Sampled=0;Lineage=1:87986af1:0', 'x-amzn-RequestId': 'f3613313-1509-4ebf-a901-7274da605558', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", 'x-amz-apigw-id': 'cWHCAFGNoAMEvZA=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 185e69fae2e25450e587a1fae1f63962.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'IAD61-P3', 'X-Amz-Cf-Id': 'U53oaaodG3dj1gwK5K6FRxq1gsvJ2XLW_l9fYL9mtQGGzlRaj_zjNg=='})
2026-04-24 22:16:07,438    INFO MainThread test.integration_test: … with a response body of length 128 being b'{"Code":"InternalServerError","Message":"An internal server error occurred.","RequestId":"6bf40948-42ec-4231-95ff-0dfb140ee877"}'
2026-04-24 22:16:07,438    INFO MainThread test.integration_test: Making GET request to 'https://service.anvil.gi.ucsc.edu/fetch/repository/files/00d3b050-30ec-4eb8-81e3-2a5c2bb9513f?catalog=anvil-it&version=2022-06-01T00%3A00%3A00.000000Z'
2026-04-24 22:16:07,438    INFO MainThread test.integration_test: … without a request body
2026-04-24 22:16:07,440    INFO MainThread test.integration_test: … without request headers
2026-04-24 22:16:08,672    INFO MainThread test.integration_test: Got 500 response after 1.233s from GET to https://service.anvil.gi.ucsc.edu/fetch/repository/files/00d3b050-30ec-4eb8-81e3-2a5c2bb9513f?catalog=anvil-it&version=2022-06-01T00%3A00%3A00.000000Z
2026-04-24 22:16:08,672    INFO MainThread test.integration_test: … with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '128', 'Connection': 'keep-alive', 'Date': 'Fri, 24 Apr 2026 22:16:08 GMT', 'X-Amzn-Trace-Id': 'Root=1-69ebeba7-209781c148e8d26758344741;Parent=02c0d679b83c5fe2;Sampled=0;Lineage=1:87986af1:0', 'x-amzn-RequestId': '1fbb5dda-4044-4e98-9f43-1e0a87fa5500', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", 'x-amz-apigw-id': 'cWHCOF9UIAMEvzQ=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 185e69fae2e25450e587a1fae1f63962.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'IAD61-P3', 'X-Amz-Cf-Id': 'YiD0Rss-m12N_rnkOu1XwR7nfI4-00NewLvRn9LX9n-p9yIVkILk0g=='})
2026-04-24 22:16:08,672    INFO MainThread test.integration_test: … with a response body of length 128 being b'{"Code":"InternalServerError","Message":"An internal server error occurred.","RequestId":"393b6012-d7d9-4596-8661-241491e84d11"}'
2026-04-24 22:16:08,672    INFO MainThread test.integration_test: Failed sub-test [repository_files] {'catalog': 'anvil-it'}
  test_indexing (integration_test.IndexingIntegrationTest.test_indexing) [repository_files] (catalog='anvil-it') ... ERROR
…
======================================================================
ERROR: test_indexing (integration_test.IndexingIntegrationTest.test_indexing) [repository_files] (catalog='anvil-it')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builds/ucsc/azul/test/integration_test.py", line 402, in subTest
    yield
  File "/builds/ucsc/azul/test/integration_test.py", line 1189, in _test_repository_files
    self._test_file_download(source, inner_file)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/builds/ucsc/azul/test/integration_test.py", line 1201, in _test_file_download
    response = self._get_url_unchecked(GET, file_url)
  File "/builds/ucsc/azul/test/integration_test.py", line 1014, in _get_url_unchecked
    response = http.request(method=method,
                            url=url,
    ...<2 lines>...
                            timeout=float(config.api_gateway_lambda_timeout + 1),
                            preload_content=not stream)
  File "/build/.venv/lib/python3.14/site-packages/urllib3/_request_methods.py", line 135, in request
    return self.request_encode_url(
           ~~~~~~~~~~~~~~~~~~~~~~~^
        method,
        ^^^^^^^
    ...<3 lines>...
        **urlopen_kw,
        ^^^^^^^^^^^^^
    )
    ^
  File "/build/.venv/lib/python3.14/site-packages/urllib3/_request_methods.py", line 182, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builds/ucsc/azul/src/azul/http.py", line 446, in urlopen
    raise urllib3.exceptions.MaxRetryError(pool, url)
urllib3.exceptions.MaxRetryError: test.integration_test.LoggingHTTPSConnectionPool(host='service.anvil.gi.ucsc.edu', port=443): Max retries exceeded with url: https://service.anvil.gi.ucsc.edu/fetch/repository/files/00d3b050-30ec-4eb8-81e3-2a5c2bb9513f?catalog=anvil-it&version=2022-06-01T00%3A00%3A00.000000Z (Caused by None)
----------------------------------------------------------------------
Ran 16 tests in 392.103s
FAILED (errors=1)
make: *** [Makefile:288: integration_test] Error 1
Cleaning up project directory and file based variables 00:01
ERROR: Job failed: exit code 1

Azul error traceback:

[ERROR]	2026-04-24T22:16:08.662Z	393b6012-d7d9-4596-8661-241491e84d11	azul-service-anvildev	Caught exception for path /fetch/repository/files/{file_uuid}
Traceback (most recent call last):
  File ""/var/task/chalice/app.py"", line 1922, in _get_view_function_response
    response = view_function(**function_args)
  File ""/var/task/azul/service/repository_controller.py"", line 261, in get_fetch_repository_files
    body = self.download_file(file_uuid, fetch=True)
  File ""/var/task/azul/service/repository_controller.py"", line 313, in download_file
    return self._download_file(catalog=catalog,
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
                               fetch=fetch,
                               ^^^^^^^^^^^^
    ...<2 lines>...
                               headers=headers,
                               ^^^^^^^^^^^^^^^^
                               authentication=authentication)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/var/task/azul/service/repository_controller.py"", line 417, in _download_file
    download.update(authentication)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File ""/var/task/azul/plugins/repository/tdr.py"", line 285, in update
    access = drs_client.get(access_method=AccessMethod.gs)
  File ""/var/task/azul/drs.py"", line 365, in get
    return self._get(access_method)
           ~~~~~~~~~^^^^^^^^^^^^^^^
  File ""/var/task/azul/drs.py"", line 387, in _get
    return self._get_access(access_id, AccessMethod.https)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/var/task/azul/drs.py"", line 427, in _get_access
    raise DRSStatusException(url, response)
azul.drs.DRSStatusException: ('Unexpected response from https://jade.datarepo-dev.broadinstitute.org/ga4gh/drs/v1/objects/v2_4ffefed7-76a2-4807-8f3b-b4dde17ad3b5/access/gcp-us-central1*74089fe0-c732-40b0-b828-d7e091f0f54b', 400, b'{""msg"":""Snapshot requires an x-user-project header for DRS access URL requests"",""status_code"":400}')"

CloudWatch Logs Insights
region: us-east-1
log-group-names: /aws/lambda/azul-service-anvildev
data-sources:
facets:
start-time: -86400s
end-time: 0s
query-string:

fields @timestamp, @message
# | filter @message like /Returning 5|ServiceUnavailableError/
# | filter @message like "fetch/repository/files/00d3b050-30ec-4eb8-81e3-2a5c2bb9513f"
| filter @message like /393b6012/
| sort @timestamp asc
| limit 1000

---

@timestamp	@message
2026-04-24 22:16:07.471	START RequestId: 393b6012-d7d9-4596-8661-241491e84d11 Version: 138
2026-04-24 22:16:07.472	[INFO] 2026-04-24T22:16:07.472Z 393b6012-d7d9-4596-8661-241491e84d11 azul.chalice Received GET request for '/fetch/repository/files/00d3b050-30ec-4eb8-81e3-2a5c2bb9513f', with {"query": {"catalog": "anvil-it", "version": "2022-06-01T00:00:00.000000Z"}, "headers": {"accept-encoding": "identity", "cloudfront-forwarded-proto": "https", "cloudfront-is-desktop-viewer": "true", "cloudfront-is-mobile-viewer": "false", "cloudfront-is-smarttv-viewer": "false", "cloudfront-is-tablet-viewer": "false", "cloudfront-viewer-asn": "14618", "cloudfront-viewer-country": "US", "host": "service.anvil.gi.ucsc.edu", "user-agent": "python-urllib3/2.6.3", "via": "1.1 185e69fae2e25450e587a1fae1f63962.cloudfront.net (CloudFront)", "x-amz-cf-id": "YiD0Rss-m12N_rnkOu1XwR7nfI4-00NewLvRn9LX9n-p9yIVkILk0g==", "x-amzn-trace-id": "Root=1-69ebeba7-209781c148e8d26758344741", "x-forwarded-for": "54.211.146.213, 18.68.30.172", "x-forwarded-port": "443", "x-forwarded-proto": "https"}}.
2026-04-24 22:16:07.472	[INFO] 2026-04-24T22:16:07.472Z 393b6012-d7d9-4596-8661-241491e84d11 azul.chalice Did not authenticate request.
2026-04-24 22:16:07.472	[INFO] 2026-04-24T22:16:07.472Z 393b6012-d7d9-4596-8661-241491e84d11 azul.chalice … without a request body
2026-04-24 22:16:07.473	[INFO] 2026-04-24T22:16:07.473Z 393b6012-d7d9-4596-8661-241491e84d11 opensearch … with a request body of length 399 being b'{"query":{"bool":{"must":[{"constant_score":{"filter":{"terms":{"contents.files.document_id.keyword":["00d3b050-30ec-4eb8-81e3-2a5c2bb9513f"]}}}},{"constant_score":{"filter":{"terms":{"contents.files.version.keyword":["2022-06-01T00:00:00.000000Z"]}}}},{"constant_score":{"filter":{"terms":{"sources.id.keyword":["74089fe0-c732-40b0-b828-d7e091f0f54b","790795c4-49b1-4ac8-a060-207b92ea08c5"]}}}}]}}}'
2026-04-24 22:16:07.473	[INFO] 2026-04-24T22:16:07.473Z 393b6012-d7d9-4596-8661-241491e84d11 opensearch Making POST request to https://vpc-azul-index-anvildev-jubbechxp4wxzfj2kf2hiewjgi.us-east-1.es.amazonaws.com:443/azul_v2_anvildev_anvil-it_files_aggregate/_search
2026-04-24 22:16:07.483	[INFO] 2026-04-24T22:16:07.483Z 393b6012-d7d9-4596-8661-241491e84d11 opensearch Got 200 response after 0.010s from POST to https://vpc-azul-index-anvildev-jubbechxp4wxzfj2kf2hiewjgi.us-east-1.es.amazonaws.com:443/azul_v2_anvildev_anvil-it_files_aggregate/_search
2026-04-24 22:16:07.483	[INFO] 2026-04-24T22:16:07.483Z 393b6012-d7d9-4596-8661-241491e84d11 opensearch … with a response body of length 2762 starting in '{"took":2,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":1,"relation":"eq"},"max_score":3.0,"hits":[{"_index":"azul_v2_anvildev_anvil-it_files_aggregate","_id":"00d3b050-30ec-4eb8-81e3-2a5c2bb9513f","_score":3.0,"_source":{"entity_id":"00d3b050-30ec-4eb8-81e3-2a5c2bb9513f","contents":{"activities":[{"document_id":["e0cd75c0-ab72-4456-a8db-47e54988191e"],"source_datarepo_row_ids":["sequencing:40a6eff9-3886-49ce-b3c0-0002ef8fef2d"],"activity_id":["87ad00ec-986e-bf1c-c139-f755d601a669"],"activity_table":["anvil_sequencingactivity"],"activity_type":["Sequencing"],"assay_type":["~null"],"data_modality":["~null"],"reference_assembly":["~null"]}],"biosamples":[{"document_id":["1bb209b9-7104-4fcc-9ce9-b9a931e23d1a"],"source_datarepo_row_ids":["sample:badc7520-3f9e-4f6e-9397-85c9b4082406"],"biosample_id":["715e8ba4-6f8b-c29c-24f3-fbfdb8d96d71"],"anatomical_site":["sample"],"apriori_cell_type":["~null"],"biosample_type":["sample"],"disease":["~null"],"donor_age_a…'
2026-04-24 22:16:07.485	[INFO] 2026-04-24T22:16:07.485Z 393b6012-d7d9-4596-8661-241491e84d11 azul.boto3 s3.HeadObject: Making HEAD request to https://s3.amazonaws.com/edu-ucsc-gi-platform-anvil-dev-mirror-anvildev.us-east-1/_it/info/a5a2273ce9a6df96e3a0861a12a8a2ba.md5.json
2026-04-24 22:16:07.485	[INFO] 2026-04-24T22:16:07.485Z 393b6012-d7d9-4596-8661-241491e84d11 azul.boto3 s3.HeadObject: … without a request body
2026-04-24 22:16:07.499	[INFO] 2026-04-24T22:16:07.499Z 393b6012-d7d9-4596-8661-241491e84d11 azul.boto3 s3.HeadObject: Got 404 response
2026-04-24 22:16:07.499	[INFO] 2026-04-24T22:16:07.499Z 393b6012-d7d9-4596-8661-241491e84d11 azul.boto3 s3.HeadObject: … without a response body
2026-04-24 22:16:07.500	[INFO] 2026-04-24T22:16:07.500Z 393b6012-d7d9-4596-8661-241491e84d11 azul.terra … with request headers [('x-goog-api-client', 'cred-type/sa'), ('authorization', 'REDACTED')]
2026-04-24 22:16:07.500	[INFO] 2026-04-24T22:16:07.500Z 393b6012-d7d9-4596-8661-241491e84d11 azul.terra … without a request body
2026-04-24 22:16:07.500	[INFO] 2026-04-24T22:16:07.500Z 393b6012-d7d9-4596-8661-241491e84d11 azul.terra Making GET request to 'https://jade.datarepo-dev.broadinstitute.org/ga4gh/drs/v1/objects/v2_4ffefed7-76a2-4807-8f3b-b4dde17ad3b5'
2026-04-24 22:16:07.993	[INFO] 2026-04-24T22:16:07.993Z 393b6012-d7d9-4596-8661-241491e84d11 azul.terra … with response headers HTTPHeaderDict({'Date': 'Fri, 24 Apr 2026 22:16:07 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'Access-Control-Allow-Headers': 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,Accept,Referer,X-App-Id,Origin', 'Access-Control-Allow-Methods': 'GET,POST,DELETE,PUT,PATCH,OPTIONS,HEAD', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains', 'Cache-Control': 'no-cache,no-store,must-revalidate', 'X-Request-ID': 'a00kzY21', 'Content-Type': 'application/json', 'Content-Length': '2495', 'Set-Cookie': 'JSESSIONID=1EB5041F637C39BCBD5093E66282B55D; Path=/; Secure; HttpOnly; SameSite=Lax', 'Vary': 'Origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000'})
2026-04-24 22:16:07.993	[INFO] 2026-04-24T22:16:07.993Z 393b6012-d7d9-4596-8661-241491e84d11 azul.terra Got 200 response after 0.493s from GET to https://jade.datarepo-dev.broadinstitute.org/ga4gh/drs/v1/objects/v2_4ffefed7-76a2-4807-8f3b-b4dde17ad3b5
2026-04-24 22:16:07.993	[INFO] 2026-04-24T22:16:07.993Z 393b6012-d7d9-4596-8661-241491e84d11 azul.terra … with a response body of length 2495 starting in b'{"id":"v2_4ffefed7-76a2-4807-8f3b-b4dde17ad3b5","name":"10137_10137.merged.matefixed.sorted.markeddups.recal.bam","self_uri":"drs://jade.datarepo-dev.broadinstitute.org/v2_4ffefed7-76a2-4807-8f3b-b4dde17ad3b5","size":1840,"created_time":"2023-02-24T20:08:34.007Z","updated_time":"2023-02-24T20:08:34.007Z","version":"0","mime_type":"application/octet-stream","checksums":[{"checksum":"c2c3a933","type":"crc32c"},{"checksum":"a5a2273ce9a6df96e3a0861a12a8a2ba","type":"md5"}],"access_methods":[{"type":"gs","access_url":{"url":"gs://fc-73435f45-96f6-4077-8dea-89782c5c1de5/10137/10137.merged.matefixed.sorted.markeddups.recal.bam","headers":null},"access_id":"gcp-us-central1*74089fe0-c732-40b0-b828-d7e091f0f54b","region":"us-central1","authorizations":{"supported_types":["BearerAuth"],"passport_auth_issuers":null,"bearer_auth_issuers":null}},{"type":"https","access_url":{"url":"https://www.googleapis.com/storage/v1/b/fc-73435f45-96f6-4077-8dea-89782c5c1de5/o/10137%2F10137.merged.matefixed.sorted.markeddups.recal.bam...'
2026-04-24 22:16:07.993	[INFO] 2026-04-24T22:16:07.993Z 393b6012-d7d9-4596-8661-241491e84d11 azul.terra … without a request body
2026-04-24 22:16:07.993	[INFO] 2026-04-24T22:16:07.993Z 393b6012-d7d9-4596-8661-241491e84d11 azul.terra Making GET request to 'https://jade.datarepo-dev.broadinstitute.org/ga4gh/drs/v1/objects/v2_4ffefed7-76a2-4807-8f3b-b4dde17ad3b5/access/gcp-us-central1*74089fe0-c732-40b0-b828-d7e091f0f54b'
2026-04-24 22:16:07.994	[INFO] 2026-04-24T22:16:07.994Z 393b6012-d7d9-4596-8661-241491e84d11 azul.terra … with request headers [('x-goog-api-client', 'cred-type/sa'), ('authorization', 'REDACTED')]
2026-04-24 22:16:08.662	[INFO] 2026-04-24T22:16:08.662Z 393b6012-d7d9-4596-8661-241491e84d11 azul.terra … with a response body of length 98 being b'{"msg":"Snapshot requires an x-user-project header for DRS access URL requests","status_code":400}'
2026-04-24 22:16:08.662	[INFO] 2026-04-24T22:16:08.662Z 393b6012-d7d9-4596-8661-241491e84d11 azul.terra … with response headers HTTPHeaderDict({'Date': 'Fri, 24 Apr 2026 22:16:08 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'Access-Control-Allow-Headers': 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,Accept,Referer,X-App-Id,Origin', 'Access-Control-Allow-Methods': 'GET,POST,DELETE,PUT,PATCH,OPTIONS,HEAD', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains', 'Cache-Control': 'no-cache,no-store,must-revalidate', 'X-Request-ID': 'axr3o84G', 'Content-Type': 'application/json', 'Content-Length': '98', 'Set-Cookie': 'JSESSIONID=85A991CEC23DAA65AEB4AA5000E69046; Path=/; Secure; HttpOnly; SameSite=Lax', 'Vary': 'Origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000'})
2026-04-24 22:16:08.662	[INFO] 2026-04-24T22:16:08.662Z 393b6012-d7d9-4596-8661-241491e84d11 azul.terra Got 400 response after 0.668s from GET to https://jade.datarepo-dev.broadinstitute.org/ga4gh/drs/v1/objects/v2_4ffefed7-76a2-4807-8f3b-b4dde17ad3b5/access/gcp-us-central1*74089fe0-c732-40b0-b828-d7e091f0f54b
2026-04-24 22:16:08.664	[ERROR] 2026-04-24T22:16:08.662Z 393b6012-d7d9-4596-8661-241491e84d11 azul-service-anvildev Caught exception for path /fetch/repository/files/{file_uuid} Traceback (most recent call last): File "/var/task/chalice/app.py", line 1922, in _get_view_function_response response = view_function(**function_args) File "/var/task/azul/service/repository_controller.py", line 261, in get_fetch_repository_files body = self.download_file(file_uuid, fetch=True) File "/var/task/azul/service/repository_controller.py", line 313, in download_file return self._download_file(catalog=catalog, ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ fetch=fetch, ^^^^^^^^^^^^ ...<2 lines>... headers=headers, ^^^^^^^^^^^^^^^^ authentication=authentication) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/task/azul/service/repository_controller.py", line 417, in _download_file download.update(authentication) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^ File "/var/task/azul/plugins/repository/tdr.py", line 285, in update access = drs_client.get(access_method=AccessMethod.gs) File "/var/task/azul/drs.py", line 365, in get return self._get(access_method) ~~~~~~~~~^^^^^^^^^^^^^^^ File "/var/task/azul/drs.py", line 387, in _get return self._get_access(access_id, AccessMethod.https) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/task/azul/drs.py", line 427, in _get_access raise DRSStatusException(url, response) azul.drs.DRSStatusException: ('Unexpected response from https://jade.datarepo-dev.broadinstitute.org/ga4gh/drs/v1/objects/v2_4ffefed7-76a2-4807-8f3b-b4dde17ad3b5/access/gcp-us-central1*74089fe0-c732-40b0-b828-d7e091f0f54b', 400, b'{"msg":"Snapshot requires an x-user-project header for DRS access URL requests","status_code":400}')
2026-04-24 22:16:08.664	[INFO] 2026-04-24T22:16:08.664Z 393b6012-d7d9-4596-8661-241491e84d11 azul.chalice … with a response body of length 133 being {"Code": "InternalServerError", "Message": "An internal server error occurred.", "RequestId": "393b6012-d7d9-4596-8661-241491e84d11"}
2026-04-24 22:16:08.664	[INFO] 2026-04-24T22:16:08.664Z 393b6012-d7d9-4596-8661-241491e84d11 azul.chalice Returning 500 response with headers {"headers": {"Access-Control-Allow-Origin": "*", "Access-Control-Allow-Headers": "Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key", "Content-Security-Policy": "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", "Referrer-Policy": "strict-origin-when-cross-origin", "Strict-Transport-Security": "max-age=63072000; includeSubDomains; preload", "X-Content-Type-Options": "nosniff", "X-Frame-Options": "DENY", "X-XSS-Protection": "1; mode=block", "Cache-Control": "no-store"}}.
2026-04-24 22:16:08.666	REPORT RequestId: 393b6012-d7d9-4596-8661-241491e84d11 Duration: 1194.81 ms Billed Duration: 1195 ms Memory Size: 768 MB Max Memory Used: 183 MB
2026-04-24 22:16:08.666	END RequestId: 393b6012-d7d9-4596-8661-241491e84d11

---

[hannes-ucsc]

I added hotfixes/workarounds:

• 9c2257a disables the mirroring IT in anvildev, anvilbox and tempdev (this needs to be reverted when blocker Use requester-pays for mirroring in AnVIL deployments #7794 lands)
• c7e824e prevents an unwanted side effect of the previous commit (this needs to be reverted when blocker Use requester-pays for mirroring in AnVIL deployments #7794 lands)
• 4b7cf46 causes the /repository/files endpoint to return 403 when DRS returns the 400 with the Snapshot requires an x-user-project header for DRS access body, and prevents the IT from failing in that case
• 0e5baa4 adds a FIXME to make the IT leniency from the previous commit conditional so that a 403 from /repository/files is considered an error if requester-pays is enabled

The PR for blocker #7794 should revert the first two commits, and address the FIXME in the fourth.