--mode verify on 000026: Zarr ... entry '0/1/29/72' added, but Dandiset draft timestamp was not updated on server #340
Assignees
Comments
|
@yarikoptic I don't know how this happened, but somehow the backup of Zarr 92ab6451-8123-4a04-94b0-571ed9650d8d that was committed at Fri Mar 31 08:16:28 2023 -0400 ended up with a Running a (non-verify) backup with |
|
interesting, ok, running on drogon |
|
I think --zar-mode force run indeed fixed it, so let's consider addressed. Thank you @jwodder for the analysis and clear instructions on how to mitigate. |
|
it seems that issue persists/came back. Got two runs failing and then I will try to redo that invocation |
|
I think that run helped but now run ran into it again! There seems to be some logic issue somewhere... will need to analyze later. Meanwhile will try to force again |
|
apparently it is not sufficient to bring the dataset into correct state: here is the error I got originally in email from cron job: tools.backups2datalad.util.UnexpectedChangeError: Dandiset 000026: Zarr fe01bdb8-9e6c-4fca-8607-7cbd417a4a35: entry '0/0/29/20' added, but Dandiset draft timestamp was not updated on server rerunning the the zarr is said to be complete ❯ curl -X 'GET' 'https://api.dandiarchive.org/api/zarr/fe01bdb8-9e6c-4fca-8607-7cbd417a4a35/' -H 'accept: application/json'
{"name":"sub-I59/ses-SPIM/micr/sub-I59_ses-SPIM_sample-BrocaAreaS21_stain-NeuN_SPIM.ome.zarr","dandiset":"000026","zarr_id":"fe01bdb8-9e6c-4fca-8607-7cbd417a4a35","status":"Complete","checksum":"00a168a8ecf111c307f9930d2a44f0a5-16267--25462074744","file_count":16267,"size":25462074744}locally we do not have that path in the zarr but it does exist on the archive so the so after that date (but again -- missing that path). So it seems that we ended up in an inconsistent state... edit 1: Running now with force for overall check: |
|
we got similar one but on .nwb in some new dandiset, there is an email ( Before then dandiset was empty and diff (now reset) to dandiset.yaml wasdandi@drogon:/mnt/backup/dandi/dandisets/000640$ git diff --cached
diff --git a/dandiset.yaml b/dandiset.yaml
index 0579ef8..4d69bc7 100644
--- a/dandiset.yaml
+++ b/dandiset.yaml
@@ -6,9 +6,21 @@ access:
- schemaKey: AccessRequirements
status: dandi:OpenAccess
assetsSummary:
- numberOfBytes: 0
- numberOfFiles: 0
+ approach: []
+ dataStandard:
+ - identifier: RRID:SCR_015242
+ name: Neurodata Without Borders (NWB)
+ schemaKey: StandardsType
+ measurementTechnique: []
+ numberOfBytes: 14849530840
+ numberOfFiles: 214
+ numberOfSubjects: 14
schemaKey: AssetsSummary
+ species:
+ - identifier: http://purl.obolibrary.org/obo/NCBITaxon_10116
+ name: Rattus norvegicus - Norway rat
+ schemaKey: SpeciesType
+ variableMeasured: []
citation: Koloski, Miranda (2023) 32-CH Local Field Potential Data During Probabilistic
Reversal Learning Task (Version draft) [Data set]. DANDI archive. https://doi.org/10.80507/dandi.123456/0.123456.1234
contributor:
@@ -20,7 +32,7 @@ contributor:
- dcite:ContactPerson
schemaKey: Person
dateCreated: '2023-09-22T00:16:23.478717+00:00'
-datePublished: '2023-09-22T00:16:41.387621+00:00'
+datePublished: '2023-10-25T07:39:28.735955+00:00'
description: Male Long Evans Rats 7-12 months of age. 24 total animals. 256 sessions. 13
animals received a frontal traumatic brain injury (controlled cortical impact).
11 animals received Sham (no craniotomy no TBI) conditions. One week later, rats
@@ -43,13 +55,13 @@ manifestLocation:
- https://api.dandiarchive.org/api/dandisets/000640/versions/draft/assets/
name: 32-CH Local Field Potential Data During Probabilistic Reversal Learning Task
publishedBy:
- endDate: '2023-09-22T00:16:41.387621+00:00'
- id: urn:uuid:7e1289a2-dd48-4c5f-bd6c-5dfc120f38c2
+ endDate: '2023-10-25T07:39:28.735955+00:00'
+ id: urn:uuid:8b37e074-dfe9-46e1-9ca7-86cd566fb2cd
name: DANDI publish
schemaKey: PublishActivity
- startDate: '2023-09-22T00:16:41.387621+00:00'
+ startDate: '2023-10-25T07:39:28.735955+00:00'
wasAssociatedWith:
- - id: urn:uuid:50e8ca2c-95bb-44b4-83cf-7c5ccefdd47d
+ - id: urn:uuid:b5d9f6fc-1244-4c96-bcdc-f4645a3505c3
identifier: RRID:SCR_017571
name: DANDI API
schemaKey: Software
edit: running here is the diff between diffs of above and committed changes to dandiset.yaml -- nothing about date/time but has difference in stats, so dandiset was still "being uploaded to" --- a/dandiset.yaml
+++ b/dandiset.yaml
@@ -6,9 +6,21 @@ access:
@@ -14,9 +20,9 @@
+ name: Neurodata Without Borders (NWB)
+ schemaKey: StandardsType
+ measurementTechnique: []
-+ numberOfBytes: 14849530840
-+ numberOfFiles: 214
-+ numberOfSubjects: 14
++ numberOfBytes: 18708666000
++ numberOfFiles: 247
++ numberOfSubjects: 19
schemaKey: AssetsSummary
+ species:
+ - identifier: http://purl.obolibrary.org/obo/NCBITaxon_10116
@@ -53,4 +59,3 @@
identifier: RRID:SCR_017571
name: DANDI API
schemaKey: Software
- |
fresh one on 0000262023-10-27T01:50:10-0400 [WARNING ] dandi: A newer version (0.56.2) of dandi/dandi-cli is available. You are using 0.55.1
2023-10-27T02:06:48-0400 [ERROR ] backups2datalad: Job failed on input <Dandiset 000026/draft>:
Traceback (most recent call last):
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 173, in dowork
outp = await func(inp)
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 151, in update_dandiset
changed = await self.sync_dataset(
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 188, in sync_dataset
await syncer.sync_assets(error_on_change)
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/syncer.py", line 36, in sync_assets
self.report = await async_assets(
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/asyncer.py", line 522, in async_assets
async with AsyncAnnex(ds.pathobj) as annex, httpx.AsyncClient(
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
raise exceptions[0]
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 560, in sync_zarr
await zsync.run()
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 140, in run
if not await self.needs_sync(client, last_sync, local_paths):
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 361, in needs_sync
self.check_change(f"entry {path!r} added")
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 489, in check_change
raise UnexpectedChangeError(
tools.backups2datalad.util.UnexpectedChangeError: Dandiset 000026: Zarr 5c37c233-222f-4e60-96e7-a7536e08ef61: entry '0/1/31/70' added, but Dandiset draft timestamp was not updated on server
2023-10-27T02:10:36-0400 [WARNING ] backups2datalad: Retrying GET request to /dandisets/000053/versions/draft/ in 1.009969 seconds as it raised ConnectError: All connection attempts failed
2023-10-27T02:10:38-0400 [WARNING ] backups2datalad: Retrying GET request to /dandisets/000049/versions/ in 1.004621 seconds as it raised ConnectError: All connection attempts failed
2023-10-27T02:10:43-0400 [WARNING ] backups2datalad: Retrying GET request to /dandisets/000051/versions/ in 1.007710 seconds as it raised ConnectError: All connection attempts failed
2023-10-27T02:57:02-0400 [WARNING ] backups2datalad: Retrying GET request to /dandisets/000292/versions/ in 0.959762 seconds as it raised ConnectError: All connection attempts failed
2023-10-27T03:48:06-0400 [WARNING ] backups2datalad: Failed [rc=1]: git -c receive.autogc=0 -c gc.auto=0 annex add dandiset.yaml [cwd=/mnt/backup/dandi/dandisets/000623]
Stdout: <empty>
Stderr:
fatal: git show-ref: bad ref refs/heads/git-annex (0000000000000000000000000000000000000000)
fatal: git show-ref: bad ref refs/heads/git-annex (0000000000000000000000000000000000000000)
error: object file .git/objects/b6/d1d20e316838110ae3385d8aeff437db79fe9f is empty
fatal: b6d1d20e316838110ae3385d8aeff437db79fe9f is not a valid object
git-annex: user error (git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","commit-tree","b6d1d20e316838110ae3385d8aeff437db79fe9f","--no-gpg-sign"] exited 128)
2023-10-27T03:48:06-0400 [ERROR ] backups2datalad: Job failed on input <Dandiset 000623/draft>:
Traceback (most recent call last):
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 173, in dowork
outp = await func(inp)
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 151, in update_dandiset
changed = await self.sync_dataset(
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 187, in sync_dataset
await update_dandiset_metadata(dandiset, ds, log=manager.log)
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/util.py", line 185, in update_dandiset_metadata
await ds.add(dandiset_metadata_file)
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/adataset.py", line 233, in add
await self.call_annex("add", path)
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/adataset.py", line 146, in call_annex
await aruncmd("git", *GIT_OPTIONS, "annex", *args, cwd=self.path, **kwargs)
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 217, in aruncmd
raise e
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 202, in aruncmd
r = await anyio.run_process(argstrs, **kwargs)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_core/_subprocesses.py", line 90, in run_process
raise CalledProcessError(cast(int, process.returncode), command, output, errors)
subprocess.CalledProcessError: Command '['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'add', 'dandiset.yaml']' returned non-zero exit status 1.
2023-10-27T03:58:45-0400 [ERROR ] backups2datalad: An error occurred:
Traceback (most recent call last):
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/__main__.py", line 111, in wrapped
await f(datasetter, *args, **kwargs)
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/__main__.py", line 213, in update_from_backup
await datasetter.update_from_backup(dandisets, exclude=exclude)
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 97, in update_from_backup
raise RuntimeError(
RuntimeError: Backups for 2 Dandisets failed
Logs saved to /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2023.10.27.05.50.09Z.log
action summary:
publish (notneeded: 2) edit: then it "cleared up" with some next run... may be related to the fact that zarr lifecycle is yet "in flux" and might need to get adjusted, see dandi/dandi-archive#1698 . Although above case with .nwb suggests that it might be a generic issue with asset lifecycle although could be some other altogether since we do not typically observe them for nwb |
|
more of fresh ones: 000344 README.md2024-02-09T02:14:49-0500 [ERROR ] backups2datalad: Job failed on input <Dandiset 000344/draft>:
+ Exception Group Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/aioutil.py", line 176, in dowork
| outp = await func(inp)
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 159, in update_dandiset
| changed = await self.sync_dataset(
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 203, in sync_dataset
| await syncer.sync_assets()
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/syncer.py", line 77, in sync_assets
| report = await async_assets(
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 498, in async_assets
| async with (
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 664, in __aexit__
| raise BaseExceptionGroup(
| exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob
| raise UnexpectedChangeError(
| backups2datalad.util.UnexpectedChangeError: Dandiset 000344: Metadata for asset README.md was changed/added but draft timestamp was not updated on server
+------------------------------------ 000729 sub-256/dwi/sub-256_dwi.json2024-02-09T02:17:53-0500 [ERROR ] backups2datalad: Job failed on input <Dandiset 000729/draft>:
+ Exception Group Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/aioutil.py", line 176, in dowork
| outp = await func(inp)
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 159, in update_dandiset
| changed = await self.sync_dataset(
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 203, in sync_dataset
| await syncer.sync_assets()
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/syncer.py", line 77, in sync_assets
| report = await async_assets(
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 498, in async_assets
| async with (
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 664, in __aexit__
| raise BaseExceptionGroup(
| exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (6 sub-exceptions)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob
| raise UnexpectedChangeError(
| backups2datalad.util.UnexpectedChangeError: Dandiset 000729: Metadata for asset derivatives/DWIprep_V.1.0/dataset_description.json was changed/added but draft timestamp was not updated on server
+---------------- 2 ----------------
| Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob
| raise UnexpectedChangeError(
| backups2datalad.util.UnexpectedChangeError: Dandiset 000729: Metadata for asset dataset_description.json was changed/added but draft timestamp was not updated on server
+---------------- 3 ----------------
| Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob
| raise UnexpectedChangeError(
| backups2datalad.util.UnexpectedChangeError: Dandiset 000729: Metadata for asset participants.json was changed/added but draft timestamp was not updated on server
+---------------- 4 ----------------
| Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob
| raise UnexpectedChangeError(
| backups2datalad.util.UnexpectedChangeError: Dandiset 000729: Metadata for asset participants.tsv was changed/added but draft timestamp was not updated on server
+---------------- 5 ----------------
| Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob
| raise UnexpectedChangeError(
| backups2datalad.util.UnexpectedChangeError: Dandiset 000729: Metadata for asset sub-243/dwi/sub-243_dwi.json was changed/added but draft timestamp was not updated on server
+---------------- 6 ----------------
| Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob
| raise UnexpectedChangeError(
| backups2datalad.util.UnexpectedChangeError: Dandiset 000729: Metadata for asset sub-256/dwi/sub-256_dwi.json was changed/added but draft timestamp was not updated on server
+------------------------------------ 000731 -- multiple files2024-02-09T02:17:56-0500 [ERROR ] backups2datalad: Job failed on input <Dandiset 000731/draft>:
+ Exception Group Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/aioutil.py", line 176, in dowork
| outp = await func(inp)
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 159, in update_dandiset
| changed = await self.sync_dataset(
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 203, in sync_dataset
| await syncer.sync_assets()
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/syncer.py", line 77, in sync_assets
| report = await async_assets(
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 498, in async_assets
| async with (
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 664, in __aexit__
| raise BaseExceptionGroup(
| exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (4 sub-exceptions)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob
| raise UnexpectedChangeError(
| backups2datalad.util.UnexpectedChangeError: Dandiset 000731: Metadata for asset dataset_description.json was changed/added but draft timestamp was not updated on server
+---------------- 2 ----------------
| Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob
| raise UnexpectedChangeError(
| backups2datalad.util.UnexpectedChangeError: Dandiset 000731: Metadata for asset derivatives/dataset_description.json was changed/added but draft timestamp was not updated on server
+---------------- 3 ----------------
| Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob
| raise UnexpectedChangeError(
| backups2datalad.util.UnexpectedChangeError: Dandiset 000731: Metadata for asset participants.json was changed/added but draft timestamp was not updated on server
+---------------- 4 ----------------
| Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob
| raise UnexpectedChangeError(
| backups2datalad.util.UnexpectedChangeError: Dandiset 000731: Metadata for asset participants.tsv was changed/added but draft timestamp was not updated on server @jwodder -- please troubleshoot this further -- we might need more output to be provided (e.g. what in metadata has changed to be listed in the exception or some ERROR log) to be able to actually troubleshoot whenever it happens next. the issue is currently easily reproducible so good time to get that done:dandi@drogon:~$ flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron-nonzarr.lock bash
dandi@drogon:~$ crontab -l | grep backup.*veri
50 1 * * * chronic flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron-nonzarr.lock bash -c '/mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --mode verify'
dandi@drogon:~$ /mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --mode verify 000731
2024-02-09T09:42:45-0500 [ERROR ] backups2datalad: Job failed on input <Dandiset 000731/draft>:
+ Exception Group Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/aioutil.py", line 176, in dowork
| outp = await func(inp)
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 159, in update_dandiset
| changed = await self.sync_dataset(
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 203, in sync_dataset
| await syncer.sync_assets()
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/syncer.py", line 77, in sync_assets
| report = await async_assets(
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 498, in async_assets
| async with (
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 664, in __aexit__
| raise BaseExceptionGroup(
| exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (4 sub-exceptions)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob
| raise UnexpectedChangeError(
| backups2datalad.util.UnexpectedChangeError: Dandiset 000731: Metadata for asset dataset_description.json was changed/added but draft timestamp was not updated on server
+---------------- 2 ----------------
| Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob
| raise UnexpectedChangeError(
| backups2datalad.util.UnexpectedChangeError: Dandiset 000731: Metadata for asset derivatives/dataset_description.json was changed/added but draft timestamp was not updated on server
+---------------- 3 ----------------
| Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob
| raise UnexpectedChangeError(
| backups2datalad.util.UnexpectedChangeError: Dandiset 000731: Metadata for asset participants.json was changed/added but draft timestamp was not updated on server
+---------------- 4 ----------------
| Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 256, in process_blob
| raise UnexpectedChangeError(
| backups2datalad.util.UnexpectedChangeError: Dandiset 000731: Metadata for asset participants.tsv was changed/added but draft timestamp was not updated on server
+------------------------------------
2024-02-09T09:42:45-0500 [ERROR ] backups2datalad: An error occurred:
Traceback (most recent call last):
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/__main__.py", line 119, in wrapped
await f(datasetter, *args, **kwargs)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/__main__.py", line 228, in update_from_backup
await datasetter.update_from_backup(dandisets, exclude=exclude)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 94, in update_from_backup
raise RuntimeError(
RuntimeError: Backups for 1 Dandiset failed
Logs saved to /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.02.09.14.42.44Z.log
|
|
those recent were all private and we required --force 'full rerun, see |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Might be the result of what I have forecasted in #246 but also reminds some issue on dandi-archive level but I have failed to find an open issue so might be not that.
Twice in a row we have cron job fails on that 000026 which now is acquiring some zarrs
more of traceback
needs to be troubleshooted and addressed .
The text was updated successfully, but these errors were encountered: