You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Sometimes, especially on community resources from transport.data.gouv.fr, our fs_filename is not synced with url. After our cleanup, it lead to purge-datasets failing because it tries to remove a not existing file.
Tests done so far
On demo.data.gouv.fr:
upload and reupload a community resource through the admin: no problem
>>> for d in Dataset.objects:
... for r in d.resources:
... if r.fs_filename and (not r.url.endswith(r.fs_filename)):
... print(d.id, r.id, r.url, r.fs_filename)
...
5d13a8b6634f41070a43dff3 1ac234c7-1da4-49cf-a122-646b21d64b43 https://static.data.gouv.fr/resources/catalogue-des-donnees-de-data-gouv-fr/20200926-074922/export-tag-20200926-074922.csv catalogue-des-donnees-de-data-gouv-fr/20200822-074919/export-tag-20200822-074919.csv
5d13a8b6634f41070a43dff3 970aafa0-3778-4d8b-b9d1-de937525e379 https://static.data.gouv.fr/resources/catalogue-des-donnees-de-data-gouv-fr/20200926-074920/export-reuse-20200926-074920.csv catalogue-des-donnees-de-data-gouv-fr/20200822-074917/export-reuse-20200822-074917.csv
5d13a8b6634f41070a43dff3 b7bbfedc-2448-4135-a6c7-104548d396e7 https://static.data.gouv.fr/resources/catalogue-des-donnees-de-data-gouv-fr/20200926-074909/export-organization-20200926-074909.csv catalogue-des-donnees-de-data-gouv-fr/20200822-074906/export-organization-20200822-074906.csv
5d13a8b6634f41070a43dff3 d77705e1-4ecd-461c-8c24-662d47c4c2f9 https://static.data.gouv.fr/resources/catalogue-des-donnees-de-data-gouv-fr/20200926-074906/export-discussion-20200926-074906.csv catalogue-des-donnees-de-data-gouv-fr/20200822-074903/export-discussion-20200822-074903.csv
5d13a8b6634f41070a43dff3 4babf5f2-6a9c-45b5-9144-ca5eae6a7a6d https://static.data.gouv.fr/resources/catalogue-des-donnees-de-data-gouv-fr/20200926-074811/export-resource-20200926-074811.csv catalogue-des-donnees-de-data-gouv-fr/20200822-074809/export-resource-20200822-074809.csv
5d13a8b6634f41070a43dff3 f868cca6-8da1-4369-a78d-47463f19a9a3 https://static.data.gouv.fr/resources/catalogue-des-donnees-de-data-gouv-fr/20200926-074505/export-dataset-20200926-074505.csv catalogue-des-donnees-de-data-gouv-fr/20200822-074502/export-dataset-20200822-074502.csv
5448d3e0c751df01f85d0572 50625621-18bd-43cb-8fde-6b8c24bdabb3 https://static.data.gouv.fr/resources/fichier-consolide-des-bornes-de-recharge-pour-vehicules-electriques/20200920-224338/bornes-irve-20200920.csv fichier-consolide-des-bornes-de-recharge-pour-vehicules-electriques/20200820-224444/bornes-irve-20200820.csv
Cf #2542 for catalogue-des-donnees-de-data-gouv-fr. No idea why it failed for fichier-consolide-des-bornes-de-recharge-pour-vehicules-electriques, this is a pretty standard API upload.
The text was updated successfully, but these errors were encountered:
Going further: we probably should not allow setting the URL from the API when a resource is of type file (ie not remote). This would have prevented this whole mess.
Sometimes, especially on community resources from transport.data.gouv.fr, our
fs_filename
is not synced withurl
. After our cleanup, it lead topurge-datasets
failing because it tries to remove a not existing file.Tests done so far
On demo.data.gouv.fr:
CommunityResources
300+ occurrences, cf complete list https://gist.github.com/abulte/f283a2c2e3dc9102d8f767f0c908637e.
It happened again this morning, cf
Resources
Cf #2542 for
catalogue-des-donnees-de-data-gouv-fr
. No idea why it failed forfichier-consolide-des-bornes-de-recharge-pour-vehicules-electriques
, this is a pretty standard API upload.The text was updated successfully, but these errors were encountered: