Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

xLoader re-submit all resources when adding a new one to a dataset #75

Closed
pdelboca opened this issue Oct 13, 2019 · 6 comments
Closed

xLoader re-submit all resources when adding a new one to a dataset #75

pdelboca opened this issue Oct 13, 2019 · 6 comments

Comments

@pdelboca
Copy link
Member

pdelboca commented Oct 13, 2019 •
edited
Loading

How to Reproduce

  • Using a plain vainilla CKAN 2.8.3 instance with xloader 0.4.0 installed (I have used plain ofkn/docker-ckan but this one was also detected in a custom implementation for our customer)
  • Create a Dataset
  • Add a new Resource
  • Add another Resource
  • Check logs

Details

In this logs I had a dataset with a Resource called mini-csv.csv, then I edited the dataset to add a new one called mini-csv-2.csv. As the logs shows, the instance was running, I created the new resource but the two resources where submited:

ckan-dev_1    | 2019-10-13 13:14:49,387 INFO  [rq.worker] *** Listening on ckan:default:default...
ckan-dev_1    | 2019-10-13 13:15:27,562 INFO  [ckan.lib.base]  /dataset/edit/testing-xloader render time 0.209 seconds
ckan-dev_1    | 2019-10-13 13:15:29,448 INFO  [ckan.lib.base]  /dataset/resources/testing-xloader render time 0.176 seconds
ckan-dev_1    | 2019-10-13 13:15:32,241 INFO  [ckan.lib.base]  /dataset/new_resource/testing-xloader render time 0.134 seconds
ckan-dev_1    | 2019-10-13 13:15:45,179 DEBUG [ckanext.xloader.plugin] Submitting resource 8371b91c-b59e-46f9-b6da-f4abf9e37ec9 to be xloadered
ckan-dev_1    | 2019-10-13 13:15:45,214 INFO  [ckanext.xloader.action] Added background job bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f to queue "default"
ckan-dev_1    | 2019-10-13 13:15:45,214 DEBUG [ckanext.xloader.action] Enqueued xloader job=bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f res_id=8371b91c-b59e-46f9-b6da-f4abf9e37ec9
ckan-dev_1    | 2019-10-13 13:15:45,215 INFO  [rq.worker] ckan:default:default: ckanext.xloader.jobs.xloader_data_into_datastore({'result_url': u'http://ckan:5000/api/3/action/xloader_hook', 'api_key': u'3b32cbad-c8cc-4798-b927-d5828374f0a2', 'job_type': 'xloader_to_datastore', 'metadata': {'original_url': u'http://ckan:5000/dataset/bdc78b01-c24d-4416-a9ff-a6a1cc748e59/resource/8371b91c-b59e-46f9-b6da-f4abf9e37ec9/download/mini-csv-2.csv', 'ckan_url': u'http://ckan:5000', 'resource_id': u'8371b91c-b59e-46f9-b6da-f4abf9e37ec9', 'set_url_type': False, 'task_created': '2019-10-13 13:15:45.205091', 'ignore_hash': False}}) (bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f)
ckan-dev_1    | 2019-10-13 13:15:45,216 INFO  [ckan.lib.jobs] Worker rq:worker:6da86188ebe4.72 starts job bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f from queue "default"
ckan-dev_1    | 2019-10-13 13:15:45,221 DEBUG [ckanext.xloader.plugin] Submitting resource 77e90203-3cd7-4b46-a30d-779d2c4659f5 to be xloadered
ckan-dev_1    | 2019-10-13 13:15:45,258 INFO  [ckanext.xloader.action] Added background job 05803921-a69c-4b42-a430-89f1444a0290 to queue "default"
ckan-dev_1    | 2019-10-13 13:15:45,258 DEBUG [ckanext.xloader.action] Enqueued xloader job=05803921-a69c-4b42-a430-89f1444a0290 res_id=77e90203-3cd7-4b46-a30d-779d2c4659f5
ckan-dev_1    | 2019-10-13 13:15:45,428 INFO  [ckan.lib.base]  /dataset/new_resource/testing-xloader render time 0.341 seconds

Down belong the logs shows how xloader executed the jobs for both:

ckan-dev_1    | Express Load starting: /dataset/testing-xloader/resource/8371b91c-b59e-46f9-b6da-f4abf9e37ec9
ckan-dev_1    | 2019-10-13 13:15:45,700 INFO  [bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f] Express Load starting: /dataset/testing-xloader/resource/8371b91c-b59e-46f9-b6da-f4abf9e37ec9
ckan-dev_1    | Fetching from: http://ckan:5000/dataset/bdc78b01-c24d-4416-a9ff-a6a1cc748e59/resource/8371b91c-b59e-46f9-b6da-f4abf9e37ec9/download/mini-csv-2.csv
ckan-dev_1    | 2019-10-13 13:15:45,703 INFO  [bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f] Fetching from: http://ckan:5000/dataset/bdc78b01-c24d-4416-a9ff-a6a1cc748e59/resource/8371b91c-b59e-46f9-b6da-f4abf9e37ec9/download/mini-csv-2.csv
ckan-dev_1    | 2019-10-13 13:15:45,792 INFO  [ckan.lib.base]  /dataset/bdc78b01-c24d-4416-a9ff-a6a1cc748e59/resource/8371b91c-b59e-46f9-b6da-f4abf9e37ec9/download/mini-csv-2.csv render time 0.037 seconds
ckan-dev_1    | Downloaded ok - 51.0 bytes
ckan-dev_1    | 2019-10-13 13:15:45,801 INFO  [bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f] Downloaded ok - 51.0 bytes
ckan-dev_1    | File hash: b5ae401bd035a5be32a120c675592683
ckan-dev_1    | 2019-10-13 13:15:45,805 INFO  [bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f] File hash: b5ae401bd035a5be32a120c675592683
ckan-dev_1    | Loading CSV
ckan-dev_1    | 2019-10-13 13:15:45,830 INFO  [bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f] Loading CSV
ckan-dev_1    | Ensuring character coding is UTF8
ckan-dev_1    | 2019-10-13 13:15:45,837 INFO  [bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f] Ensuring character coding is UTF8
ckan-dev_1    | Fields: [{'type': 'text', 'id': 'column_1'}, {'type': 'text', 'id': 'column_2'}]
ckan-dev_1    | 2019-10-13 13:15:45,857 INFO  [bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f] Fields: [{'type': 'text', 'id': 'column_1'}, {'type': 'text', 'id': 'column_2'}]
ckan-dev_1    | 2019-10-13 13:15:45,868 INFO  [ckan.lib.base]  /dataset/testing-xloader render time 0.420 seconds
ckan-dev_1    | 2019-10-13 13:15:45,910 DEBUG [ckanext.datastore.logic.action] Setting datastore_active=True on resource 8371b91c-b59e-46f9-b6da-f4abf9e37ec9
ckan-dev_1    | Copying to database...
ckan-dev_1    | 2019-10-13 13:15:46,069 INFO  [bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f] Copying to database...
ckan-dev_1    | ...copying done
ckan-dev_1    | 2019-10-13 13:15:46,082 INFO  [bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f] ...copying done
ckan-dev_1    | Creating search index...
ckan-dev_1    | 2019-10-13 13:15:46,101 INFO  [bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f] Creating search index...
ckan-dev_1    | ...search index created
ckan-dev_1    | 2019-10-13 13:15:46,113 INFO  [bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f] ...search index created
ckan-dev_1    | Calculating record count (running ANALYZE on the table)
ckan-dev_1    | 2019-10-13 13:15:46,122 INFO  [bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f] Calculating record count (running ANALYZE on the table)
ckan-dev_1    | Setting resource.datastore_active = True
ckan-dev_1    | 2019-10-13 13:15:46,126 INFO  [bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f] Setting resource.datastore_active = True
ckan-dev_1    | Setting resource.datastore_contains_all_records_of_source_file = True
ckan-dev_1    | 2019-10-13 13:15:46,142 INFO  [bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f] Setting resource.datastore_contains_all_records_of_source_file = True
ckan-dev_1    | Data now available to users: /dataset/testing-xloader/resource/8371b91c-b59e-46f9-b6da-f4abf9e37ec9
ckan-dev_1    | 2019-10-13 13:15:46,374 INFO  [bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f] Data now available to users: /dataset/testing-xloader/resource/8371b91c-b59e-46f9-b6da-f4abf9e37ec9
ckan-dev_1    | Creating column indexes (a speed optimization for queries)...
ckan-dev_1    | 2019-10-13 13:15:46,399 INFO  [bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f] Creating column indexes (a speed optimization for queries)...
ckan-dev_1    | ...column indexes created.
ckan-dev_1    | 2019-10-13 13:15:46,439 INFO  [bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f] ...column indexes created.
ckan-dev_1    | Express Load completed
ckan-dev_1    | 2019-10-13 13:15:46,451 INFO  [bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f] Express Load completed
ckan-dev_1    | 2019-10-13 13:15:46,560 INFO  [rq.worker] ckan:default:default: Job OK (bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f)
ckan-dev_1    | 2019-10-13 13:15:46,560 INFO  [rq.worker] Result is kept for 500 seconds
ckan-dev_1    | 2019-10-13 13:15:46,565 INFO  [ckan.lib.jobs] Worker rq:worker:6da86188ebe4.72 has finished job bfd457e6-61f4-43d3-bcdf-4ea1dc31b54f from queue "default"
ckan-dev_1    | 2019-10-13 13:15:46,566 INFO  [rq.worker]
ckan-dev_1    | 2019-10-13 13:15:46,566 INFO  [rq.worker] *** Listening on ckan:default:default...
ckan-dev_1    | 2019-10-13 13:15:46,568 INFO  [rq.worker] ckan:default:default: ckanext.xloader.jobs.xloader_data_into_datastore({'result_url': u'http://ckan:5000/api/3/action/xloader_hook', 'api_key': u'3b32cbad-c8cc-4798-b927-d5828374f0a2', 'job_type': 'xloader_to_datastore', 'metadata': {'original_url': u'http://ckan:5000/dataset/bdc78b01-c24d-4416-a9ff-a6a1cc748e59/resource/77e90203-3cd7-4b46-a30d-779d2c4659f5/download/mini-csv.csv', 'ckan_url': u'http://ckan:5000', 'resource_id': u'77e90203-3cd7-4b46-a30d-779d2c4659f5', 'set_url_type': False, 'task_created': '2019-10-13 13:15:45.248526', 'ignore_hash': False}}) (05803921-a69c-4b42-a430-89f1444a0290)
ckan-dev_1    | 2019-10-13 13:15:46,569 INFO  [ckan.lib.jobs] Worker rq:worker:6da86188ebe4.72 starts job 05803921-a69c-4b42-a430-89f1444a0290 from queue "default"
ckan-dev_1    | Express Load starting: /dataset/testing-xloader/resource/77e90203-3cd7-4b46-a30d-779d2c4659f5
ckan-dev_1    | 2019-10-13 13:15:47,062 INFO  [05803921-a69c-4b42-a430-89f1444a0290] Express Load starting: /dataset/testing-xloader/resource/77e90203-3cd7-4b46-a30d-779d2c4659f5
ckan-dev_1    | Fetching from: http://ckan:5000/dataset/bdc78b01-c24d-4416-a9ff-a6a1cc748e59/resource/77e90203-3cd7-4b46-a30d-779d2c4659f5/download/mini-csv.csv
ckan-dev_1    | 2019-10-13 13:15:47,088 INFO  [05803921-a69c-4b42-a430-89f1444a0290] Fetching from: http://ckan:5000/dataset/bdc78b01-c24d-4416-a9ff-a6a1cc748e59/resource/77e90203-3cd7-4b46-a30d-779d2c4659f5/download/mini-csv.csv
ckan-dev_1    | 2019-10-13 13:15:47,133 INFO  [ckan.lib.base]  /dataset/bdc78b01-c24d-4416-a9ff-a6a1cc748e59/resource/77e90203-3cd7-4b46-a30d-779d2c4659f5/download/mini-csv.csv render time 0.027 seconds
ckan-dev_1    | Downloaded ok - 40.0 bytes
ckan-dev_1    | 2019-10-13 13:15:47,135 INFO  [05803921-a69c-4b42-a430-89f1444a0290] Downloaded ok - 40.0 bytes
ckan-dev_1    | File hash: a4c2cdeaefdb2c659151f0a64706ed5a
ckan-dev_1    | 2019-10-13 13:15:47,160 INFO  [05803921-a69c-4b42-a430-89f1444a0290] File hash: a4c2cdeaefdb2c659151f0a64706ed5a
ckan-dev_1    | Loading CSV
ckan-dev_1    | 2019-10-13 13:15:47,172 INFO  [05803921-a69c-4b42-a430-89f1444a0290] Loading CSV
ckan-dev_1    | Ensuring character coding is UTF8
ckan-dev_1    | 2019-10-13 13:15:47,195 INFO  [05803921-a69c-4b42-a430-89f1444a0290] Ensuring character coding is UTF8
ckan-dev_1    | Deleting "77e90203-3cd7-4b46-a30d-779d2c4659f5" from DataStore.
ckan-dev_1    | 2019-10-13 13:15:47,258 INFO  [05803921-a69c-4b42-a430-89f1444a0290] Deleting "77e90203-3cd7-4b46-a30d-779d2c4659f5" from DataStore.
ckan-dev_1    | 2019-10-13 13:15:47,274 DEBUG [ckanext.datastore.logic.action] Setting datastore_active=False on resource 77e90203-3cd7-4b46-a30d-779d2c4659f5
ckan-dev_1    | Fields: [{'type': 'text', 'id': 'column_1'}, {'type': 'text', 'id': 'column_2'}]
ckan-dev_1    | 2019-10-13 13:15:47,410 INFO  [05803921-a69c-4b42-a430-89f1444a0290] Fields: [{'type': 'text', 'id': 'column_1'}, {'type': 'text', 'id': 'column_2'}]
ckan-dev_1    | 2019-10-13 13:15:47,459 DEBUG [ckanext.datastore.logic.action] Setting datastore_active=True on resource 77e90203-3cd7-4b46-a30d-779d2c4659f5
ckan-dev_1    | Copying to database...
ckan-dev_1    | 2019-10-13 13:15:47,621 INFO  [05803921-a69c-4b42-a430-89f1444a0290] Copying to database...
ckan-dev_1    | ...copying done
ckan-dev_1    | 2019-10-13 13:15:47,639 INFO  [05803921-a69c-4b42-a430-89f1444a0290] ...copying done
ckan-dev_1    | Creating search index...
ckan-dev_1    | 2019-10-13 13:15:47,649 INFO  [05803921-a69c-4b42-a430-89f1444a0290] Creating search index...
ckan-dev_1    | ...search index created
ckan-dev_1    | 2019-10-13 13:15:47,662 INFO  [05803921-a69c-4b42-a430-89f1444a0290] ...search index created
ckan-dev_1    | Calculating record count (running ANALYZE on the table)
ckan-dev_1    | 2019-10-13 13:15:47,677 INFO  [05803921-a69c-4b42-a430-89f1444a0290] Calculating record count (running ANALYZE on the table)
ckan-dev_1    | Setting resource.datastore_active = True
ckan-dev_1    | 2019-10-13 13:15:47,683 INFO  [05803921-a69c-4b42-a430-89f1444a0290] Setting resource.datastore_active = True
ckan-dev_1    | Setting resource.datastore_contains_all_records_of_source_file = True
ckan-dev_1    | 2019-10-13 13:15:47,704 INFO  [05803921-a69c-4b42-a430-89f1444a0290] Setting resource.datastore_contains_all_records_of_source_file = True
ckan-dev_1    | Data now available to users: /dataset/testing-xloader/resource/77e90203-3cd7-4b46-a30d-779d2c4659f5
ckan-dev_1    | 2019-10-13 13:15:47,905 INFO  [05803921-a69c-4b42-a430-89f1444a0290] Data now available to users: /dataset/testing-xloader/resource/77e90203-3cd7-4b46-a30d-779d2c4659f5
ckan-dev_1    | Creating column indexes (a speed optimization for queries)...
ckan-dev_1    | 2019-10-13 13:15:47,916 INFO  [05803921-a69c-4b42-a430-89f1444a0290] Creating column indexes (a speed optimization for queries)...
ckan-dev_1    | ...column indexes created.
ckan-dev_1    | 2019-10-13 13:15:47,955 INFO  [05803921-a69c-4b42-a430-89f1444a0290] ...column indexes created.
ckan-dev_1    | Express Load completed
ckan-dev_1    | 2019-10-13 13:15:47,965 INFO  [05803921-a69c-4b42-a430-89f1444a0290] Express Load completed
ckan-dev_1    | 2019-10-13 13:15:48,027 INFO  [rq.worker] ckan:default:default: Job OK (05803921-a69c-4b42-a430-89f1444a0290)
ckan-dev_1    | 2019-10-13 13:15:48,028 INFO  [rq.worker] Result is kept for 500 seconds
ckan-dev_1    | 2019-10-13 13:15:48,032 INFO  [ckan.lib.jobs] Worker rq:worker:6da86188ebe4.72 has finished job 05803921-a69c-4b42-a430-89f1444a0290 from queue "default"
ckan-dev_1    | 2019-10-13 13:15:48,033 INFO  [rq.worker]
ckan-dev_1    | 2019-10-13 13:15:48,033 INFO  [rq.worker] *** Listening on ckan:default:default...

If I repeat the process and add a new file called mini-csv-3.csv it happens again, after render new_resource all three resources are being submited to xloader:

ckan-dev_1    | 2019-10-13 13:15:48,033 INFO  [rq.worker] *** Listening on ckan:default:default...
ckan-dev_1    | 2019-10-13 13:32:03,759 INFO  [ckan.lib.base]  /dataset/edit/testing-xloader render time 0.202 seconds
ckan-dev_1    | 2019-10-13 13:32:12,751 INFO  [ckan.lib.base]  /dataset/resources/testing-xloader render time 0.173 seconds
ckan-dev_1    | 2019-10-13 13:32:19,004 INFO  [ckan.lib.base]  /dataset/new_resource/testing-xloader render time 0.106 seconds
ckan-dev_1    | 2019-10-13 13:32:28,162 DEBUG [ckanext.xloader.plugin] Submitting resource 8037364f-eabc-444f-a2a2-13dc00fb4ba3 to be xloadered
ckan-dev_1    | 2019-10-13 13:32:28,197 INFO  [ckanext.xloader.action] Added background job 8ec95761-7eb5-4e05-acf6-396108094f88 to queue "default"
ckan-dev_1    | 2019-10-13 13:32:28,198 DEBUG [ckanext.xloader.action] Enqueued xloader job=8ec95761-7eb5-4e05-acf6-396108094f88 res_id=8037364f-eabc-444f-a2a2-13dc00fb4ba3
ckan-dev_1    | 2019-10-13 13:32:28,198 INFO  [rq.worker] ckan:default:default: ckanext.xloader.jobs.xloader_data_into_datastore({'result_url': u'http://ckan:5000/api/3/action/xloader_hook', 'api_key': u'3b32cbad-c8cc-4798-b927-d5828374f0a2', 'job_type': 'xloader_to_datastore', 'metadata': {'original_url': u'http://ckan:5000/dataset/bdc78b01-c24d-4416-a9ff-a6a1cc748e59/resource/8037364f-eabc-444f-a2a2-13dc00fb4ba3/download/mini-csv-3.csv', 'ckan_url': u'http://ckan:5000', 'resource_id': u'8037364f-eabc-444f-a2a2-13dc00fb4ba3', 'set_url_type': False, 'task_created': '2019-10-13 13:32:28.189102', 'ignore_hash': False}}) (8ec95761-7eb5-4e05-acf6-396108094f88)
ckan-dev_1    | 2019-10-13 13:32:28,199 INFO  [ckan.lib.jobs] Worker rq:worker:6da86188ebe4.72 starts job 8ec95761-7eb5-4e05-acf6-396108094f88 from queue "default"
ckan-dev_1    | 2019-10-13 13:32:28,204 DEBUG [ckanext.xloader.plugin] Submitting resource 77e90203-3cd7-4b46-a30d-779d2c4659f5 to be xloadered
ckan-dev_1    | 2019-10-13 13:32:28,245 INFO  [ckanext.xloader.action] Added background job a16ff460-695f-4f4a-8551-68d3974f6995 to queue "default"
ckan-dev_1    | 2019-10-13 13:32:28,245 DEBUG [ckanext.xloader.action] Enqueued xloader job=a16ff460-695f-4f4a-8551-68d3974f6995 res_id=77e90203-3cd7-4b46-a30d-779d2c4659f5
ckan-dev_1    | 2019-10-13 13:32:28,256 DEBUG [ckanext.xloader.plugin] Submitting resource 8371b91c-b59e-46f9-b6da-f4abf9e37ec9 to be xloadered
ckan-dev_1    | 2019-10-13 13:32:28,307 INFO  [ckanext.xloader.action] Added background job 4bf98711-d2ef-4e6f-8045-134580a0bcc2 to queue "default"
ckan-dev_1    | 2019-10-13 13:32:28,308 DEBUG [ckanext.xloader.action] Enqueued xloader job=4bf98711-d2ef-4e6f-8045-134580a0bcc2 res_id=8371b91c-b59e-46f9-b6da-f4abf9e37ec9
ckan-dev_1    | 2019-10-13 13:32:28,483 INFO  [ckan.lib.base]  /dataset/new_resource/testing-xloader render time 0.427 seconds
ckan-dev_1    | 2019-10-13 13:32:28,529 WARNI [ckan.lib.maintain] Function _resource_preview() in module ckan.controllers.package has been deprecated and will be removed in a later release of ckan. Resource preview is deprecated. Please use the new resource views
ckan-dev_1    | 2019-10-13 13:32:28,533 WARNI [ckan.lib.maintain] Function _resource_preview() in module ckan.controllers.package has been deprecated and will be removed in a later release of ckan. Resource preview is deprecated. Please use the new resource views
ckan-dev_1    | 2019-10-13 13:32:28,538 WARNI [ckan.lib.maintain] Function _resource_preview() in module ckan.controllers.package has been deprecated and will be removed in a later release of ckan. Resource preview is deprecated. Please use the new resource views

Some debug insights

I don't know exactly why is this happening but here are some insights from my debugging.

  • Each time a new resource is added to the dataset, the method lib.dictization.resource_dict_save is executed for each resource
  • Inside that method, this condition is executed due that when uploading a resource the __init__ method of ResourceUpload object sets the last_modified field.
  • So for each Uploaded Resource the setting the obj.url_changed = True will be set each time another resource is edited/added to the package.
  • This causes the notify() method of xloader to be called since it implements IResourceUrlChanged
@davidread
Copy link
Contributor

I agree, this seems to be a problem. It occurs only with resources which are uploaded files, rather than links. I'm afraid I've not done much testing with uploads before.

It doesn't seem right that IResourceUrlChanged fires for an uploaded resource every time that you save the dataset again in future. I wonder if that was unintended when @wardi implemented it: ckan/ckan@3e3a7c8 ?

@davidread
Copy link
Contributor

Ah, looks like this is being resolved here: ckan/ckan#5027

@pdelboca
Copy link
Member Author

@davidread yes! I have been debugging and working on this. The fix we are doing to CKAN will solve this behaviour but will worth noticing that this will still happening until it is backported to 2.8.3. (and will happen with all previous versions of CKAN)

This weekend I will do the PR in CKAN.

@pdelboca
Copy link
Member Author

@davidread The PR is created ckan/ckan#5033. I have test it locally and it solves this behaviour

Do you think it is worth it to add this info in the Troubleshooting section of the README.md file of xloader?

@davidread
Copy link
Contributor

Nice one!
Yes please to adding it to the Readme! You can mention that this is fixed in CKAN versions 2.6.9 2.7.7 2.8.4 and 2.9.0+

pdelboca added a commit to pdelboca/ckanext-xloader that referenced this issue Oct 23, 2019
@pdelboca
Document ckan#75 in Readme.rst
d64beb3
@pdelboca pdelboca mentioned this issue Oct 23, 2019
Merged
@pdelboca
Copy link
Member Author

pdelboca commented Oct 23, 2019 •
edited
Loading

Done! You can merge the PR #78 and close this issue!

davidread pushed a commit that referenced this issue Oct 25, 2019
Merge pull request #78 from pdelboca/75-add-info-to-readme-file
2ee5b5d 
Document #75 in README file
@lrromero lrromero mentioned this issue Dec 17, 2019
Closed
JVickery-TBS pushed a commit to JVickery-TBS/ckanext-xloader that referenced this issue Jan 15, 2024
@ThrawnCA
Merge pull request ckan#75 from qld-gov-au/QOLSVC-3902-reduce-deadlocks
6cae283 
[QOLSVC-3902] reduce lock timeout to avoid deadlocks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@davidread @pdelboca