Datapusher error when root_path is set

[AdrianMBarrera]

Hi,

After setting root_path variable in CKAN config file, Datapusher no longer works. We always get:

Error: Process completed but unable to post to result_url 

Contents of datapusher.error.log:

[Fri Feb 05 09:20:00.905675 2016] [:error] [pid 5997:tid 140341959157632] Exception AttributeError: "'NoneType' object has no attribute 'Error'" in <generator object raw at 0x7fa3d9384410> ignored
[Fri Feb 05 11:16:52.104411 2016] [:error] [pid 1502:tid 140224457529088] /usr/lib/ckan/datapusher/lib/python2.7/site-packages/sqlalchemy/sql/sqltypes.py:185: SAWarning: Unicode type received non-unicode bind param value 'ca8c86c6-ca87-4b36-b6bb-3...'. (this warning may be suppressed after 10 occurrences)
[Fri Feb 05 11:16:52.119255 2016] [:error] [pid 1502:tid 140224457529088]   (util.ellipses_string(value),))
[Fri Feb 05 11:16:57.624008 2016] [:error] [pid 1502:tid 140224457529088] Job "push_to_datastore (trigger: RunTriggerNow, run = True, next run at: None)" raised an exception
[Fri Feb 05 11:16:57.624065 2016] [:error] [pid 1502:tid 140224457529088] Traceback (most recent call last):
[Fri Feb 05 11:16:57.624096 2016] [:error] [pid 1502:tid 140224457529088]   File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/apscheduler/scheduler.py", line 512, in _run_job
[Fri Feb 05 11:16:57.624106 2016] [:error] [pid 1502:tid 140224457529088]     retval = job.func(*job.args, **job.kwargs)
[Fri Feb 05 11:16:57.624116 2016] [:error] [pid 1502:tid 140224457529088]   File "/usr/lib/ckan/datapusher/src/datapusher/datapusher/jobs.py", line 287, in push_to_datastore
[Fri Feb 05 11:16:57.624125 2016] [:error] [pid 1502:tid 140224457529088]     resource = get_resource(resource_id, ckan_url, api_key)
[Fri Feb 05 11:16:57.624134 2016] [:error] [pid 1502:tid 140224457529088]   File "/usr/lib/ckan/datapusher/src/datapusher/datapusher/jobs.py", line 233, in get_resource
[Fri Feb 05 11:16:57.624143 2016] [:error] [pid 1502:tid 140224457529088]     check_response(r, url, 'CKAN')
[Fri Feb 05 11:16:57.624151 2016] [:error] [pid 1502:tid 140224457529088]   File "/usr/lib/ckan/datapusher/src/datapusher/datapusher/jobs.py", line 144, in check_response
[Fri Feb 05 11:16:57.624160 2016] [:error] [pid 1502:tid 140224457529088]     response=response.text)
[Fri Feb 05 11:16:57.624168 2016] [:error] [pid 1502:tid 140224457529088] HTTPError

If we don't use the root_path variable and set our subpath in site_url then Datapusher works again but some CKAN extensions like archiver/qa get broken as they need the root_path variable to get the subpath.

We think this is related to #2599

Any help to fix this would be appreciated. Thank you in advance.

[amercader]

The result_url is the callback URL that CKAN tells the DataPusher to ping once it has finished its importing. It should point to the /api/3/action/datapusher_hook endpoint. Right now this URL is constructed using ckan.site_url and my gut feeling is that this is the correct approach.

I think there is definitely some confusion regarding ckan.site_url and ckan.root_path.
@wardi here you seem to suggest that ckan.site_url should not have path components but that's not what the docs say and a quick grep over the code shows that is generally used as being the full base url of the CKAN site (eg).

I'm not sure what ckan.root_path was intended for (I guess for some i18n stuff) but I think that if extensions are relying on it to build URLs that's probably wrong

http://docs.ckan.org/en/ckan-2.5.1/maintaining/configuration.html#ckan-site-url

[wardi]

@amercader here's where I learned about root_path: #2599 (comment)
Given that routes supports defining the structure of the complete url from hostname down with this variable, it's incorrect to build urls with site_url alone. We need to use url_for which should take root_path into account. I'd say the docs and other places in the code where urls are constructed manually need to be updated.

[AdrianMBarrera]

So, which is the best approach? Setting ckan.site_url (then some extensions don't work) or setting ckan.root_path (then DataPusher fails)?

[drnextgis]

I'm facing with the same issue.