Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Missing file at ckanext.bigquery.google_cloud_credentials #11

Open
ccancellieri opened this issue Oct 1, 2021 · 1 comment
Open

Missing file at ckanext.bigquery.google_cloud_credentials #11

ccancellieri opened this issue Oct 1, 2021 · 1 comment

Comments

@ccancellieri
Copy link

Hi,
I'm eager to use your plugin! could be a great asset!

After having installed and configured your master branch i get the follow:

ERROR [src.api_tracker] Invalid URL 'None': No schema supplied. Perhaps you meant http://None?
 /etc/ckan/default/${MY_PROJECT}-270XXXXX.json
 None
 Error - <type 'exceptions.TypeError'>: invalid file: None
 URL: http://127.0.0.1:8080/dataset/hih-datasets-view-templates-csv/resource/5d8ac57c-fb6c-4476-a872-fc577132b5a2
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/weberror/errormiddleware.py', line 171 in __call__
   app_iter = self.application(environ, sr_checker)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
   resp = self.call_func(req, *args, **self.kwargs)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
   return self.func(req, *args, **kwargs)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/fanstatic/publisher.py', line 234 in __call__
   return request.get_response(self.app)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
   application, catch_exc_info=False)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
   app_iter = application(self.environ, start_response)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
   resp = self.call_func(req, *args, **self.kwargs)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
   return self.func(req, *args, **kwargs)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/fanstatic/injector.py', line 54 in __call__
   response = request.get_response(self.app)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
   application, catch_exc_info=False)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
   app_iter = application(self.environ, start_response)
 File '/usr/lib/ckan/default/src/ckan/ckan/config/middleware/pylons_app.py', line 264 in inner
   result = application(environ, start_response)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/beaker/middleware.py', line 73 in __call__
   return self.app(environ, start_response)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/beaker/middleware.py', line 156 in __call__
   return self.wrap_app(environ, session_start_response)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/routes/middleware.py', line 131 in __call__
   response = self.app(environ, start_response)
 File '/usr/lib/ckan/default/src/ckan/ckan/config/middleware/common_middleware.py', line 33 in __call__
   return self.app(environ, start_response)
 File '/usr/lib/ckan/default/src/ckan/ckan/config/middleware/common_middleware.py', line 59 in __call__
   return self.app(environ, start_response)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/wsgiapp.py', line 125 in __call__
   response = self.dispatch(controller, environ, start_response)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/wsgiapp.py', line 324 in dispatch
   return controller(environ, start_response)
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 242 in __call__
   res = WSGIController.__call__(self, environ, start_response)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/controllers/core.py', line 221 in __call__
   response = self._dispatch_call()
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call
   response = self._inspect_call(func)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call
   result = self._perform_call(func, args)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/controllers/core.py', line 60 in _perform_call
   return func(**args)
 File '/usr/lib/ckan/default/src/ckan/ckan/controllers/package.py', line 1147 in resource_read
   return render(template, extra_vars=vars)
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 127 in render
   return cached_template(template_name, renderer)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/templating.py', line 249 in cached_template
   return render_func()
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 164 in render_template
   return render_jinja2(template_name, globs)
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 96 in render_jinja2
   return template.render(**extra_vars)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/jinja2/environment.py', line 989 in render
   return self.environment.handle_exception(exc_info, True)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/jinja2/environment.py', line 754 in handle_exception
   reraise(exc_type, exc_value, tb)
 File '/usr/lib/ckan/default/src/ckan/ckanext/datastore/templates/package/resource_read.html', line 1 in top-level template code
   {% ckan_extends %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/package/resource_read.html', line 3 in top-level template code
   {% set res = c.resource %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/package/base.html', line 3 in top-level template code
   {% set pkg = c.pkg_dict or pkg_dict %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/page.html', line 1 in top-level template code
   {% extends "base.html" %}
 File '/usr/lib/ckan/default/src/${MY_PREFIX}-ckanext-theme/ckanext/${CHANGE_ME}_theme/templates/base.html', line 1 in top-level template code
   {% ckan_extends %}
 File '/usr/lib/ckan/default/src/ckanext-scheming_dcat/ckanext/scheming_dcat/templates/base.html', line 1 in top-level template code
   {% ckan_extends %}
 File '/usr/lib/ckan/default/src/ckanext-googleanalytics/ckanext/googleanalytics/plugin/../templates/base.html', line 1 in top-level template code
   {% ckan_extends %}
 File '/usr/lib/ckan/default/src/ckanext-geoview/ckanext/geoview/plugin/../templates/base.html', line 1 in top-level template code
   {% ckan_extends %}
 File '/usr/lib/ckan/default/src/ckanext-scheming/ckanext/scheming/templates/base.html', line 1 in top-level template code
   {% ckan_extends %}
 File '/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/plugin/../templates/base.html', line 1 in top-level template code
   {% ckan_extends %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/base.html', line 101 in top-level template code
   {%- block page %}{% endblock -%}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/page.html', line 19 in block "page"
   {%- block content %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/page.html', line 22 in block "content"
   {% block main_content %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/page.html', line 53 in block "main_content"
   {% block pre_primary %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/package/resource_read.html', line 22 in block "pre_primary"
   {% block resource %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/package/resource_read.html', line 24 in block "resource"
   {% block resource_inner %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/package/resource_read.html', line 92 in block "resource_inner"
   {% block data_preview %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/package/resource_read.html', line 93 in block "data_preview"
   {% block resource_view %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/package/resource_read.html', line 106 in block "resource_view"
   {% block resource_view_content %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/package/resource_read.html', line 116 in block "resource_view_content"
   {% snippet 'package/snippets/resource_view.html',
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/jinja_extensions.py', line 268 in _call
   return base.render_snippet(args[0], **kwargs)
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 86 in render_snippet
   output = render(template_name, extra_vars=kw)
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 127 in render
   return cached_template(template_name, renderer)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/templating.py', line 249 in cached_template
   return render_func()
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 164 in render_template
   return render_jinja2(template_name, globs)
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 96 in render_jinja2
   return template.render(**extra_vars)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/jinja2/environment.py', line 989 in render
   return self.environment.handle_exception(exc_info, True)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/jinja2/environment.py', line 754 in handle_exception
   reraise(exc_type, exc_value, tb)
 File '/usr/lib/ckan/default/src/${MY_PREFIX}-ckanext-theme/ckanext/${CHANGE_ME}_theme/templates/package/snippets/resource_view.html', line 1 in top-level template code
   {% ckan_extends %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/package/snippets/resource_view.html', line 3 in top-level template code
   {% block resource_view %}
 File '/usr/lib/ckan/default/src/${MY_PREFIX}-ckanext-theme/ckanext/${CHANGE_ME}_theme/templates/package/snippets/resource_view.html', line 14 in block "resource_view"
   {{ super() }}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/package/snippets/resource_view.html', line 24 in block "resource_view"
   {% snippet 'package/snippets/resource_view_filters.html', resource=resource %}
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/jinja_extensions.py', line 268 in _call
   return base.render_snippet(args[0], **kwargs)
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 86 in render_snippet
   output = render(template_name, extra_vars=kw)
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 127 in render
   return cached_template(template_name, renderer)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/templating.py', line 249 in cached_template
   return render_func()
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 164 in render_template
   return render_jinja2(template_name, globs)
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 96 in render_jinja2
   return template.render(**extra_vars)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/jinja2/environment.py', line 989 in render
   return self.environment.handle_exception(exc_info, True)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/jinja2/environment.py', line 754 in handle_exception
   reraise(exc_type, exc_value, tb)
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/package/snippets/resource_view_filters.html', line 5 in top-level template code
   data-module-fields="{{ h.dump_json(h.resource_view_get_fields(resource)) }}"
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/helpers.py', line 2309 in resource_view_get_fields
   result = logic.get_action('datastore_search')({}, data)
 File '/usr/lib/ckan/default/src/ckan/ckan/logic/__init__.py', line 467 in wrapped
   result = _action(context, data_dict, **kw)
 File '/usr/lib/ckan/default/src/ckan/ckanext/datastore/logic/action.py', line 466 in datastore_search
   result = backend.search(context, data_dict)
 File '/usr/lib/ckan/default/src/ckanext-datastore-bigquery/ckanext/bigquery/backend/bigquery.py', line 43 in search
   engine = self._get_engine()
 File '/usr/lib/ckan/default/src/ckanext-datastore-bigquery/ckanext/bigquery/backend/bigquery.py', line 30 in _get_engine
   self._engine = ckan2bq.Client(project, dataset, creds, read_only_creds)
 File '/usr/lib/ckan/default/src/ckanext-datastore-bigquery/src/ckan_to_bigquery.py', line 22 in __init__
   self.bqclient_readonly = bigquery.Client.from_service_account_json(read_only_creds)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/google/cloud/client.py', line 77 in from_service_account_json
   with io.open(json_credentials_path, "r", encoding="utf-8") as json_fi:
 TypeError: invalid file: None
 
 
 CGI Variables
 -------------
   AUTH_TYPE: 'cookie'
   CKAN_CURRENT_URL: '/dataset/hih-datasets-view-templates-csv/resource/5d8ac57c-fb6c-4476-a872-XXXXXXXX'
   CKAN_LANG: 'en'
   CKAN_LANG_IS_DEFAULT: True
   CONTEXT_DOCUMENT_ROOT: '/var/www/html'
   DOCUMENT_ROOT: '/var/www/html'
   GATEWAY_INTERFACE: 'CGI/1.1'
   HTTP_ACCEPT: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
   HTTP_ACCEPT_ENCODING: 'gzip, deflate, br'
   HTTP_ACCEPT_LANGUAGE: 'en-US,en;q=0.9,it;q=0.8'
...
 WSGI Variables
 --------------
   __no_cache__: True
   apache.version: (2, 4, 29)
   application: <fanstatic.publisher.Delegator object at 0x7fe04c26f950>
   beaker.cache: <beaker.cache.CacheManager object at 0x7fe04c450690>
   beaker.get_session: <bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x7fe04c26f7d0>>
   beaker.session: {'_accessed_time': 1633119825.489249, '_creation_time': 1633104578.343546}
   ckan.app: 'pylons_app'
   fanstatic.needed: <fanstatic.core.NeededResources object at 0x7fe03ea53290>
   mod_wsgi.application_group: '${MY_PROJECT}-ckanx.{ZONE}.c.${MY_PROJECT}.internal:8080|/ckanx'
   mod_wsgi.callable_object: 'application'
   mod_wsgi.daemon_connects: '1'
   mod_wsgi.daemon_restarts: '0'
   mod_wsgi.daemon_start: '1633119825439052'
   mod_wsgi.enable_sendfile: '0'
   mod_wsgi.handler_script: ''
   mod_wsgi.ignore_activity: '0'
   mod_wsgi.listener_host: ''
   mod_wsgi.listener_port: '8080'
   mod_wsgi.path_info: '/dataset/hih-datasets-view-templates-csv/resource/5d8ac57c-fb6c-4476-a872-fc577132b5a2'
   mod_wsgi.process_group: 'ck

I've properly place my json file and granted right permissions.

Then configured production.ini with all the documented params plus:
ckanext.bigquery.google_cloud_credentials = /etc/ckan/default/{MY_PROJECT}-270XXXXXX.json

Will give the above log.

SOLVED:

Configuring the following undocumented param the file is found:

ckanext.bigquery.google_cloud_credentials_read_only= /etc/ckan/default/{MY_PROJECT}-270XXXXXX.json

At the moment this is not the best, would you accept a quick contribution to add an option to use the machine service account?
The machine has a service account configured so I don't need to manage this kind of secrets during the deploy.
If so would you like to suggest or give indications on the way you prefer?

thanks
C.
@ccancellieri
Copy link
Author

Probably the above depicted behaviour is due to a wrong ckan.datastore.write_url setting, but my (OT) question is still valid.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ccancellieri