Latin characters not accepted in JSON DataStore fields #2969
Description
CKAN version if known (or site URL)
2.3 (private deployment)
Please describe the expected behaviour
The following dataset upsert should return 200 OK:
> curl -X POST -k -H "Content-Type: application/json" -H "Accept: application/json" -H "Authorization: MY_API_KEY" -H "Cache-Control: no-cache" -d '{
"resource_id": "bba47654-df03-4c43-b2ff-b0f144223101",
"records": [
{
"recvTime": "2016-04-06T16:42:02.533333",
"temperature": "ñ"
}
],
"method": "insert",
"force": "true"
}' http://10.0.0.2:5055/api/3/action/datastore_upsert
Please describe the actual behaviour
500 Inernal Server Error
<html>
<head>
<title>Server Error</title>
</head>
<body>
<h1>Server Error</h1>
An internal server error occurred
</body>
</html>
The following traces were taken from the log:
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] Error - <type 'exceptions.UnicodeEncodeError'>: 'ascii' codec can't encode character u'\\xf1' in position 0: ordinal not in range(128)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] URL: https://192.168.21.64:8443/api/3/action/datastore_upsert
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/weberror/errormiddleware.py', line 162 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] app_iter = self.application(environ, sr_checker)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/dec.py', line 147 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] resp = self.call_func(req, *args, **self.kwargs)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/dec.py', line 208 in call_func
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] return self.func(req, *args, **kwargs)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/fanstatic/publisher.py', line 234 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] return request.get_response(self.app)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/request.py', line 1053 in get_response
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] application, catch_exc_info=False)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/request.py', line 1022 in call_application
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] app_iter = application(self.environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/dec.py', line 147 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] resp = self.call_func(req, *args, **self.kwargs)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/dec.py', line 208 in call_func
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] return self.func(req, *args, **kwargs)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/fanstatic/injector.py', line 54 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] response = request.get_response(self.app)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/request.py', line 1053 in get_response
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] application, catch_exc_info=False)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/request.py', line 1022 in call_application
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] app_iter = application(self.environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/beaker/middleware.py', line 73 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] return self.app(environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/beaker/middleware.py', line 155 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] return self.wrap_app(environ, session_start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/routes/middleware.py', line 131 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] response = self.app(environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/pylons/wsgiapp.py', line 125 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] response = self.dispatch(controller, environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/pylons/wsgiapp.py', line 324 in dispatch
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] return controller(environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckan/controllers/api.py', line 78 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] result = base.BaseController.__call__(self, environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckan/lib/base.py', line 338 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] res = WSGIController.__call__(self, environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/pylons/controllers/core.py', line 221 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] response = self._dispatch_call()
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] response = self._inspect_call(func)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/pylons/controllers/core.py', line 107 in _inspect_call
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] result = self._perform_call(func, args)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/pylons/controllers/core.py', line 60 in _perform_call
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] return func(**args)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckan/controllers/api.py', line 208 in action
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] result = function(context, request_data)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckan/logic/__init__.py', line 424 in wrapped
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] result = _action(context, data_dict, **kw)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckanext/datastore/logic/action.py', line 212 in datastore_upsert
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] result = db.upsert(context, data_dict)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckanext/datastore/db.py', line 1121 in upsert
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] upsert_data(context, data_dict)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckanext/datastore/db.py', line 642 in upsert_data
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] row.append(_to_full_text(fields, record))
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckanext/datastore/db.py', line 805 in _to_full_text
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] full_text.extend(json_get_values(value))
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckanext/datastore/db.py', line 258 in json_get_values
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] current_list.append(str(obj))
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] UnicodeEncodeError: 'ascii' codec can't encode character u'\\xf1' in position 0: ordinal not in range(128)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] CGI Variables
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] -------------
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] CKAN_CURRENT_URL: '/api/3/action/datastore_upsert'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] CKAN_LANG: 'en'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] CKAN_LANG_IS_DEFAULT: True
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] CONTENT_LENGTH: '206'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] CONTENT_TYPE: 'application/json; charset=utf-8'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] DOCUMENT_ROOT: '/var/www/html'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] GATEWAY_INTERFACE: 'CGI/1.1'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] HTTPS: '1'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] HTTP_ACCEPT: 'application/json'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] HTTP_AUTHORIZATION: '3d20cdae-44ad-4eeb-a8b7-8de485140133'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] HTTP_CACHE_CONTROL: 'no-cache'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] HTTP_CONNECTION: 'close'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] HTTP_HOST: '192.168.21.64:8443'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] HTTP_USER_AGENT: 'curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.18 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] PATH_INFO: '/api/3/action/datastore_upsert'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] PATH_TRANSLATED: '/etc/ckan/valencia/apache.wsgi/api/3/action/datastore_upsert'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] REMOTE_ADDR: '10.0.0.11'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] REMOTE_PORT: '52839'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] REQUEST_METHOD: 'POST'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] REQUEST_URI: '/api/3/action/datastore_upsert'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] SCRIPT_FILENAME: '/etc/ckan/valencia/apache.wsgi'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] SERVER_ADDR: '10.0.0.27'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] SERVER_ADMIN: 'root@localhost'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] SERVER_NAME: '192.168.21.64'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] SERVER_PORT: '8443'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] SERVER_PROTOCOL: 'HTTP/1.0'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] SERVER_SOFTWARE: 'Apache'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] WSGI Variables
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] --------------
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] application: <fanstatic.publisher.Delegator object at 0x7f69d1c19250>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] beaker.cache: <beaker.cache.CacheManager object at 0x7f69d1c19190>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] beaker.get_session: <bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x7f69d19a0750>>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] beaker.session: {'_accessed_time': 1460985026.8013899, '_creation_time': 1460985026.8013899}
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] fanstatic.needed: <fanstatic.core.NeededResources object at 0x7f69d1c18f10>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] mod_ssl.is_https: <built-in method ssl_is_https of mod_wsgi.Adapter object at 0x7f69cdc2e0a8>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] mod_ssl.var_lookup: <built-in method ssl_var_lookup of mod_wsgi.Adapter object at 0x7f69cdc2e0a8>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] mod_wsgi.application_group: 'ckan-valencia.ost:8443|'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] mod_wsgi.callable_object: 'application'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] mod_wsgi.handler_script: ''
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] mod_wsgi.input_chunked: '0'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] mod_wsgi.listener_host: ''
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] mod_wsgi.listener_port: '443'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] mod_wsgi.process_group: ''
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] mod_wsgi.request_handler: 'wsgi-script'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] mod_wsgi.script_reloading: '1'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] mod_wsgi.version: (3, 2)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] paste.cookies: (<SimpleCookie: >, '')
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] paste.registry: <paste.registry.Registry object at 0x7f69d1c18c90>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] paste.throw_errors: True
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] pylons.action_method: <bound method ApiController.action of <ckan.controllers.api.ApiController object at 0x7f69d1f51c10>>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] pylons.controller: <ckan.controllers.api.ApiController object at 0x7f69d1f51c10>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] pylons.environ_config: {'session': 'beaker.session', 'cache': 'beaker.cache'}
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] pylons.pylons: <pylons.util.PylonsContext object at 0x7f69d1c19210>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] pylons.routes_dict: {'action': u'action', 'controller': u'api', 'ver': 3, 'logic_function': u'datastore_upsert'}
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] pylons.status_code_redirect: True
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] repoze.who.api: <repoze.who.api.API object at 0x7f69d1c18cd0>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] repoze.who.logger: <logging.Logger instance at 0x7f69d1b2e7a0>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] repoze.who.plugins: {'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator object at 0x7f69d1f51cd0>, 'friendlyform': <FriendlyFormPlugin 140092465969168>, 'auth_tkt': <CkanAuthTktCookiePlugin 140092459979728>}
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] routes.route: <routes.route.Route object at 0x7f69d1f22350>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] routes.url: <routes.util.URLGenerator object at 0x7f69d1c19490>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] webob._parsed_query_vars: (GET([]), '')
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] webob.adhoc_attrs: {'response': <Response at 0x7f69d1ac3050 200 OK>, 'language': 'en-us'}
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] webob.is_body_seekable: True
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] wsgi process: 'Multiprocess'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] wsgi.file_wrapper: <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f69cdc2e0a8>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] wsgi.version: (1, 1)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] wsgiorg.routing_args: (<routes.util.URLGenerator object at 0x7f69d1c19490>, {'action': u'action', 'controller': u'api', 'ver': 3, 'logic_function': u'datastore_upsert'})
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] ------------------------------------------------------------
What steps can be taken to reproduce the issue?
Data upsert using a latin character: ñ, ç, á, etc.