Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Latin characters not accepted in JSON DataStore fields #2969

Closed
frbattid opened this issue Apr 19, 2016 · 6 comments

Comments

Projects
None yet
3 participants
@frbattid
Copy link

commented Apr 19, 2016

CKAN version if known (or site URL)

2.3 (private deployment)

Please describe the expected behaviour

The following dataset upsert should return 200 OK:

> curl -X POST -k -H "Content-Type: application/json" -H "Accept: application/json" -H "Authorization: MY_API_KEY" -H "Cache-Control:  no-cache" -d '{
  "resource_id": "bba47654-df03-4c43-b2ff-b0f144223101",
  "records": [
    {
      "recvTime": "2016-04-06T16:42:02.533333",
      "temperature": "ñ"
    }
  ],
  "method": "insert",
  "force": "true"
}' http://10.0.0.2:5055/api/3/action/datastore_upsert

Please describe the actual behaviour

500 Inernal Server Error

<html>
    <head>
    <title>Server Error</title>

    </head>
    <body>
    <h1>Server Error</h1>
    An internal server error occurred

    </body>
</html>

The following traces were taken from the log:

[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] Error - <type 'exceptions.UnicodeEncodeError'>: 'ascii' codec can't encode character u'\\xf1' in position 0: ordinal not in range(128)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] URL: https://192.168.21.64:8443/api/3/action/datastore_upsert
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/weberror/errormiddleware.py', line 162 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   app_iter = self.application(environ, sr_checker)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/dec.py', line 147 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   resp = self.call_func(req, *args, **self.kwargs)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/dec.py', line 208 in call_func
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   return self.func(req, *args, **kwargs)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/fanstatic/publisher.py', line 234 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   return request.get_response(self.app)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/request.py', line 1053 in get_response
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   application, catch_exc_info=False)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/request.py', line 1022 in call_application
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   app_iter = application(self.environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/dec.py', line 147 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   resp = self.call_func(req, *args, **self.kwargs)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/dec.py', line 208 in call_func
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   return self.func(req, *args, **kwargs)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/fanstatic/injector.py', line 54 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   response = request.get_response(self.app)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/request.py', line 1053 in get_response
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   application, catch_exc_info=False)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/request.py', line 1022 in call_application
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   app_iter = application(self.environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/beaker/middleware.py', line 73 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   return self.app(environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/beaker/middleware.py', line 155 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   return self.wrap_app(environ, session_start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/routes/middleware.py', line 131 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   response = self.app(environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/pylons/wsgiapp.py', line 125 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   response = self.dispatch(controller, environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/pylons/wsgiapp.py', line 324 in dispatch
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   return controller(environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckan/controllers/api.py', line 78 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   result =  base.BaseController.__call__(self, environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckan/lib/base.py', line 338 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   res = WSGIController.__call__(self, environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/pylons/controllers/core.py', line 221 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   response = self._dispatch_call()
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   response = self._inspect_call(func)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/pylons/controllers/core.py', line 107 in _inspect_call
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   result = self._perform_call(func, args)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/pylons/controllers/core.py', line 60 in _perform_call
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   return func(**args)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckan/controllers/api.py', line 208 in action
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   result = function(context, request_data)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckan/logic/__init__.py', line 424 in wrapped
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   result = _action(context, data_dict, **kw)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckanext/datastore/logic/action.py', line 212 in datastore_upsert
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   result = db.upsert(context, data_dict)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckanext/datastore/db.py', line 1121 in upsert
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   upsert_data(context, data_dict)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckanext/datastore/db.py', line 642 in upsert_data
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   row.append(_to_full_text(fields, record))
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckanext/datastore/db.py', line 805 in _to_full_text
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   full_text.extend(json_get_values(value))
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckanext/datastore/db.py', line 258 in json_get_values
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   current_list.append(str(obj))
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] UnicodeEncodeError: 'ascii' codec can't encode character u'\\xf1' in position 0: ordinal not in range(128)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] CGI Variables
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] -------------
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   CKAN_CURRENT_URL: '/api/3/action/datastore_upsert'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   CKAN_LANG: 'en'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   CKAN_LANG_IS_DEFAULT: True
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   CONTENT_LENGTH: '206'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   CONTENT_TYPE: 'application/json; charset=utf-8'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   DOCUMENT_ROOT: '/var/www/html'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   GATEWAY_INTERFACE: 'CGI/1.1'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   HTTPS: '1'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   HTTP_ACCEPT: 'application/json'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   HTTP_AUTHORIZATION: '3d20cdae-44ad-4eeb-a8b7-8de485140133'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   HTTP_CACHE_CONTROL: 'no-cache'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   HTTP_CONNECTION: 'close'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   HTTP_HOST: '192.168.21.64:8443'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   HTTP_USER_AGENT: 'curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.18 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   PATH_INFO: '/api/3/action/datastore_upsert'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   PATH_TRANSLATED: '/etc/ckan/valencia/apache.wsgi/api/3/action/datastore_upsert'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   REMOTE_ADDR: '10.0.0.11'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   REMOTE_PORT: '52839'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   REQUEST_METHOD: 'POST'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   REQUEST_URI: '/api/3/action/datastore_upsert'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   SCRIPT_FILENAME: '/etc/ckan/valencia/apache.wsgi'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   SERVER_ADDR: '10.0.0.27'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   SERVER_ADMIN: 'root@localhost'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   SERVER_NAME: '192.168.21.64'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   SERVER_PORT: '8443'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   SERVER_PROTOCOL: 'HTTP/1.0'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   SERVER_SOFTWARE: 'Apache'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] WSGI Variables
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] --------------
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   application: <fanstatic.publisher.Delegator object at 0x7f69d1c19250>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   beaker.cache: <beaker.cache.CacheManager object at 0x7f69d1c19190>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   beaker.get_session: <bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x7f69d19a0750>>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   beaker.session: {'_accessed_time': 1460985026.8013899, '_creation_time': 1460985026.8013899}
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   fanstatic.needed: <fanstatic.core.NeededResources object at 0x7f69d1c18f10>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_ssl.is_https: <built-in method ssl_is_https of mod_wsgi.Adapter object at 0x7f69cdc2e0a8>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_ssl.var_lookup: <built-in method ssl_var_lookup of mod_wsgi.Adapter object at 0x7f69cdc2e0a8>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_wsgi.application_group: 'ckan-valencia.ost:8443|'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_wsgi.callable_object: 'application'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_wsgi.handler_script: ''
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_wsgi.input_chunked: '0'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_wsgi.listener_host: ''
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_wsgi.listener_port: '443'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_wsgi.process_group: ''
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_wsgi.request_handler: 'wsgi-script'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_wsgi.script_reloading: '1'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_wsgi.version: (3, 2)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   paste.cookies: (<SimpleCookie: >, '')
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   paste.registry: <paste.registry.Registry object at 0x7f69d1c18c90>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   paste.throw_errors: True
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   pylons.action_method: <bound method ApiController.action of <ckan.controllers.api.ApiController object at 0x7f69d1f51c10>>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   pylons.controller: <ckan.controllers.api.ApiController object at 0x7f69d1f51c10>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   pylons.environ_config: {'session': 'beaker.session', 'cache': 'beaker.cache'}
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   pylons.pylons: <pylons.util.PylonsContext object at 0x7f69d1c19210>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   pylons.routes_dict: {'action': u'action', 'controller': u'api', 'ver': 3, 'logic_function': u'datastore_upsert'}
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   pylons.status_code_redirect: True
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   repoze.who.api: <repoze.who.api.API object at 0x7f69d1c18cd0>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   repoze.who.logger: <logging.Logger instance at 0x7f69d1b2e7a0>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   repoze.who.plugins: {'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator object at 0x7f69d1f51cd0>, 'friendlyform': <FriendlyFormPlugin 140092465969168>, 'auth_tkt': <CkanAuthTktCookiePlugin 140092459979728>}
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   routes.route: <routes.route.Route object at 0x7f69d1f22350>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   routes.url: <routes.util.URLGenerator object at 0x7f69d1c19490>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   webob._parsed_query_vars: (GET([]), '')
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   webob.adhoc_attrs: {'response': <Response at 0x7f69d1ac3050 200 OK>, 'language': 'en-us'}
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   webob.is_body_seekable: True
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   wsgi process: 'Multiprocess'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   wsgi.file_wrapper: <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f69cdc2e0a8>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   wsgi.version: (1, 1)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   wsgiorg.routing_args: (<routes.util.URLGenerator object at 0x7f69d1c19490>, {'action': u'action', 'controller': u'api', 'ver': 3, 'logic_function': u'datastore_upsert'})
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] ------------------------------------------------------------

What steps can be taken to reproduce the issue?

Data upsert using a latin character: ñ, ç, á, etc.

@amercader amercader self-assigned this Apr 21, 2016

@amercader

This comment has been minimized.

Copy link
Member

commented Apr 21, 2016

@frbattid Can you check if the following patch fixes the issue for you?

diff --git a/ckanext/datastore/db.py b/ckanext/datastore/db.py
index 1121c9e..895971c 100644
--- a/ckanext/datastore/db.py
+++ b/ckanext/datastore/db.py
@@ -255,7 +255,7 @@ def json_get_values(obj, current_list=None):
     elif isinstance(obj, dict):
         json_get_values(obj.items(), current_list)
     elif obj:
-        current_list.append(str(obj))
+        current_list.append(unicode(obj))
     return current_list

Also can you share the table structure of that DataStore resource? Something like:

> curl -H "Authorization: MY_API_KEY" http://10.0.0.2:5055/api/3/action/datastore_search

It's really bizarre because I can see the problem on the stack trace you pasted, but I didn't manage to reproduce the issue, ie I couldn't make that code execute with the same request.

@manucarrace

This comment has been minimized.

Copy link

commented Apr 21, 2016

@amercader this is what we got:

curl -h "Authorization: MY_API_KEY" http://10.0.0.2:5055/api/action/datastore_search?resource_id=bba47654-df03-4c43-b2ff-b0f144223101&limit=1

{
  "help": "https://192.168.21.64:8443/api/3/action/help_show?name=datastore_search",
  "success": true,
  "result": {
    "resource_id": "bba47654-df03-4c43-b2ff-b0f144223101",
    "fields": [
      {
        "type": "int4",
        "id": "_id"
      },
      {
        "type": "timestamp",
        "id": "recvTime"
      },
      {
        "type": "json",
        "id": "fiwareServicePath"
      },
      {
        "type": "json",
        "id": "entityType"
      },
      {
        "type": "json",
        "id": "entityId"
      },
      {
        "type": "timestamp",
        "id": "TimeInstant"
      },
      {
        "type": "json",
        "id": "temperature"
      },
      {
        "type": "json",
        "id": "temperature_md"
      },
      {
        "type": "json",
        "id": "pressure"
      },
      {
        "type": "json",
        "id": "pressure_md"
      },
      {
        "type": "json",
        "id": "presence"
      },
      {
        "type": "json",
        "id": "presence_md"
      },
      {
        "type": "json",
        "id": "humidity"
      },
      {
        "type": "json",
        "id": "humidity_md"
      },
      {
        "type": "json",
        "id": "fillLevel"
      },
      {
        "type": "json",
        "id": "fillLevel_md"
      }
    ],
    "records": [
      {
        "TimeInstant": null,
        "recvTime": "2016-03-15T11:52:37.670000",
        "temperature": "5701",
        "presence": null,
        "pressure_md": null,
        "fillLevel": null,
        "entityType": "room",
        "humidity": null,
        "pressure": null,
        "humidity_md": null,
        "fiwareServicePath": "electricidad",
        "entityId": "room008",
        "_id": 1,
        "temperature_md": null,
        "presence_md": null,
        "fillLevel_md": null
      }
    ],
    "_links": {
      "start": "/api/action/datastore_search?limit=1&resource_id=bba47654-df03-4c43-b2ff-b0f144223101",
      "next": "/api/action/datastore_search?offset=1&limit=1&resource_id=bba47654-df03-4c43-b2ff-b0f144223101"
    },
    "limit": 1,
    "total": 81534
  }
}
@amercader

This comment has been minimized.

Copy link
Member

commented Apr 22, 2016

@frbattid That what I imagined, as the fields have a "json" type. Is there a reason why fields like "Temperature" are stored as JSON? Are they actually JSON values?

Of course the original issue remains, so can you confirm if the patch I pasted on my last message fixes it?

@amercader

This comment has been minimized.

Copy link
Member

commented Apr 27, 2016

@manucarrace

This comment has been minimized.

Copy link

commented Apr 27, 2016

@amercader our fields can be a simple value or a compund value (json value), which is the case we need the type json.

On the other hand, I confirm the patch works for us. Do you plan to include it in further releases?

Thank you in advance

amercader added a commit that referenced this issue Apr 28, 2016

@amercader amercader changed the title Latin characters not accepted in CKAN 2.3 Latin characters not accepted in JSON DataStore fields Apr 28, 2016

@amercader

This comment has been minimized.

Copy link
Member

commented Apr 28, 2016

@manucarrace yes, I've created a PR with the fix (#2985) and will backport it once it gets merged

@wardi wardi closed this in #2985 May 12, 2016

wardi added a commit that referenced this issue May 12, 2016

Merge pull request #2985 from ckan/2969-fix-unicode-datastore-indexing
[#2969] Fix unicode error when indexing JSON DataStore fields

amercader added a commit that referenced this issue Aug 26, 2016

amercader added a commit that referenced this issue Aug 26, 2016

[#2969] Add a couple of tests
Conflicts:
	ckanext/datastore/tests/test_upsert.py

amercader added a commit that referenced this issue Aug 26, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.