Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validation Error on datastore_search when sorting timestamp fields #4288

Merged
merged 1 commit into from Jul 20, 2018

Conversation

Projects
None yet
4 participants
@wardi
Copy link
Contributor

commented Jun 24, 2018

CKAN Version if known (or site URL)

2.7+

Please describe the expected behaviour

The datastore_search API should be able to perform a distinct or sort on fields with a timestamp data type. A data table or data explorer resource view should be able to reorder rows based on a timestamp field.

Please describe the actual behaviour

A validation error is returned from the datastore_search API when performing a distinct or sort on fields with a timestamp data type. The data table and data explorer view can't reorder rows based on a timestamp field.

The problem seems to be that timestamps are converted to char than to json before sorting.
https://github.com/ckan/ckan/blob/ckan-2.7.3/ckanext/datastore/backend/postgres.py#L1726-L1729

What steps can be taken to reproduce the issue?

Call the datastore_search endpoint with distinct=true and sorting on a timestamp field. /api/3/action/datastore_search?resource_id={resource_id}&fields="DateOpened"&sort="DateOpened"&distinct=true

Returned Statement

SELECT array_to_json(array_agg(j))::text FROM (
  SELECT DISTINCT to_json(to_char("DateOpened", 'YYYY-MM-DD"T"HH24:MI:SS')) as "DateOpened"
  FROM "resource_id" 
  ORDER BY "DateOpened" asc LIMIT 100 OFFSET 0
) AS j"

Returned Validation Error

Could not identify an ordering operator for type json
LINE 5:  ORDER BY "DateOpened" asc LIMIT 100 OFFSET...
HINT:  Use an explicit ordering operator or modify the query.

@wardi wardi self-assigned this Jun 11, 2018

@wardi

This comment has been minimized.

Copy link
Contributor

commented Jun 11, 2018

datastore_search needs some expanded tests to cover this case and a bunch of other combinations of types, sorting and records_format values to catch all the edge cases

@amercader

This comment has been minimized.

Copy link
Member

commented Jul 18, 2018

@smotornyuk can you prioritize this review so we can get this on 2.8.1? cheers

@amercader amercader merged commit 600be27 into master Jul 20, 2018

4 checks passed

ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
security/snyk - requirements.txt (davidread) No manifest changes detected

tino097 added a commit that referenced this pull request Jul 20, 2018

jguo144 added a commit to OpenGov-OpenData/ckan that referenced this pull request Aug 20, 2018

@smotornyuk smotornyuk deleted the 4288-datastore-tests branch Dec 19, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.