Searching by full TimeStamp; Observation date search is not working

[rhinozD]

There is my data in Observation record:
"effectiveDateTime": "2019-11-28T19:00:06Z",
When I make a search request such as:
/Observation?date=2019-11-28T19:00:06Z
It is not working.
Error:
{ "resourceType": "OperationOutcome", "text": { "status": "generated", "div": "<div xmlns=\"http://www.w3.org/1999/xhtml\"><h1>Operation Outcome</h1><table border=\"0\"><tr><td style=\"font-weight: bold;\">error</td><td>[]</td><td><pre>Internal server error</pre></td></tr></table></div>" }, "issue": [ { "severity": "error", "code": "exception", "diagnostics": "Internal server error" } ] }

If I change to:
/Observation?date=2019-11-28
it returns:

{
    "resourceType": "Bundle",
    "id": "832febb9-ab25-4b5d-85cb-ba8448f23d76",
    "meta": {
        "lastUpdated": "2021-01-20T08:43:43.985Z"
    },
    "type": "searchset",
    "total": 0,
    "link": [
        {
            "relation": "self",
            "url": "https://<DOMAIN>.execute-api.<REGION>.amazonaws.com/dev/Observation?date=2019-11-28"
        }
    ],
    "entry": []
}

When I use yyyy-mm-dd format to search _lastUpdated, its work, but it seems not working with yyyy-mm-ddThh:mm:ss[Z|(+|-)hh:mm] format.

Please correct me if I was wrong somewhere.
Thanks in advance!!

[rsmayda]

Thank for bringing this up we are currently using dynamic mapping of Elastic Search indexes meaning that FWoA search parameters could be classified as a string instead of a date. Resulting in this odd behavior

We already have a issue open #129. I am in favor of combining the issues into one, but to confirm did I understand your issue correctly?

[rhinozD]

Sorry for making you confuse.
There is 2 problems:

1. About https://github.com/awslabs/fhir-works-on-aws-deployment/issues/129's issue,
I confirmed that I can search with: {{API_URL}}/Patient?birthdate=1987-02-20 on current FWoA version, but only {{API_URL}}/Observation?date=2015-10-11 still not work.

The FHIR document defined:
https://www.hl7.org/fhir/observation.html#search

I could not search by this search parameter.

2. Search based on dates only work with yyyy-mm-dd format(example: /Patient/_lastUpdated=2021-01-21T), but not work with yyyy-mm-ddThh:mm:ss[Z|(+|-)hh:mm] format(for example: /Patient/_lastUpdated=2021-01-21T01:12:02.086Z).

[rsmayda]

Ah okay thank you for explaining! Searching for time stamps has not been a known use case and we will put it on our backlog. For your first issue I'm not sure why you are getting a 500 could you look at your logs and share what the 500 is?

[rhinozD]

Ah okay thank you for explaining! Searching for time stamps has not been a known use case and we will put it on our backlog. For your first issue I'm not sure why you are getting a 500 could you look at your logs and share what the 500 is?

Sorry, my first issue was not explained clearly.
I mean it responses with 0 records.

With the information below I think I can search Observation resource by date search parameter, but I can't.

In the FHIR document, date search parameter was defined:

It means I can search by Observation.effective.
Observation.effective = Observation.effectiveDateTime | Observation.effectivePeriod | Observation.effectiveTiming | Observation.effectiveInstant

The {{API_URL}}/metadata request responses:

{
                    "type": "Observation",
                    ...
                    ""searchParam": [
                    ...
                    {
                            "name": "date",
                            "definition": "http://hl7.org/fhir/SearchParameter/clinical-date",
                            "type": "date",
                            "documentation": "Multiple Resources: \r\n\r\n* [AllergyIntolerance](allergyintolerance.html): Date first version of the resource instance was recorded\r\n* [CarePlan](careplan.html): Time period plan covers\r\n* [CareTeam](careteam.html): Time period team covers\r\n* [ClinicalImpression](clinicalimpression.html): When the assessment was documented\r\n* [Composition](composition.html): Composition editing time\r\n* [Consent](consent.html): When this Consent was created or indexed\r\n* [DiagnosticReport](diagnosticreport.html): The clinically relevant time of the report\r\n* [Encounter](encounter.html): A date within the period the Encounter lasted\r\n* [EpisodeOfCare](episodeofcare.html): The provided date search value falls within the episode of care's period\r\n* [FamilyMemberHistory](familymemberhistory.html): When history was recorded or last updated\r\n* [Flag](flag.html): Time period when flag is active\r\n* [Immunization](immunization.html): Vaccination  (non)-Administration Date\r\n* [List](list.html): When the list was prepared\r\n* [Observation](observation.html): Obtained date/time. If the obtained element is a period, a date that falls in the period\r\n* [Procedure](procedure.html): When the procedure was performed\r\n* [RiskAssessment](riskassessment.html): When was assessment made?\r\n* [SupplyRequest](supplyrequest.html): When the request was made\r\n"
                        }
                  ...
               }

The compiledSearchParameters.4.0.1.json defined:

Additional information:

• I can search Observation by date search parameter with HAPI FHIR Public Test Server.

Thank you for replying!

[carvantes]

Hi, Thanks again for the detailed issue report. I looked into it, there are 2 things that are wrong:

1. The 500 error is due to improper handling of reserved characters (: in this specific case)

"failed_shards": [
          {
            "shard": 0,
            "index": "observation",
            "node": "y4OiDdf7TxufLzZY96XMTA",
            "reason": {
              "type": "query_shard_exception",
              "reason": "Failed to parse query [2019-11-28T19:00:06Z]",
              "index_uuid": "KwM1-IHQQjq-kokhtuvMeA",
              "index": "observation",
              "caused_by": {
                "type": "parse_exception",
                "reason": "Cannot parse '2019-11-28T19:00:06Z': Encountered \" \":\" \": \"\" at line 1, column 16.\nWas expecting one of:\n    <EOF> \n    <AND> ...\n    <OR> ...\n    <NOT> ...\n    \"+\" ...\n    \"-\" ...\n    <BAREOPER> ...\n    \"(\" ...\n    \"*\" ...\n    \"^\" ...\n    <QUOTED> ...\n    <TERM> ...\n    <FUZZY_SLOP> ...\n    <PREFIXTERM> ...\n    <WILDTERM> ...\n    <REGEXPTERM> ...\n    \"[\" ...\n    \"{\" ...\n    <NUMBER> ...\n    ",
                "caused_by": {
                  "type": "parse_exception",
                  "reason": "Encountered \" \":\" \": \"\" at line 1, column 16.\nWas expecting one of:\n    <EOF> \n    <AND> ...\n    <OR> ...\n    <NOT> ...\n    \"+\" ...\n    \"-\" ...\n    <BAREOPER> ...\n    \"(\" ...\n    \"*\" ...\n    \"^\" ...\n    <QUOTED> ...\n    <TERM> ...\n    <FUZZY_SLOP> ...\n    <PREFIXTERM> ...\n    <WILDTERM> ...\n    <REGEXPTERM> ...\n    \"[\" ...\n    \"{\" ...\n    <NUMBER> ...\n    "
                }
              }
            }
          }
        ]

In ElasticSearch query_string queries, the reserved characters are: + - = && || > < ! ( ) { } [ ] ^ " ~ * ? : \ /. This actually affects also queries not related to dates (i.e. /Patient?name=a:b:c)

2. The compiled search parameter for Observation.effective is wrong. It is failing to consider that Observation.effective has multiple types. For Observation.effective[x], Type: dateTime|Period|Timing|instant. This leads to queries returning empty results since the field Observation.effective does not actually exist.

https://www.hl7.org/fhir/observation-definitions.html#Observation.effective_x_

[rsmayda]

With the above merges I believe we have addressed the issue here. Please re-open if needed!