Update placeholder # hack to work better

[whitlockjc]

As mentioned by @generikvault in #10 (comment), there is a "hack" for getting the path to the value(s) returned by jsonpath.Get(). This works great for wildcard paths but for absolute paths, the onus is on the jsonpath consumer to be able to accurately parse the provided JSONPath to be able to append to the provided path. Here is an example of some JSONPath values and the returned path using the aforementioned link:

JSONPath: $
  $
JSONPath: $.info
  $
JSONPath: $.paths[*]["get","put","post","delete","options","head","patch","trace"]
  $["/people/{personId}"]["delete"]
  $["/people"]["get"]
  $["/people"]["post"]
  $["/people/{personId}"]["get"]
  $["/people/{personId}"]["put"]
JSONPath: $.paths..parameters[*]
  $["/people"]["get"]["0"]
  $["/people"]["get"]["1"]
  $["/people/{personId}"]["0"]
JSONPath: $.components.parameters[*]
JSONPath: $.paths
  $
JSONPath: $.paths["/people"]
  $
JSONPath: $.paths["/people"].get.parameters[?(@.name == "pageSize")]
  $["0"]
JSONPath: $.paths["/people"].get.parameters[?(@.name == "pageToken")]
  $["1"]
JSONPath: $.paths["/people"].get.responses["200"].content["application/json"].schema.properties
  $

As you can tell, the path provided by the placeholder # hack is far from the actual path of the resolved value provided by jsonpath.Get(). Is there any possibility of extending the placeholder # hack to be more accurate for non-wildcard JSONPath values, or could there be an API exposed (or hack) that would make it where we can parse the provided JSONPath (consistently) so that we can add the proper suffix to the value provided by the placeholder # hack?

[whitlockjc]

Being open source, I'll take a peek and see if there is a PR I can craft up. But in the meantime, I'd love any sort of discussion around this.

[whitlockjc]

I'm starting to think this could be a feature request for this repo or https://github.com/PaesslerAG/gVal because I'm not sure there is a way to parse the JSONPath (complex enough to avoid doing at all/most costs) and use the placeholder hack to generate full paths for all resolved values. I think for simple cases (direct value resolution, single-segment wildcards, single-levels of recursion) it could be possible but more complex cases (recursion with segments after recursion, any level of nested recursion) fails based on my understanding. I think there are other cases that I remember being difficult/impossible but that's the gist of it.

I'll keep digging but as of right now, this is definitely not straight forward.

[whitlockjc]

To shed some light on this based on a slightly convoluted example (modified from https://jsonpath.com), here is an example JSON document and the example placeholders returned for a few JSONPath expressions:

JSON

{
  "firstName": "John",
  "lastName": "doe",
  "age": 26,
  "address": {
    "streetAddress": "naist street",
    "city": "Nara",
    "postalCode": "630-0192"
  },
  "phoneNumbers": [
    {
      "type": "iPhone",
      "number": "0123-4567-8888"
    },
    {
      "type": "home",
      "number": "0123-4567-8910"
    },
    {
      "miscellaneous": [
        {
          "type": "mobile",
          "number": "0123-4567-8910",
          "nested": [
            {
              "type": "work",
              "number": "0123-4567-8910"
            }
          ]
        }
      ]
    }
  ],
  "others": [
    {
      "number": {
        "details": {
          "type": "home",
          "content": "0123-4567-8910"
        }
      }
    },
    {
      "number": {
        "details": {
          "type": "work",
          "content": "0123-4567-8910"
        }
      }
    }
  ],
  "nested": {
    "others": [
      {
        "number": {
          "details": {
            "type": "home",
            "content": "0123-4567-8910"
          }
        }
      },
      {
        "number": {
          "details": {
            "type": "work",
            "content": "0123-4567-8910"
          }
        }
      }
    ]
  }
}

Results

jsonpath.Get($)
  * $
jsonpath.Get($..number)
  * $["others"]["1"]
  * $["phoneNumbers"]["0"]
  * $["phoneNumbers"]["1"]
  * $["phoneNumbers"]["2"]["miscellaneous"]["0"]
  * $["phoneNumbers"]["2"]["miscellaneous"]["0"]["nested"]["0"]
  * $["nested"]["others"]["0"]
  * $["nested"]["others"]["1"]
  * $["others"]["0"]
jsonpath.Get($.phoneNumbers[?(@.type == "home")].number)
  * $["1"]
jsonpath.Get($..phoneNumbers..number)
  * $["0"]
  * $["1"]
  * $["2"]["miscellaneous"]["0"]
  * $["2"]["miscellaneous"]["0"]["nested"]["0"]
jsonpath.Get($.phoneNumbers..number)
  * $["0"]
  * $["1"]
  * $["2"]["miscellaneous"]["0"]
  * $["2"]["miscellaneous"]["0"]["nested"]["0"]
jsonpath.Get($.phoneNumbers[*].number)
  * $["0"]
  * $["1"]
jsonpath.Get($.others..details.type)
  * $["1"]["number"]
  * $["0"]["number"]
jsonpath.Get($..number..type)
  * $["nested"]["others"]["0"]["details"]
  * $["nested"]["others"]["1"]["details"]
  * $["others"]["0"]["details"]
  * $["others"]["1"]["details"]

Based on these examples (convoluted, I know...but I'm trying to identify edge cases), assuming I could correctly parse the JSONPath, I could reconstruct the actual path for all non-multi-nested expressions. But for multi-nested expressions (Example: $..number..type), I can't tell which segments belong to the first .. and which belong to the second ...

[whitlockjc]

Actually, https://github.com/PaesslerAG/jsonpath/blob/master/jsonpath.go#L7 may have just given me a possibility, but if I understand things, it won't be ideal due to the fact you'd have to evaluate the JSONPath expression N times (where N is the number of wildcards).

[whitlockjc]

After a good deal of mucking around, it seems like the context between the JSONPath parsing and the gVal execution is different, so attempting to update parse.go and placeholder.go to work in uniform isn't possible from what I can tell. I can get parse.go to keep track of the path segments it knows of and I can get placeholder.go to keep track of the wildcard path segments, but tying them together appropriately hasn't worked (yet).

If anyone more knowledgeable on gVal and/or jsonpath can share some ideas, let me know. I'm going to punt for a bit, I've been stuck on this for a few days and I need to make progress elsewhere. @generikvault maybe?

[whitlockjc]

Okay, I just found out about gval.NewEvaluableWithContext...gonna give this a shot.

[whitlockjc]

Using gval.NewEvaluableWithContext doesn't seem to yield any success, the disconnect is still there.

[whitlockjc]

What if we just created a new API like GetWithPaths that works like the placeholder hack but it returns a map of full paths to resolved values? I'll look into this, but figuring out the best way to keep track of path segments for nested selectors has been painful thus far.

[whitlockjc]

Okay, I will have a PR together that should handle this and it does not use the "placeholder hack". While I would also consider this to be a hack of sorts, it works and instead of hoping to share a common scope to keep track of these automatically, I resort to a much simpler approach. Here is the process:

1. At parse time, turn the provided JSONPath into a more sanitized path replacing ambiguous selectors with *. (For example, $.phoneNumbers[*].number would become ["phoneNumbers", "*", "number"], and $..number would become ["*", "number"].)
2. parse.go was updated with the following:
   a. parser was updated to have a sanitizedPath set at parse time
   b. .parseRootPath sets a context variable (computePathsContextKey{}) to true
   c. .parseCurrentPath sets a context variable (computePathsContextKey{}) to false
   d. parser.parse was updated to return a closure that checks the value of the aforementioned context variable and returns either the raw value (parseCurrentPath) or a map whose keys are the full JSONPath to the matched value and whose values are the matched value.
3. jsonpath.go was updated to have a new GetWithPaths function that will return a map (key is the quoted JSONPath similar to the placeholder approach, value is the resolved value) (thanks to a helper method that makes it where Get and GetWithPaths work appropriately).

For the example above, here are the new keys (matched values are identical to Get prior to these changes):

jsonpath.Get($)
  $
jsonpath.Get($.phoneNumbers[0].type)
  $["phoneNumbers"]["0"]["type"]
jsonpath.Get($.phoneNumbers..nested[0].number)
  $["phoneNumbers"]["2"]["miscellaneous"]["0"]["nested"]["0"]["number"]
jsonpath.Get($.phoneNumbers[*])
  $["phoneNumbers"]["0"]
  $["phoneNumbers"]["1"]
  $["phoneNumbers"]["2"]
jsonpath.Get($.phoneNumbers)
  $["phoneNumbers"]
jsonpath.Get($..number)
  $["nested"]["others"]["1"]["number"]
  $["others"]["0"]["number"]
  $["others"]["1"]["number"]
  $["phoneNumbers"]["0"]["number"]
  $["phoneNumbers"]["1"]["number"]
  $["phoneNumbers"]["2"]["miscellaneous"]["0"]["number"]
  $["phoneNumbers"]["2"]["miscellaneous"]["0"]["nested"]["0"]["number"]
  $["nested"]["others"]["0"]["number"]
jsonpath.Get($.phoneNumbers[?(@.type == "home")].number)
  $["phoneNumbers"]["1"]["number"]
jsonpath.Get($..phoneNumbers..number)
  $["phoneNumbers"]["2"]["miscellaneous"]["0"]["nested"]["0"]["number"]
  $["phoneNumbers"]["0"]["number"]
  $["phoneNumbers"]["1"]["number"]
  $["phoneNumbers"]["2"]["miscellaneous"]["0"]["number"]
jsonpath.Get($.phoneNumbers..number)
  $["phoneNumbers"]["0"]["number"]
  $["phoneNumbers"]["1"]["number"]
  $["phoneNumbers"]["2"]["miscellaneous"]["0"]["number"]
  $["phoneNumbers"]["2"]["miscellaneous"]["0"]["nested"]["0"]["number"]
jsonpath.Get($.phoneNumbers[*].number)
  $["phoneNumbers"]["0"]["number"]
  $["phoneNumbers"]["1"]["number"]
jsonpath.Get($.others..details.type)
  $["others"]["0"]["number"]["details"]["type"]
  $["others"]["1"]["number"]["details"]["type"]
jsonpath.Get($.others..details["type"])
  $["others"]["0"]["number"]["details"]["type"]
  $["others"]["1"]["number"]["details"]["type"]
jsonpath.Get($..number..type)
  $["others"]["1"]["number"]["details"]["type"]
  $["nested"]["others"]["0"]["number"]["details"]["type"]
  $["nested"]["others"]["1"]["number"]["details"]["type"]
  $["others"]["0"]["number"]["details"]["type"]

[whitlockjc]

I ended up with a much simpler approach, although sanitizing the paths returned by ambiguousPath required a little extra work to figure out. All-in-all, I think PR #44 is in decent shape. Let me know what your thoughts are.