Accessing the index of current array element within "unnest" subquery

[philrz]

In a recent community Slack thread, a user was manipulating an array in an over scope in a way that required knowing the index of each value. I responded with an example that achieved this using count(), to which @mattnibs commented:

I kind of think having to using streaming count() for this case here is kind of a hack and that you should be able to access the index of the current element from within the over scope. The hard part is coming up with a elegant way of doing this...

Details

Repro is with Zed commit 6e61673.

The specific question posed by the user:

is there a way to change this: ["a","b"] to [{"a":1},{"b":2}]based on the index location of each item? (for any length array...)

My response:

$ zq -version
Version: v1.16.0-15-g6e61673c

$ echo '["a","b","c"]' | zq -j '
over this => (yield [{key:this,value:count()}] | unflatten(this))
| collect(this)
' -
[{"a":1},{"b":2},{"c":3}]

The user noted how Ruby provides this via each_with_index in Ruby.

[philrz]

I bounced this one off @mccanne. FWIW, he didn't think the count() approach was hacky. But in terms of other ways to address this, he thought the proposed "path" work (#4909) combined with over would solve this problem well.

[philrz]

I was reminded of this issue again when responding to another community Slack thread.

The use case was "zipping two arrays together". The input data.json:

{
  "data": {
    "id": "0",
    "type": "timeseries_response",
    "attributes": {
      "series": [
        {
          "group_tags": [
            "foo:bar"
          ],
          "query_index": 0,
          "unit": null
        },
        {
          "group_tags": [
            "foo:quux"
          ],
          "query_index": 0,
          "unit": null
        }
      ],
      "values": [
        [
          13,
          12,
          11,
          10
        ],
        [
          12,
          12,
          11,
          10
        ]
      ]
    }
  }
}

The program I ended up showing the user, with output:

$ zq -version
Version: v1.17.0-37-g4360baf9

$ zq -z '    
over data.attributes.series with v=data.attributes.values => (
  yield {c:count(), s:this}
  | yield {tags:s.group_tags,sum:(over v[c-1] | max(this))}
)' data.json

{tags:["foo:bar"],sum:13}
{tags:["foo:quux"],sum:12}

This got the job done, but it did have me thinking: I could start finding more places in the docs to say "here's how you use count() inside lateral subqueries if you need to access array elements by index", or we could find more direct ways to offer this to users that seek it. Since other programming languages offer direct approaches, users might keep expecting to find it.

[philrz]

Update: While a direct approach access such an array index does not yet exist, the changes in #6344 and other recent language enhancements change how the interim solution looks. At current tip of main d64b887:

$ super -version
Version: d64b8878c

$ echo '["a","b","c"]' | super -j -c "
unnest this into (
  count {value, key:this}
  | unflatten([this])
)
| collect(this)" -

[{"a":1},{"b":2},{"c":3}]