Iterate over result in bash

[k-bx]

Hi! So, I'm using Riak indexes, which looks something like this:

curl -Ss http://myriak.server:8098/buckets/mybucket/index/index_int/100/200/
{"keys": ["foo", "bar", "baz"]}

Obvious task is to iterate over this "keys" list later in order to actually query for values and filter them with jq.

I just did this https://gist.github.com/k-bx/019fa8429b3634acd0c0 , but it looks ugly (stripping out N symbols, splitting by comma, removing "'s from keys.

Would be really cool if jq would be able to do that for me.

Thanks!

[glaudiston]

I'm not sure I understood what you want to do.
Can you give us a full output of this curl line ?

[k-bx]

I want to iterate in bash over a list inside JSON.

Full output of first curl command:

{"keys": ["foo", "bar", "baz"]}

I want in bash to do equivalent of:

for key in "foo" "bar" "baz" do
    curl http://myriak/buckets/mybucket/keys/$key | jq $'.[] | {.id};
done

[wtlangford]

You could do something like

for key in $(curl -Ss http://myriak.server:8098/buckets/mybucket/index/index_int/100/200/ | jq -r '.keys[] | @uri'); do
  curl http://myriak/buckets/mybucket/keys/$key | jq '.[] | {.id}';
done

At least until some kind of system builtin exists.
Note the @uri filter, which has jq print the keys in a url-encoded way. This way, you don't end up with spaces that break the bash for loop. It also means that the keys are properly printed for putting in a curl command.

[k-bx]

@wtlangford thanks! Problem with this approach -- it includes double-quotes from both sides of a key, so you end up doing something like this inside a loop:

curl -Ss http://myriak.com:8098/buckets/mybucket/keys/"foo"

Do you have an advice (apart from dirty hacks which I currently do :) ) how to solve this nicely?

Thanks!

[k-bx]

@wtlangford I mean, current solution from me is to use:

${key:1:-1}

inside loop. This strips off the doublequotes. So, maybe that's ok for me.

[pkoppstein]

@k-bx - The "-r" option that @wtlangford used strips off the unwanted outer double-quotation marks, so I think the following may be closer to what you are looking for if the keys have non-alphanumeric characters:

URL="http://myriak.server:8098/buckets/mybucket/index/index_int/100/200/"
curl -Ss  "$URL" | jq -r '.keys[] | @uri' |\
  while read key
  do
    curl "http://myriak/buckets/mybucket/keys/$key" | jq '.[] | {.id}'
  done

[k-bx]

@pkoppstein @wtlangford ah, I totally missed the -r flag. Will try to understand this "while read" trick, thanks.

[wtlangford]

@pkoppstein I definitely like the use of while read here over my subshell replacement.

Also, I just took the chance to look at .[] | {.id}. This doesn't work, even on master...
I'm not sure what the intended result is. If all that was wanted was the ids, then jq '.[].id' is a good choice. If objects containing the ids are the desired result ({"id":"id1"} ...), then jq '{"id": .[].id}' is what you want.

[k-bx]

@wtlangford I think it should be .[] | {id} which is the same as .[] | {id: .id}

Intention was to only select id field of objects.

p.s.: what would be even cooler -- if you'd tell me a good way to also loop through these id fields of result-objects

[pkoppstein]

The following variation only invokes jq twice:

URL="http://myriak.server:8098/buckets/mybucket/index/index_int/100/200/"
curl -Ss  "$URL" | jq -r '.keys[] | @uri' |\
while read key
do
  curl "http://myriak/buckets/mybucket/keys/$key" 
done | jq '.[] | {id}'

[k-bx]

@pkoppstein just to confirm it'll work -- second curl returns json list also (not object)

[nicowilliams]

@k-bx jq will read as many JSON tests on stdin as you feed it, not just one. So the while read key; do curl ...; done | jq ... method should work fine. Try it!

[pkoppstein]

I've included a similar (but tested) script at the jq Cookbook -- see emit-the-ids-of-json-objects-in-a-riak-database.

[nicowilliams]

Thanks @pkoppstein . I've added a note to the FAQ inviting users to edit the wiki.

[k-bx]

Fantastic! Thank you, all my problems regarding riak-querying are now solved. Just played with it a bit -- everything works great.

This (and a fact that you can use riak's streaming API) opens so many opportunities, like now it's easy to do stuff like "count-by-unique json field value" and others.

Thanks again!

[nicowilliams]

@k-bx It's really cool that Riak uses sequences of JSON text for streaming. Thanks for pointing that out! And it's wonderful that jq just works with that.

[thiagodolabella]

I'm trying to use every Key,Value of an output and pipe it to another command.
Here is what I'm trying to use:

INSTANCE_ID=$(curl http://169.254.169.254/latest/meta-data/instance-id)

aws ec2 describe-tags --filters "Name=resource-id,Values=$INSTANCE_ID"

With the above command, I have the following output:

{
"Tags": [
    {
        "ResourceType": "instance",
        "ResourceId": "i-0342a609edf80001a",
        "Value": "A-VALUE",
        "Key": "A-KEY"
    },
    {
        "ResourceType": "instance",
        "ResourceId": "i-0342a609edf80001a",
        "Value": "B-VALUE",
        "Key": "B-KEY"
    },
    {
        "ResourceType": "instance",
        "ResourceId": "i-0342a609edf80001a",
        "Value": "C-VALUE",
        "Key": "C-KEY"
    },
    {
        "ResourceType": "instance",
        "ResourceId": "i-0342a609edf80001a",
        "Value": "D-VALUE",
        "Key": "D-KEY"
    },
    {
        "ResourceType": "instance",
        "ResourceId": "i-0342a609edf80001a",
        "Value": "E-VALUE",
        "Key": "E-KEY"
    },
    {
        "ResourceType": "instance",
        "ResourceId": "i-0342a609edf80001a",
        "Value": "F-VALUE",
        "Key": "G-KEY"
    },
    {

Now I want to pipe each Key,Value to the following command:

aws ec2 create-tags --resources XXXXX --tags Key=H-KEY,Value=H-VALUE

Where the quantity and values of Key,Value are variable. So I believe I need a "for each".

May you help me?

It's like: For each Key,Value, do:

aws ec2 create-tags --resources XXXXX --tags Key=A-KEY,Value=A-VALUE

aws ec2 create-tags --resources XXXXX --tags Key=B-KEY,Value=B-VALUE

aws ec2 create-tags --resources XXXXX --tags Key=C-KEY,Value=C-VALUE

aws ec2 create-tags --resources XXXXX --tags Key=N...-KEY,Value=N...-VALUE

[ghost]

@thiagodolabella In the future, please send usage questions to StackOverflow's JQ tag, and avoid reusing existing issues for unrelated questions.

One way to do this would be to have jq generate a script you can then pipe back to the shell, something like:

jq -r '.Tags[] | "aws ec2 create-tags --resources XXXXX --tags Key=\(.Key | @sh),Value=\(.Value | @sh)"' | sh