How can I paginate an API with http_client

[wirtsi]

Hey

I am trying to read through a REST API that returns 100 results and that gives a cursor ... this cursor allows me to read the next batch. So the field cursor.next of the response contains the reference that should be included in the query params.

input:
  read_until:
    check: this.cursor.next == ""
    restart_input: true
    input:
      http_client:
        url: "<api>/products?cursor=${! json(cursor.next) }"

Gives me a

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x44ed43b]

I guess this is because json doesn't like that there is no data yet.

Am I doing this right? Or what is the suggested approach in such a case?

bests & thanks a lot

[Jeffail]

Hey @wirtsi, that runtime error is definitely a bug and Benthos should never be doing that, does it print any extra info? And which version are you running?

Currently the interpolations in the http_client input aren't able to access fields from messages as there's no clear definition of which message they should access. However, your intuition that it should be the previously read message, or nothing if this is the first invoke, I think is reasonable as long as we clearly document it.

I'm going to mark this issue as an enhancement so that we can make it work that way, but I also need to fix that runtime error as it's clearly a bug somewhere.

As an alternative for now you could use the generate input with an http_client processor and a few cache processors in order to do pagination but it's a hefty bit of config. If I have time this evening I'll try and put together an example config.

[wirtsi]

Hi @Jeffail

Thanks for the super swift response. I'll try to fight through your suggestion with the generator ....

I am now using benthos 3.45.0 installed via brew, error is still the same (before it was 3.44.1)

Stacktrace is

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x44ef5fa]

goroutine 8 [running]:
github.com/Jeffail/benthos/v3/internal/bloblang/field.QueryResolver.ResolveBytes(0x6ab4fe0, 0xc000b4b7a0, 0x0, 0x0, 0x0, 0xc000540000, 0xc00054e000, 0x4e, 0x4e)
	/private/tmp/benthos-20210424-79604-n7madc/benthos-3.45.0/internal/bloblang/field/resolver.go:69 +0x3a
github.com/Jeffail/benthos/v3/internal/bloblang/field.(*expression).resolve(0xc000b4cf30, 0x0, 0x0, 0x0, 0x6080000, 0x1, 0xc0001180b0, 0xc000090648)
	/private/tmp/benthos-20210424-79604-n7madc/benthos-3.45.0/internal/bloblang/field/expression.go:94 +0xdd
github.com/Jeffail/benthos/v3/internal/bloblang/field.(*expression).Bytes(...)
	/private/tmp/benthos-20210424-79604-n7madc/benthos-3.45.0/internal/bloblang/field/expression.go:105
github.com/Jeffail/benthos/v3/internal/bloblang/field.(*expression).String(0xc000b4cf30, 0x0, 0x0, 0x0, 0x0, 0x0)
	/private/tmp/benthos-20210424-79604-n7madc/benthos-3.45.0/internal/bloblang/field/expression.go:143 +0x76
github.com/Jeffail/benthos/v3/lib/util/http/client.(*Type).CreateRequest(0xc0009a7880, 0x0, 0x0, 0x0, 0xc7620f8, 0x203000)
	/private/tmp/benthos-20210424-79604-n7madc/benthos-3.45.0/lib/util/http/client/type.go:343 +0xb8
github.com/Jeffail/benthos/v3/lib/util/http/client.(*Type).DoWithContext(0xc0009a7880, 0x6ac32b8, 0xc00013a008, 0x0, 0x0, 0x0, 0x0, 0x0)
	/private/tmp/benthos-20210424-79604-n7madc/benthos-3.45.0/lib/util/http/client/type.go:553 +0xfc
github.com/Jeffail/benthos/v3/lib/util/http/client.(*Type).Do(...)
	/private/tmp/benthos-20210424-79604-n7madc/benthos-3.45.0/lib/util/http/client/type.go:523
github.com/Jeffail/benthos/v3/lib/input.(*HTTPClient).readNotStreamed(0xc0002a8a00, 0x6ac3280, 0xc000542080, 0x60b1640, 0xc00053c0c0, 0x6aa0718, 0xc000542080, 0xc000542080)
	/private/tmp/benthos-20210424-79604-n7madc/benthos-3.45.0/lib/input/http_client.go:262 +0x66
github.com/Jeffail/benthos/v3/lib/input.(*HTTPClient).ReadWithContext(0xc0002a8a00, 0x6ac3280, 0xc000542080, 0xc000542080, 0xc00053c0c0, 0x8413a68, 0x10, 0xc00053c0a0)
	/private/tmp/benthos-20210424-79604-n7madc/benthos-3.45.0/lib/input/http_client.go:214 +0xab
github.com/Jeffail/benthos/v3/lib/input/reader.(*AsyncPreserver).ReadWithContext(0xc000b508c0, 0x6ac3280, 0xc000542080, 0x0, 0x0, 0x0, 0x0, 0x0)
	/private/tmp/benthos-20210424-79604-n7madc/benthos-3.45.0/lib/input/reader/async_preserver.go:171 +0x41b
github.com/Jeffail/benthos/v3/lib/input.(*AsyncReader).loop(0xc000233110)
	/private/tmp/benthos-20210424-79604-n7madc/benthos-3.45.0/lib/input/async_reader.go:132 +0x495
created by github.com/Jeffail/benthos/v3/lib/input.NewAsyncReader
	/private/tmp/benthos-20210424-79604-n7madc/benthos-3.45.0/lib/input/async_reader.go:63 +0x212

[Jeffail]

Thanks for the trace, got it fixed: 2b5b316

[Jeffail]

Here's a rough example of what I meant with the generate and processor method:

input:
  read_until:
    restart_input: true
    check: 'this.cursor.next == ""'
    input:
      generate:
        interval: ""
        mapping: 'root = ""'
      processors: 
        - branch:
            processors:
              - cache:
                  operator: get
                  resource: page_token
                  key: "last_token"
              - catch: []
            result_map: 'meta page_token = content().string()'
        
        - http:
            url: '<api>/products?cursor=${! meta("page_token") }'
            verb: GET
      
        - cache:
            operator: set
            resource: page_token
            key: "last_token"
            value: '${! json("cursor.next") }'

cache_resources:
  - label: page_token
    memory: {}

It's not great but it should get the job done for now.

[wirtsi]

Hah, just wanted to post my solution but you beat me to it 😄 Thanks!!!
You sure that the meta page_token line shouldn't be root.cursor.next = content().string()?

[Jeffail]

Whoops, good point. I think we need to store the token as metadata otherwise we'd end up sending a body in the http request, but I've updated my example to reference the metadata field in the request url rather than cursor.next.

[Jeffail]

Hey @wirtsi, I forgot to update this ticket but I've updated the http_client input so that interpolation functions reference the previously consumed message: a3e20db

When the docs site next gets updated there'll be a nice example on the component page.

[wirtsi]

Wicked, thank you!