Implement got pagination support

[rarkins]

What would you like Renovate to be able to do?

Support GitHub's pagination in a got wrapper.

Describe the solution you'd like

I would like to add a lib/util/got/github-pagination.js instance that expands on the default instance and returns a paginated res.body if opts.paginate is set to true in the request.

Describe alternatives you've considered

Writing pagination logic manually at the application layer every time.

Additional context

I started writing lib/util/got/* wrappers already.

[rarkins]

@szmarczak do you have any suggestions for how to do this elegantly? I think I could probably do this with got.create() and handler BUT the default got instance I wrote that uses caching already has a handler so I don't think it's possible to "merge" two instances with handlers, right?

In that case I think I need to create a new got instance that then "calls" my existing one instead of merges with it.

[rarkins]

Ref: sindresorhus/gh-got#22

[szmarczak]

I don't think it's possible to "merge" two instances with handlers, right?

Of course you can! :)

You're right, handlers should do the job. But pagination should be supported in Got natively.

[rarkins]

@szmarczak have you ever thought about the semantics for a pagination "API"?

My approach:

• If the "per page" is specifiable then the app should do it (e.g. set ?per_page=100)
• App must opt in to pagination (e.g. opts.paginate = true) rather than automatic
• Paginate all pages by default? Use a second variable for paginationLimit or overload opts.paginate to let it be a number of pages?

[szmarczak]

have you ever thought about the semantics for a pagination "API"?

Not yet. Let's continue the talk at sindresorhus/got#722

I'll sketch something out.