maxId in DefaultPaginationParams should be able to be set externally

[SuperSonicHub1]

masto.js/src/repositories/repository.ts

Lines 1 to 10 in ebdaee8

	export interface DefaultPaginationParams {
	/** **Internal parameter.** Use HTTP Link header from response for pagination. */
	readonly maxId?: string | null;
	/** **Internal parameter.** Use HTTP Link header from response for pagination. */
	readonly sinceId?: string | null;
	/** Get a list of items with ID greater than this value excluding this ID */
	readonly minId?: string | null;
	/** Maximum number of results to return per page. Defaults to 40. NOTE: Pagination is done with the Link header from the response. */
	readonly limit?: number | null;
	}

There is no way to reverse the pagination direction of /api/v1/accounts/:id/statuses, making the ability to set the max_id parameter necessary to create a resumable application that downloads all of someone's statuses.

If an end user does manually set maxId, their AsyncIterable will get caught in an endless loop due to Masto setting the max_id parameter twice (in fact, everything's repeated): https://pony.social/api/v1/accounts/106500315995032015/statuses?exclude_replies=false&exclude_replies=false&limit=100&limit=100&max_id=109323073972972141&max_id=109332395072165340&pinned=false&pinned=false. Seeing as max_id isn't an internal detail of the Mastodon API (it's actually older than min_id), users should be able to set it without running into issues like I did.

Also, maxId doens't properly handle a null input; instead of not including the max_id parameter, it sends max_id=null, which Mastodon doesn't understand.

[neet]

@SuperSonicHub1 Pagination is a high-level abstraction for link headers in Mastodon API, and I agree that it's useless in some cases where users want to set min_id, max_id, or limit manually.

Therefore, I'm also offering a method named fetchStatuses which you can GET /api/v1/accounts/:id/statuses without using AsyncIterable. By using this you also can iterate timelines backwards, and keep a max_id in the query parameter. Doesn't this solve your problem?

Also, maxId doens't properly handle a null input; instead of not including the max_id parameter, it sends max_id=null, which Mastodon doesn't understand.

This seems like a bug. I will fix it in another PR

[SuperSonicHub1]

Therefore, I'm also offering a method named fetchStatuses which you can GET /api/v1/accounts/:id/statuses without using AsyncIterable. By using this you also can iterate timelines backwards, and keep a max_id in the query parameter. Doesn't this solve your problem?

I ideally would like to not have to reimplement getStatusesIterable in terms of fetchStatuses; I actually like the abstraction of AsyncIterable quite a bit.

I'll propose a solution. Here's what the code currently looks like:
https://github.com/neet/masto.js/blob/main/src/paginator.ts#L3-L34

The main issue here is that after our first request, we're passing both our original parameters and the new ones given to us by Mastodon in the Link header.

Here's how I would solve this.

  async next(params?: Params): Promise<IteratorResult<Result>> {
    if (this.nextUrl == undefined) {
      return { done: true, value: undefined };
    }


    const response: Response<Result> = await this.http.request({
      method: 'get',
      // if no params specified, use link header
      url: params ? this.initialUrl : this.nextUrl,
      params: params ?? this.nextParams,
    });


    this.nextUrl = this.pluckNext(response.headers?.link as string);
    this.nextParams = {};


    return {
      done: false,
      value: response.data,
    };
  }

Now, after our initial sending of parameters, nextParams is an empty object, meaning our URL params from the Link header will no longer be duplicated nor overwritten.

If you want, I can make a PR.

[neet]

@SuperSonicHub1 Sorry for my bad understanding! I haven't realised that using both user-provided params and the Link header causes a duplication of query parameters.

The snippet you've shown looks neat. I'm willing to review it if you send me a PR 👍

[github-actions]

🎉 This issue has been resolved in version 4.6.9 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀