RuntimeError: Line exceeds maximum page size 998

[sentry]

Sentry Issue: BOT-3Q

RuntimeError: Line exceeds maximum page size 998
  File "bot/pagination.py", line 158, in paginate
    paginator.add_line(line, empty=empty)
  File "bot/pagination.py", line 72, in add_line
    raise RuntimeError('Line exceeds maximum page size %s' % (self.max_size - len(self.prefix) - 2))

Failed to add line to paginator: '===============
            Status: Inactive
            User: XAR#2912 (`627187206083051541`)
            Type: **note**
            Shadow: True
            Reason: discussing hacking in a voice channel and showing off a "hacking tool" they've made [here](https://discordapp.com/channels/267624335836053506/412357430186344448/706680820060127243). there also appears to be some sort of malicious script they've created [here](https://discordapp.com/channels/2676243358360535...

[aeros]

Relevant screenshot of how the above error occurred (courtesy of Lemon):

[aeros]

I believe the preferred way of addressing this error would be via an improvement to the infraction message pagination. If the contents of a single message would exceed the character limit for a single page (998), it could be changed to overflow onto the second page. The boundaries of the infraction would be indicated by the existing separator ===============, and potentially with a new (Continued)/(Cont.) line at the top of the overflow page to show that it's a continuation of the last message.

Otherwise, we could more simply impose a hard limit on the length of the infraction message that rejects the infraction if it is exceeded or just truncate the message from the point it reaches the limit, but I think there are some more involved moderation situations where the extra length is necessary to provide adequate context.

[MarkKoz]

A general improvement to the paginator would be to move a line to a new page if it would cause the character limit for the page to be exceeded.

I feel splitting an infraction across two pages would be somewhat confusing to follow. How about we raise the character limit to the maximum (around 2000) for infraction search embeds? If it's still not enough, the infraction could be moved to its own page without any other infractions. If that's still not enough, then continue it on the following page.

[aeros]

I feel splitting an infraction across two pages would be somewhat confusing to follow. How about we raise the character limit to the maximum (around 2000) for infraction search embeds? If it's still not enough, the infraction could be moved to its own page without any other infractions. If that's still not enough, then continue it on the following page.

That sounds reasonable to me. I agree that continuation onto the next page should probably be used as a last resort, so if we can increase the character limit and then try to move it onto its own page if needed, I think it would be more readable for the majority of situations. Anything exceptional can make use of the continuation, but I suspect that infraction messages over 2k characters would be quite rare.

[aeros]

A general improvement to the paginator would be to move a line to a new page if it would cause the character limit for the page to be exceeded.

Based on my local experimentation so far and looking over the code, it looks like the paginator already does this in LinePaginator.add_line(). When attempting to add an additional line to the current page and it would exceed the the limit, it is closed, and the line is instead appended to a new page. See

bot/bot/pagination.py

Lines 80 to 81 in 007f771

	if self._count + len(line) + 1 > self.max_size:
	self.close_page()

It took me a bit to realize how this worked, as I was looking for some form of explicit open_page(). Instead, it seems that the discord.py base paginator implicitly creates a new page when the previous one is terminated.

So, instead, I think it makes sense to focus the PR on adding continuation to the paginator. I still have a bit of work remaining, but here's what I have so far:

My current implementation separates by word boundaries instead of by character to make the continuation easier to read, and avoids breaking links or other markdown. I also added a dashed line --- to more clearly separate the (Continuation) header from the actual content, which makes it significantly easier to read multi-page messages.

Also, it's worth mentioning that the continuation only happens when a single message exceeds the character limit for the page.

The main thing I have left to do is adding some unit tests for it, particularly with edge cases such as many pages. I'm not sure what the upper bound limit as far as max continuation pages would realistically be, but I can't imagine a single message going over 5 pages. It's rare enough for a message to exceed a single page (otherwise this issue would be more of an immediate concern).

[aeros]

I've been thinking further about a reasonable character limit for the infraction messages. Although upping it to 2k would help to ensure it fits most messages, there is something to be said about having a condensed infraction paginator. Frequently in the #mods channel, we search for a user currently being discussed and then talk about it. I've found that the existing limit of 1k can already get in the way sometimes by taking up too much space, so a 2k one would be even worse (especially for smaller resolutions).

So, I'd like to get some feedback from the other moderators+ about what they would prefer in this situation, since we're the ones that have to use it. I'll mention them in #mods and link to this issue to see what they think about the page size limit and the current look of the continuation.