Negative `editedAfter` value causing serialization to break

[Laiteux]

Exported 500K Telegram messages, Chat Anal worked fine. Tried with 1M from an other group so twice as much and now I'm getting some error:

msedge_e92VWudO2V.mp4

Messages -> Sentiment -> Timeline

[Laiteux]

Now that I'm thinking about it (not sure if it's related, haven't checked the code, just a random thought), couldn't it be because there was no message sent there since march 2022?

[mlomb]

Now that I'm thinking about it (not sure if it's related, haven't checked the code, just a random thought), couldn't it be because there was no message sent there since march 2022?

Hmm that would be strange, it should not cause any problem

The obfuscator is not ready yet (hopefully I will get to it soon), so I don't know how to go around this one.
In the screenshot you posted, if you can place a breakpoint and print the variable e, t or r you will find the Database object, can you share the time part of it?

[Laiteux]

Sorry for the delay. Sure, here you go:

[mlomb]

It appears that everything is as it should.
I made a PR (#87) to print when indexing dayIndex above 850:

chat-analytics/pipeline/aggregate/blocks/messages/MessagesPerPeriod.ts

Lines 48 to 50 in 544d6ac

	if (typeof msg.dayIndex !== "number" || msg.dayIndex > 850) {
	console.log("msg.dayIndex:", msg.dayIndex, "perDay.length:", res.perDay.length);
	}

Can you share a screenshot of the console when exporting the chat from the PR? https://pr-87.chat-analytics.pages.dev

[Laiteux]

Sure, thank you

[mlomb]

Interesting, the values from 851 to 860 are not printed and only one with 1023 is. Since the number of dayBits is 10, 1023 is the maximum, so all bits set.

I managed to track down the issue, it appears that the following indexOf is returning -1:

chat-analytics/pipeline/process/DatabaseBuilder.ts

Line 370 in 65b524d

msg.dayIndex = dateKeys.indexOf(Day.fromBinary(msg.dayIndex).dateKey);

Details

I added if (Math.random() < 0.1) msg.dayIndex = -1; so it drops some messages and the same sections you show crash.

I added some logging for this too in the same PR, can you show me the console but now in the app (where you generate)? https://pr-87.chat-analytics.pages.dev

chat-analytics/pipeline/process/DatabaseBuilder.ts

Lines 373 to 382 in a0df720

	if (msg.dayIndex === -1) {
	console.log(
	"dateKeys.length:",
	dateKeys.length,
	"msg.dayIndex:",
	msg.dayIndex,
	"Day.fromBinary(msg.dayIndex).dateKey:",
	Day.fromBinary(msg.dayIndex).dateKey
	);
	}

[Laiteux]

Do I have to wait for it to be updated online?

[Laiteux]

Still doesn't work so I assume not

[mlomb]

Sorry for the delay I had to go!

It updates automatically on each commit, so its live 😄
Did you already tried and no log appeared?

[Laiteux]

Yep. Could you please try yourself? Or it is possible I might be doing this wrong as well.

[mlomb]

I checked the deploy and the log is there 🤔. Maybe dayIndex is never -1.

What do you mean try yourself, is the server public?

I'll keep investigating