Messages are being count twice (duplicates)

[JustAnOkapi]

if you upload two of the same discord reports, it counts each message twice.
this is important as if you have exports from overlapping time periods, messages will not be handled correctly

[mlomb]

It should not happen, but I'll take a look

[hopperelec]

I've experienced this too, so I can confirm it happens

[mlomb]

Yes, I can confirm only recent message IDs are stored (for linking replies) right now, I was mistaken

I don't know if deduplication is feasible since we would have to store the ID for every single message and do a search for it every time a new comes through (lets say logarithmic). Since we want to support multi million messages chats, size and performance may suffer too much

[hopperelec]

I don't understand how it works, but I've been told that queries to lists/maps involving hash tables are not affected by the size of the list/map (i.e. they have a time complexity of O(1)). It doesn't look like Javascript uses hash tables by default, but implementations of it look pretty simple. I imagine being able to search through all the past messages is useful for other features, too, such as for linking replies.

[hopperelec]

Just noticed something else this would solve

chat-analytics/report/components/FilterSelect.tsx

Lines 119 to 121 in 8caf16b

	// TODO: if this is slow, make sure selected is always sorted and run binary search here
	// for now, its not necessary
	selected={selected.includes(optionIndex)}

And I think there's more places where arrays are sorted to make them easier to find items in which could be replaced with a hash table

[hopperelec]

Just realised that all non-primitive Javascript (and hence Typescript) objects use hash tables. Improving the performance should be as easy as using the most appropriate data type for each form of array and making sure to use keys which can easily be identified. From what I can see, though, this is already being done, so I'm not sure how you've found that sorting the arrays helps with performance at all