unexpected message IDs not handled

[bwarden]

In one meshchat zone on our mesh, messages have stopped showing up in the web client. Suspecting that it's another formatting issue, I obtained a copy of the sync'ed message file and found that certain non-compliant message IDs (injected by someone's experimental PHP implementation of meshchat) seem to cause the problem.

Here are the first two columns from /tmp/meshchat/messages.MeshChat (redacting messages and callsigns):

2fc70fa4        1679602416
61c9dc69        1679022908
43307271558467  1678040279
4057520357      1678040279
7a2ab7f8        1678577270
cc380e1d        1680324078
1d05be88        1681074063
f138afc6        1679520054
1203a67f        1677875628
3ad9d202        1678898380
ab200237        1679624143
184435c9        1681711764
2.4819733522097E+19     1678068339
2.8833779246934E+19     1677875624
1918315737      1678244465
808dbb75        1678413538
637173503       1677906056
1e74bae6        1681696857
464508753864118889      1677906056
71d264d9        1681710141
15309772582708309       1678222482
147677390439    1677965606
4988039448157784369     1678047183
1.0831443016181E+19     1678349464
d79a9f54        1678068339
15871682344753218       1678068339
2377976020820649495     1678348426
2095906580566786        1678246661
8b369632        1681745456
2364224544      1678353386
6dc6c0e6        1677887037
1089694653      1678348426
d0e4307a        1681274425
19249734238112025       1677977195
1006076813586995        1678304885
2804351308      1678285806
e87300f9        1677874677
99160908677     1677978207
7359541018712184102     1677887037

Guessing that the problematic IDs are those that aren't 32-bit hex, I changed the ID parsing in sort_and_trim_db() to be a little bit more strict:

        local id, epoch = line:match("^(%S{8})\t(%S+)\t")

Upon restarting meshchatsync, the rest of the messages loaded sucessfully in the web UI.

I'm not sure what's the right fix, but at least I think I've found the (or a) problem.

[bwarden]

I think this is a little bit better. I suspect it's the extended floating point IDs with exponentials that are really gumming things up.

        local id, epoch = line:match("^(%[0-9a-fA-F]+)\t(%S+)\t")

Although this will provide conflicting single-digit IDs for those offending messages.

I'm not fluent in lua's regex implementation; I'd do this in Perl to just capture the mantissa:

m/^(?:\d\.)?([0-9a-fA-F]+)(?:E\+\d+)?\t(\d+)\t"/

[bwarden]

OK, this looks more better. It doesn't tolerate those IDs, but at least trims them out of the database so things can proceed;

    for line in io.lines(messages_db_file)
    do
        local id, epoch = line:match("^(%x+)\t(%S+)\t")
        -- ignore messages that are too far in the future (assume they're errors)
        epoch = tonumber(epoch)
        if epoch and epoch < valid_time then
            messages[#messages + 1] = {
                epoch = epoch,
                id = tonumber(id, 16),
                line = line
            }
        end
        unused_count = unused_count - 1
    end