Match produces 3 MB of basis information

[nkrabben]

Running siegfried across a collection, it's really slowing down on a group of mp3's. https://www.nationalarchives.gov.uk/pronom/fmt/134

The basis field for these matches can be up to 3,000,000 characters long with repetitions of similar data such as [795 105] * 9,650 and [807 105] * 13,068. I'm not sure what's causing this. If useful, I can probably provide a copy of the files causing this bug in the new year.

[richardlehane]

thanks Nick - this is something that's cropped up before (#94). My previous fixes have been stop-gaps but I've been tinkering with a more fundamental fix & it is good to have this as a prompt to work on it.

Basically this issue occurs for "noisy" signatures with multiple segments that generate lots of partial matches. fmt/134 is probably the worst offender. I'm currently too exhaustive in following up these matches - I'd like to make this bit of the code "lazier", it has just been hard to do so without breaking everything.

[richardlehane]

Hi @nkrabben I'm working on this issue now - if you have a sample you can share (either here or via email to keep private) would be great help

[richardlehane]

this is partly fixed (the verbose basis bit) in v1.7.9. But I have more work to do on the speed side of this issue, so re-opening

[richardlehane]

second part of this bug (slowdown for MP3 with lots matches) is now fixed on develop branch and will be in next release, see this issue: #128