WIP: Substring locations

[rakuy0]

@atlas0fd00m general thoughts? The unicode portion should be much the same, but I figured it's better to get feedback on the idea now.

[atlas0fd00m]

failing checks?

[atlas0fd00m]

some additional docstrs and a few questions? overall, the direction seems pretty good.

[atlas0fd00m]

this covers the "find the big string after the little one was already discovered" case.
does this catch the "find the little string in the middle of an existing big string" case?
if so, how?

this is getting complex enough that we probably want some explanation of what's going on (specifically wrt string within a string) use-cases and how they're handled.

[williballenthin]

oh, nice point on this only capture suffixes

[williballenthin]

counter point: rather than building the string index into the location database, what about pulling it out into its own object.

class StringIndex:
  def find(self, needle) -> Tuple[str, Loc]

i suspect this would be simpler to maintain (esp when a string is removed, does this ever happen?) conceivably more performant by using dedicated algorithms/data structures.

[rakuy0]

I like the idea of a dedicated string database, my only hold back on that is actually persisting that information when saving that workspace. I don't want to lose the information, and making custom encoders/decoders breaks any kind of possible backwards compatibility.

But it is intentional that this only captures suffixes subsets, since when we call makeString we only get the VA and then we make a string based on what we calculate as the length. That's 100% an artifact of the binaries I've been running into recently, where substrings are just suffixes to longer strings

[williballenthin]

yeah, there's some complexity in persisting the database.

what if we rebuilt the database upon each load? if we keep string start locations in the .viv, then its an easy operation to iterate, extract the strings, and index in memory. upon close, throw out the index.

of course it takes some CPU and memory to keep an index in memory, but i'd argue that until you can prove otherwise, its likely in line with the overhead of storing all the substrings in the .viv.

[williballenthin]

also, could build the index lazily on first access which would help the common case where substrings aren't accessed.

[rakuy0]

I wouldn't be against rebuilding the database on every load. Wouldn't be the worst thing in the world.

If we do go that route, I can see two possible ways of doing things. First is pushing some logic into the add location handler in vivisect/base.py so that on a makeString/makeUnicode call it does something similar-ish to what I've got here, in that it handles what the base string is, indexes, and so forth. Mostly so when we load an existing workspace, we end up with the same info as when we directly analyzing a binary, since workspace loading only ever sees the event stream. Second is similar to the first, except we plumb a new event type that handles all that for us.

I like the idea of a dedicated string database, as I think it would give us (easier) finer grained control over deletion, since right now it's "delete the main string and the substrings go with it". It's a more matter of do it now or do it later.

[atlas0fd00m]

ditto