Stop running handlers when a chunk matches the whole file.

[qkaiser]

Whenever a chunk matching the whole file is found, we should just return it and stop processing others.

[qkaiser]

For that optimization to work, we should have our Yara match results ordered by increasing order on offsets. The idea is to rely on both extractor strategies (most likely formats first) and the ordering of yara match offsets to return at the earliest possible time if a chunk matches the whole file (chunk.start_offset == 0, chunk.end_offset == file.EOF).

Let's say you have an ExtFS3 filesystem full of ZIP file, we will start creating a chunk for the whole extfs3 file, then numerous chunks for each ZIP file within that ExtFS3 container.

If we implement this "return early", we will not spend CPU cycles crafting ValidChunks of ZIP content for every Yara match to then get rid of them during the call to remove_inner_chunks.

It's an easy win for when we analyze legitimate and valid files. This obviously won't work when analyzing raw memory dumps or some garbage files stitched together.

Note: I'm talking of ZIP files here, but that could be anything else.

[kissgyorgy]

Having 1 chunk covering the whole file is a good optimization for the most common case, but we can go one step further by checking if the whole file is covered with multiple chunks.

[kukovecz]

Having 1 chunk covering the whole file is a good optimization for the most common case, but we can go one step further by checking if the whole file is covered with multiple chunks.

I will think about this, but this MIGHT be an overhead. At the end of the iteration, we remove the outer Chunks anyways.
Checking that 1 Chunk covers the whole file is really really simple (chunk.size == file_size), but checking if multiple Chunks are covering the whole file could be complicated.

[kissgyorgy]

MIGHT be an overhead.

If you know this: https://gist.github.com/jboner/2841832 it's probably not an overhead (memory access vs random read from SSD).

could be complicated.

It's very simple; you have to check that the chunks are aligned from start until the end of the file.

[tests00]

I think this issue should be re-focused - we should not call calculate_chunk() for a handler where the start offset starts within an already-calculated ValidChunk.