Ignore unsigned files (only read and hash files contained in signature manifest)

[stefanberger]

Summary

This PR adds support for an option --ignore_unsigned_files to ignore files that are not part of the signature manifest. This allows to for example ignore files that were added after a signature was created. Existing test cases are extended to cover additional symlinks and regular files added after signature created and tests for expected passes and failures depending on whether --ignore_unsigned_files is used.

Checklist

• All commits are signed-off, using DCO
• All new code has docstrings and type annotations
• All new code is covered by tests. Aim for at least 90% coverage. CI is configured to highlight lines not covered by tests.
• Public facing changes are paired with documentation changes
• Release note has been added to CHANGELOG.md if needed

[mihaimaruseac]

My main worry is that we're pushing the concerns all the way to the serialization, whereas this ignore only belongs in verification. Let's still compute the manifest with all the files that exist on disk and only ignore entries during verification.

[stefanberger]

My main worry is that we're pushing the concerns all the way to the serialization, whereas this ignore only belongs in verification. Let's still compute the manifest with all the files that exist on disk and only ignore entries during verification.

I thought that's what I was doing by only passing in the files_to_hash list from the verification side and leaving it None otherwise.

[stefanberger]

... I mean it's most efficient to only hash those files that are being asked for rather than hashing all of them first and filtering after. Is this what you would want me to do, filter-out unwanted files just before comparing the manifests?

[stefanberger]

Let me open a 2nd PR for this. I think disk access is and hashing is too expensive to filter for later...

[mihaimaruseac]

We should probably benchmark this, based on real scenarios. We already have an ignore_paths in the serialization.

The most reticence comes from having to add a new flag, which we might not need in the future. We'll need to think more here.

[stefanberger]

We should probably benchmark this, based on real scenarios. We already have an ignore_paths in the serialization.

In some cases it may be desirable to let the signature drive which files are being hashed rather than the filesystem contents. To use ignore_paths, one first needs to find out which files are part of the signature and which files are in the filesystem and then one can determine which files need to be ignored. With the new option you have this automatically.

[mihaimaruseac]

Approving instead of #502.

[mihaimaruseac]

Will need to resolve conflicts, sorry for the delay that caused them

[stefanberger]

Thanks for merging.

FYI: vllm-project/vllm#21957

[mihaimaruseac]

Oh, this is awesome!!