Skip to content

Skip handling of known noisy/invalid binary files #3706

New issue
New issue
Open
Open
Skip handling of known noisy/invalid binary files#3706
Labels
enhancement
@rgmz

Description

@rgmz
rgmz
opened on Dec 2, 2024

There are many known bad test files which produce a lot of noise. It would be nice if these were skipped, as they're known to be invalid/useless.

Examples

github.com/microsoft/go-winio/archive/tar/testdata/

    2024-11-16T21:40:54-05:00	info-2	trufflehog	skipping file: size exceeds max allowed	{"repo": "https://github.com/cosmos/ethermint_abci.git", "commit": "07dad5f", "path": "vendor/github.com/microsoft/go-winio/archive/tar/testdata/writer-big-long.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "16gig.txt", "size": 17179869184, "size": 17179869184, "limit": 2147483648}
    2024-11-16T21:40:54-05:00	info-2	trufflehog	skipping file: size exceeds max allowed	{"repo": "https://github.com/cosmos/ethermint_abci.git", "commit": "07dad5f", "path": "vendor/github.com/microsoft/go-winio/archive/tar/testdata/writer-big.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "16gig.txt", "size": 17179869184, "size": 17179869184, "limit": 2147483648}

xz test files?

    2024-11-15T12:15:57-05:00	error	trufflehog	error unarchiving chunk.	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c652", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-lzma2-2.xz", "timeout": 60, "mime": "application/x-xz", "timeout": 60, "error": "error creating reader for decompressor with format: .xz unable to detect MIME type: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error unarchiving chunk.	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c652", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-lzma2-3.xz", "timeout": 60, "mime": "application/x-xz", "timeout": 60, "error": "error creating reader for decompressor with format: .xz unable to detect MIME type: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error unarchiving chunk.	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c652", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-lzma2-4.xz", "timeout": 60, "mime": "application/x-xz", "timeout": 60, "error": "error creating reader for decompressor with format: .xz unable to detect MIME type: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error unarchiving chunk.	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c652", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-lzma2-5.xz", "timeout": 60, "mime": "application/x-xz", "timeout": 60, "error": "error creating reader for decompressor with format: .xz unable to detect MIME type: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c65297865d9f8da501761f46e2a34e29af603", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-lzma2-6.xz", "error": "failed to create file reader to handle file: error identifying archive: matching rar: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error unarchiving chunk.	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c652", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-lzma2-7.xz", "timeout": 60, "mime": "application/x-xz", "timeout": 60, "error": "error creating reader for decompressor with format: .xz unable to detect MIME type: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error unarchiving chunk.	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c652", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-lzma2-8.xz", "timeout": 60, "mime": "application/x-xz", "timeout": 60, "error": "error creating reader for decompressor with format: .xz unable to detect MIME type: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error unarchiving chunk.	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c652", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-stream_flags-1.xz", "timeout": 60, "mime": "application/x-xz", "timeout": 60, "error": "error creating reader for decompressor with format: .xz unable to detect MIME type: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c65297865d9f8da501761f46e2a34e29af603", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-stream_flags-2.xz", "error": "failed to create file reader to handle file: error identifying archive: matching rar: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error unarchiving chunk.	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c652", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-stream_flags-3.xz", "timeout": 60, "mime": "application/x-xz", "timeout": 60, "error": "error creating reader for decompressor with format: .xz unable to detect MIME type: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c65297865d9f8da501761f46e2a34e29af603", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-vli-1.xz", "error": "failed to create file reader to handle file: error identifying archive: matching zip: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c65297865d9f8da501761f46e2a34e29af603", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-vli-2.xz", "error": "failed to create file reader to handle file: error identifying archive: matching tar: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c65297865d9f8da501761f46e2a34e29af603", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-2-compressed_data_padding.xz", "error": "failed to create file reader to handle file: error identifying archive: matching rar: xz: data is corrupt"}

libbogus

2024-11-14T09:47:59-05:00	error	trufflehog	error processing AR files	{"repo": "https://github.com/apple/swift-llvm.git", "commit": "d17dee5", "path": "test/tools/llvm-readobj/Inputs/library.lib", "timeout": 60, "mime": "application/x-archive", "timeout": 60, "timeout": 60, "error": "error reading AR payload: strconv.Atoi: parsing \"\": invalid syntax"}
2024-11-14T09:48:02-05:00	error	trufflehog	error processing AR files	{"repo": "https://github.com/apple/swift-llvm.git", "commit": "8a83602", "path": "test/tools/llvm-objdump/Inputs/library.lib", "timeout": 60, "mime": "application/x-archive", "timeout": 60, "timeout": 60, "error": "error reading AR payload: strconv.Atoi: parsing \"\": invalid syntax"}
2024-11-14T09:48:02-05:00	error	trufflehog	error processing AR files	{"repo": "https://github.com/apple/swift-llvm.git", "commit": "cec5045", "path": "test/tools/llvm-readobj/Inputs/library.lib", "timeout": 60, "mime": "application/x-archive", "timeout": 60, "timeout": 60, "error": "error reading AR payload: strconv.Atoi: parsing \"\": invalid syntax"}
2024-11-14T09:48:03-05:00	error	trufflehog	error processing AR files	{"repo": "https://github.com/apple/swift-llvm.git", "commit": "e5d1578", "path": "test/tools/llvm-objdump/Inputs/libbogus11.a", "timeout": 60, "mime": "application/x-archive", "timeout": 60, "timeout": 60, "error": "error reading AR payload: strconv.Atoi: parsing \"~97&\": invalid syntax"}
2024-11-14T09:48:03-05:00	error	trufflehog	error processing AR files	{"repo": "https://github.com/apple/swift-llvm.git", "commit": "e5d1578", "path": "test/tools/llvm-objdump/Inputs/libbogus12.a", "timeout": 60, "mime": "application/x-archive", "timeout": 60, "timeout": 60, "error": "error reading AR payload: strconv.Atoi: parsing \"#55!\": invalid syntax"}
2024-11-14T09:48:03-05:00	error	trufflehog	error processing AR files	{"repo": "https://github.com/apple/swift-llvm.git", "commit": "e5d1578", "path": "test/tools/llvm-objdump/Inputs/libbogus14.a", "timeout": 60, "mime": "application/x-archive", "timeout": 60, "timeout": 60, "error": "error reading AR payload: strconv.Atoi: parsing \"1foobar273\": invalid syntax"}

github.com/pierrec/lz4/fuzz/corpus

2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/226780b32ba8f87ec614fdb376aa0884011c4ca9-17", "error": "failed to create file reader to handle file: error identifying archive: matching rar: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/234cc427d9be32470f3c2e11a6bc16567f558e55-22", "error": "failed to create file reader to handle file: error identifying archive: matching 7z: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/2486a84bf0f161f45b050d9c19ea9e35f5def864-8", "error": "failed to create file reader to handle file: error identifying archive: matching 7z: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/276580343a14eec04143e89a778dae3e14df472c-17", "error": "failed to create file reader to handle file: error identifying archive: matching rar: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/27fb5dc4016dc640e55a60719a222c38c604fa6b-2", "error": "failed to create file reader to handle file: error identifying archive: matching zip: bzip2: corrupted input: mismatching stream checksum"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/2a33d8514fb512aa20b0a56800cd3e12f3952b6b-26", "error": "failed to create file reader to handle file: error identifying archive: matching rar: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/2c2a5947341d76797a7e2299f39d01e3aebb2eb8-19", "error": "failed to create file reader to handle file: error identifying archive: matching rar: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/2cc2308b75a2e8f7eafcf69370767e5fce314892-13", "error": "failed to create file reader to handle file: error identifying archive: matching zip: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/2cdafdadb156e2759c389b6b8edf6a402034886c-26", "error": "failed to create file reader to handle file: error identifying archive: matching rar: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/2de93224b5f0db491ced1ec491a9f41d71820671-11", "error": "failed to create file reader to handle file: error identifying archive: matching rar: lz4: invalid block size"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/2e8487cf61feda70c0d74f12bfb5b692b684f82a-9", "error": "failed to create file reader to handle file: error identifying archive: matching rar: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/2f0ee9cf4bb951a37efc6460d5709442bc3de54e-6", "error": "failed to create file reader to handle file: error identifying archive: matching 7z: lz4: invalid frame checksum: got 2cc5d05; expected 206e6168"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/2fad20024167a500cdb8df5334a614f113efae00-20", "error": "failed to create file reader to handle file: error identifying archive: matching rar: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/300579a548d96d64c9da8470efa15e787f1a36f1-28", "error": "failed to create file reader to handle file: error identifying archive: matching rar: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/31c6c22708d346ed9e936fa7e77c8d9ab6da8d1e-33", "error": "failed to create file reader to handle file: error identifying archive: matching rar: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:27-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/344d38ec2ec90cb617e809439938b4cbf3b11f02-10", "error": "failed to create file reader to handle file: error identifying archive: matching rar: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:27-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/352631eab692c4a2c378b231fb3407ebcc0c3039-33", "error": "failed to create file reader to handle file: error identifying archive: matching 7z: lz4: invalid source or destination buffer too short"}

Git

2024-12-02T18:10:59-05:00	info-2	trufflehog	skipping file: size exceeds max allowed	{"repo": "https://github.com/Shopify/git.git", "commit": "e51217e", "path": "t/t5000/huge-and-future.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "huge", "size": 68719476737, "size": 68719476737, "limit": 2147483648}

Misc

2024-12-04T02:20:47Z    info-2  trufflehog      skipping file: size exceeds max allowed {"repo": "https://github.com/IBM/go-with-wakeup-profile.git", "commit": "9d3d370", "path": "src/archive/tar/testdata/writer-big.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "16gig.txt", "size": 17179869184, "size": 17179869184, "limit": 21474
83648}
2024-12-04T02:20:47Z    info-2  trufflehog      skipping file: size exceeds max allowed {"repo": "https://github.com/IBM/go-with-wakeup-profile.git", "commit": "e0ab505", "path": "src/archive/tar/testdata/gnu-sparse-big.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "gnu-sparse", "size": 60000000000, "size": 60000000000, "limit":
2147483648}
2024-12-04T02:20:47Z    info-2  trufflehog      skipping file: size exceeds max allowed {"repo": "https://github.com/IBM/go-with-wakeup-profile.git", "commit": "e0ab505", "path": "src/archive/tar/testdata/pax-sparse-big.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "pax-sparse", "size": 60000000000, "size": 60000000000, "limit":
2147483648}
2024-12-04T02:20:47Z    info-2  trufflehog      skipping file: size exceeds max allowed {"repo": "https://github.com/IBM/go-with-wakeup-profile.git", "commit": "e0ab505", "path": "src/archive/tar/testdata/writer-big-long.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "16gig.txt", "size": 17179869184, "size": 17179869184, "limit":
2147483648}
2024-12-04T02:20:47Z    info-2  trufflehog      skipping file: size exceeds max allowed {"repo": "https://github.com/IBM/go-with-wakeup-profile.git", "commit": "e0ab505", "path": "src/archive/tar/testdata/writer-big.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "16gig.txt", "size": 17179869184, "size": 17179869184, "limit": 21474
83648}
2024-12-04T02:20:47Z    info-2  trufflehog      skipping file: size exceeds max allowed {"repo": "https://github.com/IBM/go-with-wakeup-profile.git", "commit": "7ae9561", "path": "src/archive/tar/testdata/writer-big-long.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "16gig.txt", "size": 17179869184, "size": 17179869184, "limit":
2147483648}
2024-12-04T02:20:47Z    info-2  trufflehog      skipping file: size exceeds max allowed {"repo": "https://github.com/IBM/go-with-wakeup-profile.git", "commit": "7ae9561", "path": "src/archive/tar/testdata/writer-big.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "16gig.txt", "size": 17179869184, "size": 17179869184, "limit": 21474
83648}
2024-12-04T02:20:47Z    info-2  trufflehog      skipping file: size exceeds max allowed {"repo": "https://github.com/IBM/go-with-wakeup-profile.git", "commit": "c991d2a", "path": "src/archive/tar/testdata/writer-big-long.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "16gig.txt", "size": 17179869184, "size": 17179869184, "limit":
2147483648}
2024-12-04T02:20:51Z    info-2  trufflehog      skipping file: size exceeds max allowed {"repo": "https://github.com/IBM/go-with-wakeup-profile.git", "commit": "2d3cd51", "path": "src/archive/tar/testdata/writer-big-long.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "16gig.txt", "size": 17179869184, "size": 17179869184, "limit":
2147483648}
2024-12-04T02:21:02Z    error   trufflehog      error handling binary file      {"repo": "https://github.com/IBM/go-with-wakeup-profile.git", "commit": "139fad21b937dc0b405d243e3abe33429890947f", "path": "src/compress/gzip/testdata/issue6550.gz", "error": "failed to create file reader to handle file: error identifying archive: matching tar: gzip: invalid heade
r"}

Related

This relates to #1517 in spirit.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancement

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions