Siegfried 1.10 incompatible with brunnhilde?

[kieranjol]

Hi,

I updated to Siegfried 1.10 and I get the following error (this is my home mac hence I haven't clamav set up correctly)
Looks like the output of sf might have changed?

 % brunnhilde.py /Users/kieranjol/Desktop/wdpd /Users/kieranjol/Desktop/test_files/test 
2023-03-28 16:05:12,346 - INFO - Brunnhilde started. Source: /Users/kieranjol/Desktop/wdpd.
2023-03-28 16:05:12,347 - INFO - Running virus scan. This might take a while...
LibClamAV Error: cli_loaddbdir: No supported database files found in /usr/local/var/lib/clamav
ERROR: Can't open file or directory

----------- SCAN SUMMARY -----------
Known viruses: 0
Engine version: 1.0.1
Scanned directories: 0
Scanned files: 0
Infected files: 0
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 0.361 sec (0 m 0 s)
Start Date: 2023:03:28 16:05:13
End Date:   2023:03:28 16:05:13
2023-03-28 16:05:13,420 - INFO - No viruses found.
2023-03-28 16:05:13,420 - INFO - Running Siegfried. This might take a while...
2023-03-28 16:05:13,825 - INFO - Siegfried scan complete. Processing results.
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/bin/brunnhilde.py", line 1385, in <module>
    main()
  File "/Library/Frameworks/Python.framework/Versions/3.10/bin/brunnhilde.py", line 1359, in main
    process_content(
  File "/Library/Frameworks/Python.framework/Versions/3.10/bin/brunnhilde.py", line 947, in process_content
    use_hash = import_csv(cursor, conn, use_hash)
  File "/Library/Frameworks/Python.framework/Versions/3.10/bin/brunnhilde.py", line 312, in import_csv
    cursor.execute(insertsql, row)
sqlite3.OperationalError: table siegfried has 11 columns but 13 values were supplied

[kieranjol]

Looks like it's the new 'class' field, which one can avoid by using a build flag with roy apparently..

[tw4l]

Thanks for the heads up @kieranjol ! Might be time to refactor how Brunnhilde reads the sf input a bit to use keys instead of column indices... I will take a look shortly

[richardlehane]

sorry for the breakage! As @kieranjol notes, a short-term fix is to do roy build -noclass

[tw4l]

No worries @richardlehane and thanks for the short-term fix! :)

[ross-spencer]

@tw4l you might consider sqlitefid https://github.com/exponential-decay/sqlitefid/releases/tag/v4.0.0rc1 but it adopts a more complicated schema than Brunnhilde and would also impact how you've written your queries. I have a related issue to try and find a better abstraction for ID report queries: exponential-decay/demystify#54 but it's not a huge priority to address just yet, but may be cool to combine efforts. I'd also appreciate more eyes on sqlitefid, but I appreciate it's a decent chunk of work.

[tw4l]

@ross-spencer To get a fix in quickly for now I'm just switching to using a csv.DictReader so that Brunnhilde will read the rows by their column header/key rather than index.

That said, sqlitefid is really cool and in the eventual Brunnhilde 2.0 Python 3-only rewrite I think it'd make a lot of sense to use that. It'd be nice to be able to support sf YAML as well as CSV in Brunnhilde. I see that Siegfried CSV reading currently isn't supported but I could likely open a PR for that if you're interested!

[tw4l]

Fix is released in Brunnhilde 1.9.6! https://github.com/tw4l/brunnhilde/releases/tag/v1.9.6

[ross-spencer]

I see that Siegfried CSV reading currently isn't supported but I could likely open a PR for that if you're interested!

Absolutely!! The SF CSV addition should be easy to implement and there shouldn't be many restrictions on how you implement it within sqlitefid. Otherwise, all of the above sounds great, thanks Tessa!