Unable to generate file list on Netgear WAX620

[m-1-k-3]

Hi, currently I do some evaluations of STACS - you can also see our integration into the Firmware Scanner EMBA here: e-m-b-a/emba#207

During these tasks I have done some tests on the Netgear firmware WAX620 from here: https://www.netgear.com/support/product/wax630.aspx#Firmware%20Version%209.5.4.6

During these tests STACS is running into the following error:

└─$ stacs --debug --rule-pack ~/git-repos/stacs-rules/credential.json ~/firmware/emba_logs_manual/netgear-WAX620/firmware                                     254 ⨯
2022-01-07 12:40:17,738 - 1934072 - [INFO] Attempting to load rule pack from /home/m1k3/git-repos/stacs-rules/credential.json
2022-01-07 12:40:17,739 - 1934072 - [INFO] Attempting to get a list of files to scan from /home/m1k3/firmware/emba_logs_manual/netgear-WAX620/firmware
2022-01-07 12:40:22,741 - 1934072 - [ERROR] Unable to generate file list: Unable to extract archive /home/m1k3/firmware/emba_logs_manual/netgear-WAX620/firmware/firmware_binwalk_emba/_firmware.extracted/_nand-ipq807x-apps.img.extracted/195C.ubi_ubi_extracted/195C.ubi/_img-1966538225_vol-kernel.ubifs-0.extracted/_Image.extracted/805788.xz to 805788: Corrupt input data

Before running STACS the extractor of EMBA has extracted the firmware and then I started STACS the shown way.

Is there something like a "ignore errors" option?

[darkarnium]

Hey there,

Thanks for the report!

Adding a flag to allow skipping unprocessable archives with a warning is definitely a great idea, and shouldn't be too much effort to implement. I'll take a look at adding support for this shortly.

I'll also have a look into this specific archive to see whether the format is non-standard, and if so, whether there's anything that can be done to have STACS support it.

[m-1-k-3]

It is also possible that the file is corrupt as we try to extract everything that is possible with different mechanisms and there it happens that files are broken.

[darkarnium]

Hey there,

Looks like you're correct for this one.

According to my tests using the same firmware image (WAX620_WAX630_firmware_V9.5.4.6.zip) the xz file appears to be corrupt when extracted from the UBIFS:

vagrant@ubuntu-bionic:~$ ls -lah 805788.xz
-rw-rw-r-- 1 vagrant vagrant 3.4M Jan  7 18:46 805788.xz
vagrant@ubuntu-bionic:~$ unxz 805788.xz
unxz: 805788.xz: Compressed data is corrupt

That said, with the changes staged ready for 0.4.3 you can now skip these unprocessable / corrupt archives using the "skip unprocessable" flag. This defaults to False so that it's not a breaking change, and so the user has to explicitly request this behaviour to avoid confusion:

$ stacs --help
Usage: stacs [OPTIONS] PATH

  STACS - Static Token And Credential Scanner.

Options:
  --version               Show the version and exit.
  --debug                 Increase verbosity of logs for debugging
  --rule-pack TEXT        The path to the rule pack to load.
  --ignore-list TEXT      The path to the ignore list to load (if required).
  --skip-unprocessable    Skip unprocessable / corrupt archives with a
                          warning.
  --cache-directory TEXT  The path to use as a cache - used when unpacking
                          archives.
  --help                  Show this message and exit.

When using this new option you will still be warned for every file which is considered unprocessable, but STACS will not error and exit:

[...] [WARNING] Skipping file at [...]/example/badarchive.xz due to error when processing: Unable to extract archive [...]

I'm just running a few manual regression tests before I cut a new release, which will trigger an upload to both PyPI and Dockerhub shortly afterwards.

Thanks again for the report, and please let us know if you're happy that this change resolves this issue.

[darkarnium]

Hey there,

This has just been released, hopefully this resolves this issue :)

Cheers

[m-1-k-3]

This looks really good. Thank you for the update. I will include this update also into EMBA.

Keep the great work!