provide way to access extracted_files on Hedgehog from Malcolm

[mmguero]

EDIT: See the comment below for the direction we decided to go for this and its progress.

---

I'm not sure what the best way to do this is, but wanted to get the idea down.

Malcolm provides the /extracted_files/ interface for browsing the carved files it processes.

Hedgehog also carves files, but it just stores them locally and doesn't really provide a way to get to them other than logging into the sensor and grabbing them manually.

However other than Arkime's PCAP payload retrieval, there's not really a good established way to surface that kind of a reachback from Malcolm to the sensor. Some careful planning and design would be needed.

Tangentially related are #330 and #329

[mmguero]

Andy Wick (Arkime) suggested it would be possible to do this via an arkime viewer plugin. As we're already using Arkime viewer to retrieve PCAP payload, this might be a good way to do it.

[mmguero]

The other way to do it (maybe easier) is to reuse the existing code in Malcolm for the download interface (as it supports zipping/encrypting the file, etc.) and then somehow proxy those requests through Malcolm to that service on hedgehog. We'd need to make sure we use the same code to restrict access to the Malcolm IP via ACL:

$ rg acl hedgehog-iso/
...

hedgehog-iso/interface/sensor_ctl/control_vars.conf
15:export ARKIME_PACKET_ACL=

hedgehog-iso/interface/sensor_ctl/supervisor.init/arkime_config_populate.sh
114:  # update the firewall ACL (via ufw) to allow retrieval of packets
...

[mmguero]

I wonder if using nginx with something like dynamic upstreams would do the trick.

• https://tenzer.dk/nginx-with-dynamic-upstreams/
• https://serverfault.com/questions/975228/replacing-uri-when-using-dynamic-hosts-in-nginx-reverse-proxy
• https://stackoverflow.com/questions/42383118/how-to-dynamically-add-an-upstream-in-nginx
• https://sosedoff.com/2012/06/11/dynamic-nginx-upstreams-with-lua-and-redis.html
• https://serverfault.com/questions/1100763/please-help-trying-to-set-dynamic-reverse-nginx-proxy-but-it-fails

[mmguero]

Here's how the work in progress is going:

Similar to how the arkime viewer session can retrieve PCAP payloads from port 8005/tcp over TLS on Hedgehog, the extracted files web interface is now available on port 8006/tcp over TLS as well:

However, in addition to being accessible directly from the hedgehog over that port (which would not be the common way people would access it), Malcolm will proxy requests to /hh-extracted-files/<hedgehog IP or hostname>/ to the Hedgehog (see the address bar in this screenshot):

Access to this port on the Hedgehog is restricted via firewall ACL to IP addresses that have been explicitly allowlisted, which would basically mean set to the Malcolm server's IP address.

For comparison's sake, here's a screenshot of the UI for the Malcolm server itself:

I think this is the bulk of this feature, the rest of it is window decoration to try to make it more convenient to navigate to the appropriate place. I'm working on putting some value action links into Arkime (arkime/arkime#2731 will make this even more convenient for this feature).

We'll see what else needs to happen. But it's coming along pretty good.

[mmguero]

I'm also working on integrating these download links (whether to Malcolm or Hedgehog) into Dashboards and Arkime:

• Arkime, via the Sessions tab when viewing for zeek.files logs

• Dashboards, in the Files dashboard

The way I'm doing this at the moment is by generating the correct download URI during Logstash enrichment base on the host.name field, for whichever Hedgehog device or for Malcolm itself. The drawback to this is that if the hostname of the hedgehog changes in the future, the link will no longer be correct. The only way I can think of around this would be to somehow store a persistent separate database mapping the node name to its host or IP address. Arkime is already doing this, in the stats index, but it's not a perfect solution because we're talking about Zeek logs, not Arkime sessions. I think it's okay: worst case scenario is the user can still navigate manually to the hh-extracted-files link (as described above) and find it there.