Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

improve the extracted_files download page #329

Closed
mmguero opened this issue Jan 9, 2024 · 2 comments
Closed

improve the extracted_files download page #329

mmguero opened this issue Jan 9, 2024 · 2 comments
Assignees
@mmguero
Labels
carving Relating to carving (extraction) of files from traffic and the scanning of those files enhancement New feature or request
Milestone
v24.02.0

Comments

@mmguero
Copy link
Collaborator

mmguero commented Jan 9, 2024

Here's what the /extracted-files/ screen looks like now:

image

We could improve this I think in a few ways:

  • general visual updates
  • put files in columns: based on filename, we can check to see if it matches the usual pattern and put the protocol, FUID, UID, timestamp and extension into individual columns
  • perhaps, based on those same things, make the FUID and UID columns actually a hyperlink into dashboards with those applied as filters
  • file size?
  • mime type?

This page is generated by this python script
@mmguero mmguero added carving Relating to carving (extraction) of files from traffic and the scanning of those files enhancement New feature or request labels Jan 9, 2024
@mmguero mmguero modified the milestone: v24.02.0 Jan 9, 2024
@mmguero mmguero mentioned this issue Jan 9, 2024
Closed
@mmguero mmguero modified the milestones: z.staging, v24.03.0 Jan 15, 2024
@mmguero mmguero modified the milestones: v24.03.0, v24.02.0 Feb 1, 2024
@mmguero mmguero self-assigned this Feb 1, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 1, 2024
@mmguero
work in progress for idaholab#329, improve extracted_files download page
2b9abe0
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 1, 2024
@mmguero
work in progress for idaholab#329, improve extracted_files download page
5ee1bc7
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 1, 2024
@mmguero
work in progress for idaholab#329, improve extracted_files download page
d7c75f4
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 1, 2024
@mmguero
work in progress for idaholab#329, improve extracted_files download page
d46acbd
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 2, 2024
@mmguero
work in progress for idaholab#329, improve extracted_files download page
0b51a04
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 2, 2024
@mmguero
work in progress for idaholab#329, improve extracted_files download page
1916158
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 2, 2024
@mmguero
work in progress for idaholab#329, improve extracted_files download page
3454c0e
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 2, 2024
@mmguero
work in progress for idaholab#329, improve extracted_files download page
b1a9aa9
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 2, 2024
@mmguero
work in progress for idaholab#329, improve extracted_files download page
86eb52b
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 2, 2024
@mmguero
work in progress for idaholab#329, improve extracted_files download page
6821248
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 2, 2024
@mmguero
work in progress for idaholab#329, improve extracted_files download page
a3bb3e0
@mmguero
Copy link
Collaborator Author

mmguero commented Feb 2, 2024 •
edited
Loading

It's shaping up nicely, here's what it looks like so far:

image

There is a new variable in zeek.env with the rest of the EXTRACTED_FILE... variables:

# Whether or not to use libmagic to show MIME types for Zeek-extracted files served
EXTRACTED_FILE_HTTP_SERVER_MAGIC=false

If set to true, it will also do a libmagic lookup on each file:

image

The links are (by column):

  • Download - link for the file download
  • Type - link to IANA.org based on the file type
  • IDs - Zeek UID and FUID (which are both stored in event.id) link to Dashboards with that filter applied

Things still to work on:

  • the time frame for the aforementioned dashboard link does not appear to work.
  • remove "unknown" from list of IDs
  • handle "XOR decrypted from FEieEe1f1SI6YJk4H5" (make source XOR and put FUID in IDs)
  • filter <error> from source

mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 2, 2024
@mmguero
work in progress for idaholab#329, improve extracted_files download page
3889eb6
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 2, 2024
@mmguero
work in progress for idaholab#329, improve extracted_files download page
9a5ef36
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 5, 2024
@mmguero
work in progress for idaholab#329, improve extracted_files download page
27185b4
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 5, 2024
@mmguero
work in progress for idaholab#329, improve extracted_files download page
51df7de
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 5, 2024
@mmguero
work in progress for idaholab#329, improve extracted_files download page
ed6cd76
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 5, 2024
@mmguero
work in progress for idaholab#329, improve documentation for extracte…
c6ead7e 
…d_files download page
@mmguero
Copy link
Collaborator Author

mmguero commented Feb 5, 2024

I've added documentation for the improvements as well. I think I'm satisfied with this feature now, closing.

@mmguero mmguero closed this as completed Feb 5, 2024
This was referenced Feb 14, 2024
Merged
Merged
mmguero added a commit that referenced this issue Feb 15, 2024
@mmguero
Merge pull request #412 from idaholab/v24.02.0_merge_idaholab
a885008 
Malcolm v24.02.0 contains new features, improvements, bug fixes and component version updates.

v24.01.0...v24.02.0

* Features and enhancements
    - [Hedgehog Linux SD card image for Raspberry Pi](https://idaholab.github.io/Malcolm/docs/hedgehog-raspi-build.html#HedgehogRaspiBuild) (#250; special thanks to @aut0exec for his work on this)
    - allow configuration of Arkime's ILM/ISM settings (#300)
    - add option for customizing which log types get NetBox enrichment (#316)
    - improve the extracted_files download page (#329)
    - include missing aggregations in API bucket queries (#386)
    - more intelligent .env file checking on startup (#387)
    - Malcolm report to itself on capture statistics (#395)
    - link to Dashboards/Arkime from NetBox devices view (#410)
    - changed default PCAP storage format to zstd(3) for new installations
    - various documentation updates and improvements
    - changed back to using official Zeek .deb files rather than building from source to reduce build times
* Component version updates
    - Arkime to [v5.0.0](https://github.com/arkime/arkime/blob/6914792d86ecba0009f9b49dabb1aa987e46ad26/CHANGELOG#L33-L130)
    - Capa to [v7.0.1](https://github.com/mandiant/capa/releases)
    - YARA to [v4.5.0](https://github.com/VirusTotal/yara/releases)
    - Beats to [v8.12.1](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.12.1.html)
    - Logstash to [v8.12.1](https://www.elastic.co/guide/en/logstash/current/logstash-8-12-1.html)
    - Zeek to [v6.1.1](https://github.com/zeek/zeek/releases/tag/v6.1.1)
* Bug fixes
    - pivot links from Arkime to Kibana in external elasticsearch are not working (#335)
    - redirect /dashboards/ link to Kibana in NGINX proxy in elasticsearch/kibana-based deployment (#403)
    - allow netbox-restore and netbox-backup to specify container name (#337)
    - fuzzy matching for manufacturers based on OUI to NetBox list is not very good (#393) (and [updated documentation](https://idaholab.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch))
    - source.ip and destination.ip not set for parsed files.log entries for uploaded PCAP (#401)
    - event.severity_tags is not being assigned correctly based on rule.category (#402)
    - basic authentication breaks with special characters (#404)
    - changed some Logstash Ruby variables from global (`$`) to instance (`@`) (see ["avoiding concurrency issues"](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html#plugins-filters-ruby-concurrency))
* Configuration changes (in [environment variables](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/idaholab/Malcolm/blob/v24.02.0/config))
    * these variables in [`arkime.env`](https://github.com/idaholab/Malcolm/blob/main/config/arkime.env.example) to allow configuration of Arkime's ILM/ISM settings (#300)
    ```
    # These variables manage setting for Arkime's ILM/ISM features (https://arkime.com/faq#ilm)
    # Whether or not Arkime should perform index management
    INDEX_MANAGEMENT_ENABLED=false
    # Time in hours/days before moving to warm and force merge (number followed by h or d)
    INDEX_MANAGEMENT_OPTIMIZATION_PERIOD=30d
    # Time in hours/days before deleting index (number followed by h or d)
    INDEX_MANAGEMENT_RETENTION_TIME=90d
    # Number of replicas for older sessions indices
    INDEX_MANAGEMENT_OLDER_SESSION_REPLICAS=0
    # Number of weeks of history to retain
    INDEX_MANAGEMENT_HISTORY_RETENTION_WEEKS=13
    # Number of segments to optimize sessions for
    INDEX_MANAGEMENT_SEGMENTS=1
    # Whether or not Arkime should use a hot/warm design (storing non-session data in a warm index)
    INDEX_MANAGEMENT_HOT_WARM_ENABLED=false
    ```
    * these variables in [`dashboards.env`](https://github.com/idaholab/Malcolm/blob/main/config/dashboards.env.example) to override the values automatically configured for pivot links (#335) and `/dashboard/` redirect (#403) for Elasticsearch backend
    ```
    # These values are used to handle the Arkime value actions to pivot from Arkime
    #   to Dashboards. The nginx-proxy container's entrypoint will try to formulate
    #   them automatically, but they may be specified explicitly here.
    NGINX_DASHBOARDS_PREFIX=
    NGINX_DASHBOARDS_PROXY_PASS=
    ```
    * these variables in [`logstash.env`](https://github.com/idaholab/Malcolm/blob/main/config/logstash.env.example) for customizing which log types get NetBox enrichment (#316) and customizing which types of Zeek logs will be ignored (dropped) by LogStash
    ```
    # Which types of logs will be enriched via NetBox (comma-separated list of provider.dataset, or the string all to enrich all logs)
    LOGSTASH_NETBOX_ENRICHMENT_DATASETS=suricata.alert,zeek.conn,zeek.known_hosts,zeek.known_services,zeek.notice,zeek.signatures,zeek.software,zeek.weird
    ```
    ```
    # Zeek log types that will be ignored (dropped) by LogStash
    LOGSTASH_ZEEK_IGNORED_LOGS=analyzer,broker,bsap_ip_unknown,bsap_serial_unknown,capture_loss,cluster,config,ecat_arp_info,loaded_scripts,packet_filter,png,print,prof,reporter,stats,stderr,stdout
    ```
    * these variables in [`netbox-common.env`](https://github.com/idaholab/Malcolm/blob/main/config/netbox-common.env.example) for adjusting [matching device manufacturers to OUIs](https://idaholab.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch) in NetBox autopopulation
    ```
    # Customize manufacturer matching/creation with LOGSTASH_NETBOX_AUTO_POPULATE (see logstash.env)
    NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER=true
    NETBOX_DEFAULT_FUZZY_THRESHOLD=0.95
    ```
    * these variables in [suricata-live.env](https://github.com/idaholab/Malcolm/blob/main/config/suricata-live.env.example) and [zeek-live.env](https://github.com/idaholab/Malcolm/blob/main/config/zeek-live.env.example) that can be used to configure Malcolm reporting to itself on its Zeek and Suricata live capture statistics (#395)
    ```
    # Whether or not enable capture statistics and include them in eve.json
    SURICATA_STATS_ENABLED=false
    SURICATA_STATS_EVE_ENABLED=false
    SURICATA_STATS_INTERVAL=30
    SURICATA_STATS_DECODER_EVENTS=false
    ```
    ```
    # Set ZEEK_DISABLE_STATS to blank to generate stats.log and capture_loss.log
    ZEEK_DISABLE_STATS=true
    ```
    * this variable in [zeek.env](https://github.com/idaholab/Malcolm/blob/main/config/zeek.env.example) related to the improvements to the extracted_files download page (#329)
    ```
    # Whether or not to use libmagic to show MIME types for Zeek-extracted files served
    EXTRACTED_FILE_HTTP_SERVER_MAGIC=false
    ```
mmguero added a commit to cisagov/Malcolm that referenced this issue Feb 15, 2024
@mmguero
Merge pull request #297 from cisagov/v24.02.0_merge_cisagov
98ac2fb 
Malcolm v24.02.0 contains new features, improvements, bug fixes and component version updates.

v24.01.0...v24.02.0

* Features and enhancements
    - [Hedgehog Linux SD card image for Raspberry Pi](https://cisagov.github.io/Malcolm/docs/hedgehog-raspi-build.html#HedgehogRaspiBuild) (idaholab#250; special thanks to @aut0exec for his work on this)
    - allow configuration of Arkime's ILM/ISM settings (idaholab#300)
    - add option for customizing which log types get NetBox enrichment (idaholab#316)
    - improve the extracted_files download page (idaholab#329)
    - include missing aggregations in API bucket queries (idaholab#386)
    - more intelligent .env file checking on startup (idaholab#387)
    - Malcolm report to itself on capture statistics (idaholab#395)
    - link to Dashboards/Arkime from NetBox devices view (idaholab#410)
    - changed default PCAP storage format to zstd(3) for new installations
    - various documentation updates and improvements
    - changed back to using official Zeek .deb files rather than building from source to reduce build times
* Component version updates
    - Arkime to [v5.0.0](https://github.com/arkime/arkime/blob/6914792d86ecba0009f9b49dabb1aa987e46ad26/CHANGELOG#L33-L130)
    - Capa to [v7.0.1](https://github.com/mandiant/capa/releases)
    - YARA to [v4.5.0](https://github.com/VirusTotal/yara/releases)
    - Beats to [v8.12.1](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.12.1.html)
    - Logstash to [v8.12.1](https://www.elastic.co/guide/en/logstash/current/logstash-8-12-1.html)
    - Zeek to [v6.1.1](https://github.com/zeek/zeek/releases/tag/v6.1.1)
* Bug fixes
    - pivot links from Arkime to Kibana in external elasticsearch are not working (idaholab#335)
    - redirect /dashboards/ link to Kibana in NGINX proxy in elasticsearch/kibana-based deployment (idaholab#403)
    - allow netbox-restore and netbox-backup to specify container name (idaholab#337)
    - fuzzy matching for manufacturers based on OUI to NetBox list is not very good (idaholab#393) (and [updated documentation](https://cisagov.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch))
    - source.ip and destination.ip not set for parsed files.log entries for uploaded PCAP (idaholab#401)
    - event.severity_tags is not being assigned correctly based on rule.category (idaholab#402)
    - basic authentication breaks with special characters (idaholab#404)
    - changed some Logstash Ruby variables from global (`$`) to instance (`@`) (see ["avoiding concurrency issues"](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html#plugins-filters-ruby-concurrency))
* Configuration changes (in [environment variables](https://cisagov.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/cisagov/Malcolm/blob/v24.02.0/config))
    * these variables in [`arkime.env`](https://github.com/cisagov/Malcolm/blob/main/config/arkime.env.example) to allow configuration of Arkime's ILM/ISM settings (idaholab#300)
    ```
    # These variables manage setting for Arkime's ILM/ISM features (https://arkime.com/faq#ilm)
    # Whether or not Arkime should perform index management
    INDEX_MANAGEMENT_ENABLED=false
    # Time in hours/days before moving to warm and force merge (number followed by h or d)
    INDEX_MANAGEMENT_OPTIMIZATION_PERIOD=30d
    # Time in hours/days before deleting index (number followed by h or d)
    INDEX_MANAGEMENT_RETENTION_TIME=90d
    # Number of replicas for older sessions indices
    INDEX_MANAGEMENT_OLDER_SESSION_REPLICAS=0
    # Number of weeks of history to retain
    INDEX_MANAGEMENT_HISTORY_RETENTION_WEEKS=13
    # Number of segments to optimize sessions for
    INDEX_MANAGEMENT_SEGMENTS=1
    # Whether or not Arkime should use a hot/warm design (storing non-session data in a warm index)
    INDEX_MANAGEMENT_HOT_WARM_ENABLED=false
    ```
    * these variables in [`dashboards.env`](https://github.com/cisagov/Malcolm/blob/main/config/dashboards.env.example) to override the values automatically configured for pivot links (idaholab#335) and `/dashboard/` redirect (idaholab#403) for Elasticsearch backend
    ```
    # These values are used to handle the Arkime value actions to pivot from Arkime
    #   to Dashboards. The nginx-proxy container's entrypoint will try to formulate
    #   them automatically, but they may be specified explicitly here.
    NGINX_DASHBOARDS_PREFIX=
    NGINX_DASHBOARDS_PROXY_PASS=
    ```
    * these variables in [`logstash.env`](https://github.com/cisagov/Malcolm/blob/main/config/logstash.env.example) for customizing which log types get NetBox enrichment (idaholab#316) and customizing which types of Zeek logs will be ignored (dropped) by LogStash
    ```
    # Which types of logs will be enriched via NetBox (comma-separated list of provider.dataset, or the string all to enrich all logs)
    LOGSTASH_NETBOX_ENRICHMENT_DATASETS=suricata.alert,zeek.conn,zeek.known_hosts,zeek.known_services,zeek.notice,zeek.signatures,zeek.software,zeek.weird
    ```
    ```
    # Zeek log types that will be ignored (dropped) by LogStash
    LOGSTASH_ZEEK_IGNORED_LOGS=analyzer,broker,bsap_ip_unknown,bsap_serial_unknown,capture_loss,cluster,config,ecat_arp_info,loaded_scripts,packet_filter,png,print,prof,reporter,stats,stderr,stdout
    ```
    * these variables in [`netbox-common.env`](https://github.com/cisagov/Malcolm/blob/main/config/netbox-common.env.example) for adjusting [matching device manufacturers to OUIs](https://cisagov.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch) in NetBox autopopulation
    ```
    # Customize manufacturer matching/creation with LOGSTASH_NETBOX_AUTO_POPULATE (see logstash.env)
    NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER=true
    NETBOX_DEFAULT_FUZZY_THRESHOLD=0.95
    ```
    * these variables in [suricata-live.env](https://github.com/cisagov/Malcolm/blob/main/config/suricata-live.env.example) and [zeek-live.env](https://github.com/cisagov/Malcolm/blob/main/config/zeek-live.env.example) that can be used to configure Malcolm reporting to itself on its Zeek and Suricata live capture statistics (idaholab#395)
    ```
    # Whether or not enable capture statistics and include them in eve.json
    SURICATA_STATS_ENABLED=false
    SURICATA_STATS_EVE_ENABLED=false
    SURICATA_STATS_INTERVAL=30
    SURICATA_STATS_DECODER_EVENTS=false
    ```
    ```
    # Set ZEEK_DISABLE_STATS to blank to generate stats.log and capture_loss.log
    ZEEK_DISABLE_STATS=true
    ```
    * this variable in [zeek.env](https://github.com/cisagov/Malcolm/blob/main/config/zeek.env.example) related to the improvements to the extracted_files download page (idaholab#329)
    ```
    # Whether or not to use libmagic to show MIME types for Zeek-extracted files served
    EXTRACTED_FILE_HTTP_SERVER_MAGIC=false
    ```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mmguero mmguero

Labels
carving Relating to carving (extraction) of files from traffic and the scanning of those files enhancement New feature or request
Projects
Malcolm
Status: Released
Milestone
v24.02.0
Development

No branches or pull requests
1 participant
@mmguero