Hedgehog Linux SD card image for Raspberry Pi #250
Assignees
Labels
enhancement
New feature or request
iso
relating to the ISO-installed environment for Malcolm and/or Hedgehog
sensor
For issues dealing with the Hedgehog OS capture sensor
Milestone
Comments
mmguero
added
enhancement
2 tasks
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Jan 17, 2024
…o/interface/flask-2.3.2 Bump flask from 2.2.3 to 2.3.2 in /sensor-iso/interface
|
Working the issues out on this now, I'll put some here as I find them:
It's looking good, though. |
Closed
|
Things are working well here from what I can see. Still testing but I'm feeling pretty good about this. |
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Jan 29, 2024
This was referenced Feb 14, 2024
Merged
mmguero
added a commit
that referenced
this issue
Feb 15, 2024
Malcolm v24.02.0 contains new features, improvements, bug fixes and component version updates. v24.01.0...v24.02.0 * Features and enhancements - [Hedgehog Linux SD card image for Raspberry Pi](https://idaholab.github.io/Malcolm/docs/hedgehog-raspi-build.html#HedgehogRaspiBuild) (#250; special thanks to @aut0exec for his work on this) - allow configuration of Arkime's ILM/ISM settings (#300) - add option for customizing which log types get NetBox enrichment (#316) - improve the extracted_files download page (#329) - include missing aggregations in API bucket queries (#386) - more intelligent .env file checking on startup (#387) - Malcolm report to itself on capture statistics (#395) - link to Dashboards/Arkime from NetBox devices view (#410) - changed default PCAP storage format to zstd(3) for new installations - various documentation updates and improvements - changed back to using official Zeek .deb files rather than building from source to reduce build times * Component version updates - Arkime to [v5.0.0](https://github.com/arkime/arkime/blob/6914792d86ecba0009f9b49dabb1aa987e46ad26/CHANGELOG#L33-L130) - Capa to [v7.0.1](https://github.com/mandiant/capa/releases) - YARA to [v4.5.0](https://github.com/VirusTotal/yara/releases) - Beats to [v8.12.1](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.12.1.html) - Logstash to [v8.12.1](https://www.elastic.co/guide/en/logstash/current/logstash-8-12-1.html) - Zeek to [v6.1.1](https://github.com/zeek/zeek/releases/tag/v6.1.1) * Bug fixes - pivot links from Arkime to Kibana in external elasticsearch are not working (#335) - redirect /dashboards/ link to Kibana in NGINX proxy in elasticsearch/kibana-based deployment (#403) - allow netbox-restore and netbox-backup to specify container name (#337) - fuzzy matching for manufacturers based on OUI to NetBox list is not very good (#393) (and [updated documentation](https://idaholab.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch)) - source.ip and destination.ip not set for parsed files.log entries for uploaded PCAP (#401) - event.severity_tags is not being assigned correctly based on rule.category (#402) - basic authentication breaks with special characters (#404) - changed some Logstash Ruby variables from global (`$`) to instance (`@`) (see ["avoiding concurrency issues"](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html#plugins-filters-ruby-concurrency)) * Configuration changes (in [environment variables](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/idaholab/Malcolm/blob/v24.02.0/config)) * these variables in [`arkime.env`](https://github.com/idaholab/Malcolm/blob/main/config/arkime.env.example) to allow configuration of Arkime's ILM/ISM settings (#300) ``` # These variables manage setting for Arkime's ILM/ISM features (https://arkime.com/faq#ilm) # Whether or not Arkime should perform index management INDEX_MANAGEMENT_ENABLED=false # Time in hours/days before moving to warm and force merge (number followed by h or d) INDEX_MANAGEMENT_OPTIMIZATION_PERIOD=30d # Time in hours/days before deleting index (number followed by h or d) INDEX_MANAGEMENT_RETENTION_TIME=90d # Number of replicas for older sessions indices INDEX_MANAGEMENT_OLDER_SESSION_REPLICAS=0 # Number of weeks of history to retain INDEX_MANAGEMENT_HISTORY_RETENTION_WEEKS=13 # Number of segments to optimize sessions for INDEX_MANAGEMENT_SEGMENTS=1 # Whether or not Arkime should use a hot/warm design (storing non-session data in a warm index) INDEX_MANAGEMENT_HOT_WARM_ENABLED=false ``` * these variables in [`dashboards.env`](https://github.com/idaholab/Malcolm/blob/main/config/dashboards.env.example) to override the values automatically configured for pivot links (#335) and `/dashboard/` redirect (#403) for Elasticsearch backend ``` # These values are used to handle the Arkime value actions to pivot from Arkime # to Dashboards. The nginx-proxy container's entrypoint will try to formulate # them automatically, but they may be specified explicitly here. NGINX_DASHBOARDS_PREFIX= NGINX_DASHBOARDS_PROXY_PASS= ``` * these variables in [`logstash.env`](https://github.com/idaholab/Malcolm/blob/main/config/logstash.env.example) for customizing which log types get NetBox enrichment (#316) and customizing which types of Zeek logs will be ignored (dropped) by LogStash ``` # Which types of logs will be enriched via NetBox (comma-separated list of provider.dataset, or the string all to enrich all logs) LOGSTASH_NETBOX_ENRICHMENT_DATASETS=suricata.alert,zeek.conn,zeek.known_hosts,zeek.known_services,zeek.notice,zeek.signatures,zeek.software,zeek.weird ``` ``` # Zeek log types that will be ignored (dropped) by LogStash LOGSTASH_ZEEK_IGNORED_LOGS=analyzer,broker,bsap_ip_unknown,bsap_serial_unknown,capture_loss,cluster,config,ecat_arp_info,loaded_scripts,packet_filter,png,print,prof,reporter,stats,stderr,stdout ``` * these variables in [`netbox-common.env`](https://github.com/idaholab/Malcolm/blob/main/config/netbox-common.env.example) for adjusting [matching device manufacturers to OUIs](https://idaholab.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch) in NetBox autopopulation ``` # Customize manufacturer matching/creation with LOGSTASH_NETBOX_AUTO_POPULATE (see logstash.env) NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER=true NETBOX_DEFAULT_FUZZY_THRESHOLD=0.95 ``` * these variables in [suricata-live.env](https://github.com/idaholab/Malcolm/blob/main/config/suricata-live.env.example) and [zeek-live.env](https://github.com/idaholab/Malcolm/blob/main/config/zeek-live.env.example) that can be used to configure Malcolm reporting to itself on its Zeek and Suricata live capture statistics (#395) ``` # Whether or not enable capture statistics and include them in eve.json SURICATA_STATS_ENABLED=false SURICATA_STATS_EVE_ENABLED=false SURICATA_STATS_INTERVAL=30 SURICATA_STATS_DECODER_EVENTS=false ``` ``` # Set ZEEK_DISABLE_STATS to blank to generate stats.log and capture_loss.log ZEEK_DISABLE_STATS=true ``` * this variable in [zeek.env](https://github.com/idaholab/Malcolm/blob/main/config/zeek.env.example) related to the improvements to the extracted_files download page (#329) ``` # Whether or not to use libmagic to show MIME types for Zeek-extracted files served EXTRACTED_FILE_HTTP_SERVER_MAGIC=false ```
mmguero
added a commit
to cisagov/Malcolm
that referenced
this issue
Feb 15, 2024
Malcolm v24.02.0 contains new features, improvements, bug fixes and component version updates. v24.01.0...v24.02.0 * Features and enhancements - [Hedgehog Linux SD card image for Raspberry Pi](https://cisagov.github.io/Malcolm/docs/hedgehog-raspi-build.html#HedgehogRaspiBuild) (idaholab#250; special thanks to @aut0exec for his work on this) - allow configuration of Arkime's ILM/ISM settings (idaholab#300) - add option for customizing which log types get NetBox enrichment (idaholab#316) - improve the extracted_files download page (idaholab#329) - include missing aggregations in API bucket queries (idaholab#386) - more intelligent .env file checking on startup (idaholab#387) - Malcolm report to itself on capture statistics (idaholab#395) - link to Dashboards/Arkime from NetBox devices view (idaholab#410) - changed default PCAP storage format to zstd(3) for new installations - various documentation updates and improvements - changed back to using official Zeek .deb files rather than building from source to reduce build times * Component version updates - Arkime to [v5.0.0](https://github.com/arkime/arkime/blob/6914792d86ecba0009f9b49dabb1aa987e46ad26/CHANGELOG#L33-L130) - Capa to [v7.0.1](https://github.com/mandiant/capa/releases) - YARA to [v4.5.0](https://github.com/VirusTotal/yara/releases) - Beats to [v8.12.1](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.12.1.html) - Logstash to [v8.12.1](https://www.elastic.co/guide/en/logstash/current/logstash-8-12-1.html) - Zeek to [v6.1.1](https://github.com/zeek/zeek/releases/tag/v6.1.1) * Bug fixes - pivot links from Arkime to Kibana in external elasticsearch are not working (idaholab#335) - redirect /dashboards/ link to Kibana in NGINX proxy in elasticsearch/kibana-based deployment (idaholab#403) - allow netbox-restore and netbox-backup to specify container name (idaholab#337) - fuzzy matching for manufacturers based on OUI to NetBox list is not very good (idaholab#393) (and [updated documentation](https://cisagov.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch)) - source.ip and destination.ip not set for parsed files.log entries for uploaded PCAP (idaholab#401) - event.severity_tags is not being assigned correctly based on rule.category (idaholab#402) - basic authentication breaks with special characters (idaholab#404) - changed some Logstash Ruby variables from global (`$`) to instance (`@`) (see ["avoiding concurrency issues"](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html#plugins-filters-ruby-concurrency)) * Configuration changes (in [environment variables](https://cisagov.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/cisagov/Malcolm/blob/v24.02.0/config)) * these variables in [`arkime.env`](https://github.com/cisagov/Malcolm/blob/main/config/arkime.env.example) to allow configuration of Arkime's ILM/ISM settings (idaholab#300) ``` # These variables manage setting for Arkime's ILM/ISM features (https://arkime.com/faq#ilm) # Whether or not Arkime should perform index management INDEX_MANAGEMENT_ENABLED=false # Time in hours/days before moving to warm and force merge (number followed by h or d) INDEX_MANAGEMENT_OPTIMIZATION_PERIOD=30d # Time in hours/days before deleting index (number followed by h or d) INDEX_MANAGEMENT_RETENTION_TIME=90d # Number of replicas for older sessions indices INDEX_MANAGEMENT_OLDER_SESSION_REPLICAS=0 # Number of weeks of history to retain INDEX_MANAGEMENT_HISTORY_RETENTION_WEEKS=13 # Number of segments to optimize sessions for INDEX_MANAGEMENT_SEGMENTS=1 # Whether or not Arkime should use a hot/warm design (storing non-session data in a warm index) INDEX_MANAGEMENT_HOT_WARM_ENABLED=false ``` * these variables in [`dashboards.env`](https://github.com/cisagov/Malcolm/blob/main/config/dashboards.env.example) to override the values automatically configured for pivot links (idaholab#335) and `/dashboard/` redirect (idaholab#403) for Elasticsearch backend ``` # These values are used to handle the Arkime value actions to pivot from Arkime # to Dashboards. The nginx-proxy container's entrypoint will try to formulate # them automatically, but they may be specified explicitly here. NGINX_DASHBOARDS_PREFIX= NGINX_DASHBOARDS_PROXY_PASS= ``` * these variables in [`logstash.env`](https://github.com/cisagov/Malcolm/blob/main/config/logstash.env.example) for customizing which log types get NetBox enrichment (idaholab#316) and customizing which types of Zeek logs will be ignored (dropped) by LogStash ``` # Which types of logs will be enriched via NetBox (comma-separated list of provider.dataset, or the string all to enrich all logs) LOGSTASH_NETBOX_ENRICHMENT_DATASETS=suricata.alert,zeek.conn,zeek.known_hosts,zeek.known_services,zeek.notice,zeek.signatures,zeek.software,zeek.weird ``` ``` # Zeek log types that will be ignored (dropped) by LogStash LOGSTASH_ZEEK_IGNORED_LOGS=analyzer,broker,bsap_ip_unknown,bsap_serial_unknown,capture_loss,cluster,config,ecat_arp_info,loaded_scripts,packet_filter,png,print,prof,reporter,stats,stderr,stdout ``` * these variables in [`netbox-common.env`](https://github.com/cisagov/Malcolm/blob/main/config/netbox-common.env.example) for adjusting [matching device manufacturers to OUIs](https://cisagov.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch) in NetBox autopopulation ``` # Customize manufacturer matching/creation with LOGSTASH_NETBOX_AUTO_POPULATE (see logstash.env) NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER=true NETBOX_DEFAULT_FUZZY_THRESHOLD=0.95 ``` * these variables in [suricata-live.env](https://github.com/cisagov/Malcolm/blob/main/config/suricata-live.env.example) and [zeek-live.env](https://github.com/cisagov/Malcolm/blob/main/config/zeek-live.env.example) that can be used to configure Malcolm reporting to itself on its Zeek and Suricata live capture statistics (idaholab#395) ``` # Whether or not enable capture statistics and include them in eve.json SURICATA_STATS_ENABLED=false SURICATA_STATS_EVE_ENABLED=false SURICATA_STATS_INTERVAL=30 SURICATA_STATS_DECODER_EVENTS=false ``` ``` # Set ZEEK_DISABLE_STATS to blank to generate stats.log and capture_loss.log ZEEK_DISABLE_STATS=true ``` * this variable in [zeek.env](https://github.com/cisagov/Malcolm/blob/main/config/zeek.env.example) related to the improvements to the extracted_files download page (idaholab#329) ``` # Whether or not to use libmagic to show MIME types for Zeek-extracted files served EXTRACTED_FILE_HTTP_SERVER_MAGIC=false ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
From GitHub user @furrnace via cisagov#277
This is an interesting idea and one that would provide some value to the project I'm sure. The main issues would be around making sure everything compiles and runs on arm64 and whether or not the Debian live-build installer can generate Pi-compatible images. Cool idea, and something I think we should look into.
The text was updated successfully, but these errors were encountered: