allow netbox-restore and netbox-backup to specify container name #337
Assignees
Labels
cloud
Relating to deployment of Malcolm in the cloud and/or with Kubernetes
enhancement
New feature or request
netbox
Related to Malcolm's use of NetBox
Milestone
Comments
mmguero
added
netbox
mmguero
changed the title
mmguero
changed the title
|
tested restore and it worked find on my K3s cluster, marking as closed |
This was referenced Feb 14, 2024
Merged
mmguero
added a commit
that referenced
this issue
Feb 15, 2024
Malcolm v24.02.0 contains new features, improvements, bug fixes and component version updates. v24.01.0...v24.02.0 * Features and enhancements - [Hedgehog Linux SD card image for Raspberry Pi](https://idaholab.github.io/Malcolm/docs/hedgehog-raspi-build.html#HedgehogRaspiBuild) (#250; special thanks to @aut0exec for his work on this) - allow configuration of Arkime's ILM/ISM settings (#300) - add option for customizing which log types get NetBox enrichment (#316) - improve the extracted_files download page (#329) - include missing aggregations in API bucket queries (#386) - more intelligent .env file checking on startup (#387) - Malcolm report to itself on capture statistics (#395) - link to Dashboards/Arkime from NetBox devices view (#410) - changed default PCAP storage format to zstd(3) for new installations - various documentation updates and improvements - changed back to using official Zeek .deb files rather than building from source to reduce build times * Component version updates - Arkime to [v5.0.0](https://github.com/arkime/arkime/blob/6914792d86ecba0009f9b49dabb1aa987e46ad26/CHANGELOG#L33-L130) - Capa to [v7.0.1](https://github.com/mandiant/capa/releases) - YARA to [v4.5.0](https://github.com/VirusTotal/yara/releases) - Beats to [v8.12.1](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.12.1.html) - Logstash to [v8.12.1](https://www.elastic.co/guide/en/logstash/current/logstash-8-12-1.html) - Zeek to [v6.1.1](https://github.com/zeek/zeek/releases/tag/v6.1.1) * Bug fixes - pivot links from Arkime to Kibana in external elasticsearch are not working (#335) - redirect /dashboards/ link to Kibana in NGINX proxy in elasticsearch/kibana-based deployment (#403) - allow netbox-restore and netbox-backup to specify container name (#337) - fuzzy matching for manufacturers based on OUI to NetBox list is not very good (#393) (and [updated documentation](https://idaholab.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch)) - source.ip and destination.ip not set for parsed files.log entries for uploaded PCAP (#401) - event.severity_tags is not being assigned correctly based on rule.category (#402) - basic authentication breaks with special characters (#404) - changed some Logstash Ruby variables from global (`$`) to instance (`@`) (see ["avoiding concurrency issues"](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html#plugins-filters-ruby-concurrency)) * Configuration changes (in [environment variables](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/idaholab/Malcolm/blob/v24.02.0/config)) * these variables in [`arkime.env`](https://github.com/idaholab/Malcolm/blob/main/config/arkime.env.example) to allow configuration of Arkime's ILM/ISM settings (#300) ``` # These variables manage setting for Arkime's ILM/ISM features (https://arkime.com/faq#ilm) # Whether or not Arkime should perform index management INDEX_MANAGEMENT_ENABLED=false # Time in hours/days before moving to warm and force merge (number followed by h or d) INDEX_MANAGEMENT_OPTIMIZATION_PERIOD=30d # Time in hours/days before deleting index (number followed by h or d) INDEX_MANAGEMENT_RETENTION_TIME=90d # Number of replicas for older sessions indices INDEX_MANAGEMENT_OLDER_SESSION_REPLICAS=0 # Number of weeks of history to retain INDEX_MANAGEMENT_HISTORY_RETENTION_WEEKS=13 # Number of segments to optimize sessions for INDEX_MANAGEMENT_SEGMENTS=1 # Whether or not Arkime should use a hot/warm design (storing non-session data in a warm index) INDEX_MANAGEMENT_HOT_WARM_ENABLED=false ``` * these variables in [`dashboards.env`](https://github.com/idaholab/Malcolm/blob/main/config/dashboards.env.example) to override the values automatically configured for pivot links (#335) and `/dashboard/` redirect (#403) for Elasticsearch backend ``` # These values are used to handle the Arkime value actions to pivot from Arkime # to Dashboards. The nginx-proxy container's entrypoint will try to formulate # them automatically, but they may be specified explicitly here. NGINX_DASHBOARDS_PREFIX= NGINX_DASHBOARDS_PROXY_PASS= ``` * these variables in [`logstash.env`](https://github.com/idaholab/Malcolm/blob/main/config/logstash.env.example) for customizing which log types get NetBox enrichment (#316) and customizing which types of Zeek logs will be ignored (dropped) by LogStash ``` # Which types of logs will be enriched via NetBox (comma-separated list of provider.dataset, or the string all to enrich all logs) LOGSTASH_NETBOX_ENRICHMENT_DATASETS=suricata.alert,zeek.conn,zeek.known_hosts,zeek.known_services,zeek.notice,zeek.signatures,zeek.software,zeek.weird ``` ``` # Zeek log types that will be ignored (dropped) by LogStash LOGSTASH_ZEEK_IGNORED_LOGS=analyzer,broker,bsap_ip_unknown,bsap_serial_unknown,capture_loss,cluster,config,ecat_arp_info,loaded_scripts,packet_filter,png,print,prof,reporter,stats,stderr,stdout ``` * these variables in [`netbox-common.env`](https://github.com/idaholab/Malcolm/blob/main/config/netbox-common.env.example) for adjusting [matching device manufacturers to OUIs](https://idaholab.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch) in NetBox autopopulation ``` # Customize manufacturer matching/creation with LOGSTASH_NETBOX_AUTO_POPULATE (see logstash.env) NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER=true NETBOX_DEFAULT_FUZZY_THRESHOLD=0.95 ``` * these variables in [suricata-live.env](https://github.com/idaholab/Malcolm/blob/main/config/suricata-live.env.example) and [zeek-live.env](https://github.com/idaholab/Malcolm/blob/main/config/zeek-live.env.example) that can be used to configure Malcolm reporting to itself on its Zeek and Suricata live capture statistics (#395) ``` # Whether or not enable capture statistics and include them in eve.json SURICATA_STATS_ENABLED=false SURICATA_STATS_EVE_ENABLED=false SURICATA_STATS_INTERVAL=30 SURICATA_STATS_DECODER_EVENTS=false ``` ``` # Set ZEEK_DISABLE_STATS to blank to generate stats.log and capture_loss.log ZEEK_DISABLE_STATS=true ``` * this variable in [zeek.env](https://github.com/idaholab/Malcolm/blob/main/config/zeek.env.example) related to the improvements to the extracted_files download page (#329) ``` # Whether or not to use libmagic to show MIME types for Zeek-extracted files served EXTRACTED_FILE_HTTP_SERVER_MAGIC=false ```
mmguero
added a commit
to cisagov/Malcolm
that referenced
this issue
Feb 15, 2024
Malcolm v24.02.0 contains new features, improvements, bug fixes and component version updates. v24.01.0...v24.02.0 * Features and enhancements - [Hedgehog Linux SD card image for Raspberry Pi](https://cisagov.github.io/Malcolm/docs/hedgehog-raspi-build.html#HedgehogRaspiBuild) (idaholab#250; special thanks to @aut0exec for his work on this) - allow configuration of Arkime's ILM/ISM settings (idaholab#300) - add option for customizing which log types get NetBox enrichment (idaholab#316) - improve the extracted_files download page (idaholab#329) - include missing aggregations in API bucket queries (idaholab#386) - more intelligent .env file checking on startup (idaholab#387) - Malcolm report to itself on capture statistics (idaholab#395) - link to Dashboards/Arkime from NetBox devices view (idaholab#410) - changed default PCAP storage format to zstd(3) for new installations - various documentation updates and improvements - changed back to using official Zeek .deb files rather than building from source to reduce build times * Component version updates - Arkime to [v5.0.0](https://github.com/arkime/arkime/blob/6914792d86ecba0009f9b49dabb1aa987e46ad26/CHANGELOG#L33-L130) - Capa to [v7.0.1](https://github.com/mandiant/capa/releases) - YARA to [v4.5.0](https://github.com/VirusTotal/yara/releases) - Beats to [v8.12.1](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.12.1.html) - Logstash to [v8.12.1](https://www.elastic.co/guide/en/logstash/current/logstash-8-12-1.html) - Zeek to [v6.1.1](https://github.com/zeek/zeek/releases/tag/v6.1.1) * Bug fixes - pivot links from Arkime to Kibana in external elasticsearch are not working (idaholab#335) - redirect /dashboards/ link to Kibana in NGINX proxy in elasticsearch/kibana-based deployment (idaholab#403) - allow netbox-restore and netbox-backup to specify container name (idaholab#337) - fuzzy matching for manufacturers based on OUI to NetBox list is not very good (idaholab#393) (and [updated documentation](https://cisagov.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch)) - source.ip and destination.ip not set for parsed files.log entries for uploaded PCAP (idaholab#401) - event.severity_tags is not being assigned correctly based on rule.category (idaholab#402) - basic authentication breaks with special characters (idaholab#404) - changed some Logstash Ruby variables from global (`$`) to instance (`@`) (see ["avoiding concurrency issues"](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html#plugins-filters-ruby-concurrency)) * Configuration changes (in [environment variables](https://cisagov.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/cisagov/Malcolm/blob/v24.02.0/config)) * these variables in [`arkime.env`](https://github.com/cisagov/Malcolm/blob/main/config/arkime.env.example) to allow configuration of Arkime's ILM/ISM settings (idaholab#300) ``` # These variables manage setting for Arkime's ILM/ISM features (https://arkime.com/faq#ilm) # Whether or not Arkime should perform index management INDEX_MANAGEMENT_ENABLED=false # Time in hours/days before moving to warm and force merge (number followed by h or d) INDEX_MANAGEMENT_OPTIMIZATION_PERIOD=30d # Time in hours/days before deleting index (number followed by h or d) INDEX_MANAGEMENT_RETENTION_TIME=90d # Number of replicas for older sessions indices INDEX_MANAGEMENT_OLDER_SESSION_REPLICAS=0 # Number of weeks of history to retain INDEX_MANAGEMENT_HISTORY_RETENTION_WEEKS=13 # Number of segments to optimize sessions for INDEX_MANAGEMENT_SEGMENTS=1 # Whether or not Arkime should use a hot/warm design (storing non-session data in a warm index) INDEX_MANAGEMENT_HOT_WARM_ENABLED=false ``` * these variables in [`dashboards.env`](https://github.com/cisagov/Malcolm/blob/main/config/dashboards.env.example) to override the values automatically configured for pivot links (idaholab#335) and `/dashboard/` redirect (idaholab#403) for Elasticsearch backend ``` # These values are used to handle the Arkime value actions to pivot from Arkime # to Dashboards. The nginx-proxy container's entrypoint will try to formulate # them automatically, but they may be specified explicitly here. NGINX_DASHBOARDS_PREFIX= NGINX_DASHBOARDS_PROXY_PASS= ``` * these variables in [`logstash.env`](https://github.com/cisagov/Malcolm/blob/main/config/logstash.env.example) for customizing which log types get NetBox enrichment (idaholab#316) and customizing which types of Zeek logs will be ignored (dropped) by LogStash ``` # Which types of logs will be enriched via NetBox (comma-separated list of provider.dataset, or the string all to enrich all logs) LOGSTASH_NETBOX_ENRICHMENT_DATASETS=suricata.alert,zeek.conn,zeek.known_hosts,zeek.known_services,zeek.notice,zeek.signatures,zeek.software,zeek.weird ``` ``` # Zeek log types that will be ignored (dropped) by LogStash LOGSTASH_ZEEK_IGNORED_LOGS=analyzer,broker,bsap_ip_unknown,bsap_serial_unknown,capture_loss,cluster,config,ecat_arp_info,loaded_scripts,packet_filter,png,print,prof,reporter,stats,stderr,stdout ``` * these variables in [`netbox-common.env`](https://github.com/cisagov/Malcolm/blob/main/config/netbox-common.env.example) for adjusting [matching device manufacturers to OUIs](https://cisagov.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch) in NetBox autopopulation ``` # Customize manufacturer matching/creation with LOGSTASH_NETBOX_AUTO_POPULATE (see logstash.env) NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER=true NETBOX_DEFAULT_FUZZY_THRESHOLD=0.95 ``` * these variables in [suricata-live.env](https://github.com/cisagov/Malcolm/blob/main/config/suricata-live.env.example) and [zeek-live.env](https://github.com/cisagov/Malcolm/blob/main/config/zeek-live.env.example) that can be used to configure Malcolm reporting to itself on its Zeek and Suricata live capture statistics (idaholab#395) ``` # Whether or not enable capture statistics and include them in eve.json SURICATA_STATS_ENABLED=false SURICATA_STATS_EVE_ENABLED=false SURICATA_STATS_INTERVAL=30 SURICATA_STATS_DECODER_EVENTS=false ``` ``` # Set ZEEK_DISABLE_STATS to blank to generate stats.log and capture_loss.log ZEEK_DISABLE_STATS=true ``` * this variable in [zeek.env](https://github.com/cisagov/Malcolm/blob/main/config/zeek.env.example) related to the improvements to the extracted_files download page (idaholab#329) ``` # Whether or not to use libmagic to show MIME types for Zeek-extracted files served EXTRACTED_FILE_HTTP_SERVER_MAGIC=false ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When running
netbox-restoreandnetbox-backupwith Kubernetes this is done by specifying the service to thePodExecfunction inmalcolm_kubernetes.py.Some users have indicated that if they are running sidecars it's necessary to specify the container name specifically. See #400 and the associated commits.
The text was updated successfully, but these errors were encountered: