Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

curl rc file for connecting to external OpenSearch without auth enabled causes logstash startup to fail #209

Closed
mmguero opened this issue Jun 14, 2023 · 0 comments
Assignees
Labels
bug Something isn't working logstash Relating to Malcolm's use of Logstash opensearch Relating to Malcolm's use of OpenSearch
Milestone

Comments

@mmguero
Copy link
Collaborator

mmguero commented Jun 14, 2023

In malcolm_utils.py we have this:

###################################################################################################
# parse a curl-formatted config file, with special handling for user:password and URL
# see https://everything.curl.dev/cmdline/configfile
# e.g.:
#
# given .opensearch.primary.curlrc containing:
# -
# user: "sikari:changethis"
# insecure
# -
#
# ParseCurlFile('.opensearch.primary.curlrc') returns:
#   {
#    'user': 'sikari',
#    'password': 'changethis',
#    'insecure': ''
#   }
def ParseCurlFile(curlCfgFileName):
    result = defaultdict(lambda: None)
...

That first line needs to be changed to:

def ParseCurlFile(curlCfgFileName):
    result = defaultdict(lambda: '')
...

To avoid a startup error like this:

logstash_1            | 2023-06-14 14:56:22,791 INFO spawned: 'logstash' with pid 24089
logstash_1            | Traceback (most recent call last):
logstash_1            |   File "<string>", line 1, in <module>
logstash_1            | TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

However I'm still not 100% sure this will fix the problem, need to repro locally. Reported by user Carlos Lopez in slack.

@mmguero mmguero added bug Something isn't working opensearch Relating to Malcolm's use of OpenSearch logstash Relating to Malcolm's use of Logstash labels Jun 14, 2023
@mmguero mmguero self-assigned this Jun 14, 2023
@mmguero mmguero added this to Malcolm Jun 14, 2023
@mmguero mmguero moved this to Todo (develop) in Malcolm Jun 14, 2023
@mmguero mmguero moved this from Todo (develop) to In Progress in Malcolm Jun 14, 2023
@mmguero mmguero added this to the v23.06.0 milestone Jun 14, 2023
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jun 14, 2023
…without auth enabled causes logstash startup to fail
@mmguero mmguero moved this from In Progress to Done in Malcolm Jul 18, 2023
@mmguero mmguero closed this as completed Jul 18, 2023
mmguero added a commit to cisagov/Malcolm that referenced this issue Jul 19, 2023
Malcolm v23.07.0 is a feature release with a number of improvements, bux fixes and component updates.

v23.05.1...v23.07.0

* New features
    - scan docker images built via GitHub actions for vulnerabilities using Trivy (idaholab#218)
    - document building and deplolying Malcolm with an AWS AMI image (idaholab#205)
    - handle Arkime field actions (idaholab#200)
    - kubernetes: document how to get running on Amazon EKS (idaholab#194)
    - Populate NetBox inventory via passively-gathered network traffic metadata (basic functionality, work in progress) (idaholab#135)

* Enhancements
    - use .tar.xz instead of .tar.gz for packaging Malcolm docker images for better compression (and smaller ISO file size)
    - Malcolm documentation edits (idaholab#204)
    - add option to enable SSH via password in hedgehog's configure-interfaces.py script (idaholab#158)
    - updated "Network Traffic Analysis with Malcolm" slides
    - use an init container in Kubernetes container startup to ensure necessary directories get created under PersistentVolume objects before startup
    - improvements to identifying source of third-party logs sent via fluent bit
    - don't do unnecessary clone of Zeek plugins, just install using URL
    - parse [bacnet_device_control.log](https://github.com/cisagov/icsnpp-bacnet/#device-control-log-bacnet_device_controllog) produced by the icsnpp-bacnet parser for Zeek

* Bug fixes
    - maxlogins value includes tmux sessions, can lock user out of SSH (idaholab#214)
    - curl rc file for connecting to external OpenSearch without auth enabled causes logstash startup to fail (idaholab#209)
    - failure to parse some suricata alerts due to integer type which should be indexed as long (idaholab#206)
    - netbox-restore doesn't work in Kubernetes (idaholab#202)
    - PCAP File with no `-` in pcapng Fails to Upload (#265)
    - disable NetBox telemetry

* Component version updates
    - Alpine (docker container image base) to [v3.18.0](https://www.alpinelinux.org/posts/Alpine-3.18.0-released.html)
    - Arkime to [v4.3.2](https://github.com/arkime/arkime/blob/8bd9d1ccaf3214eeb07da910c45d6172f9ff4ca8/CHANGELOG#L40-L55)
    - capa to [v6.0.0](https://github.com/mandiant/capa/releases/tag/v6.0.0)
    - filebeat to [v8.8.2](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.8.2.html)
    - NetBox to [v3.5.4](https://github.com/netbox-community/netbox/releases/tag/v3.5.4)
    - OpenSearch and OpenSearch Dashboards to [v2.8.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.8.0.md)
    - Supercronic to [v0.2.25](https://github.com/aptible/supercronic/releases/tag/v0.2.25)
    - YARA to [v4.3.2](https://github.com/VirusTotal/yara/releases/tag/v4.3.2)
    - Zeek to [v5.2.2](https://github.com/zeek/zeek/releases/tag/v5.2.2)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from [https://malcolm.fyi/](https://malcolm.fyi/docs/download.html).
mmguero added a commit that referenced this issue Jul 19, 2023
Malcolm v23.07.0 is a feature release with a number of improvements, bux fixes and component updates.

v23.05.1...v23.07.0

* New features
    - scan docker images built via GitHub actions for vulnerabilities using Trivy (#218)
    - document building and deplolying Malcolm with an AWS AMI image (#205)
    - handle Arkime field actions (#200)
    - kubernetes: document how to get running on Amazon EKS (#194)
    - Populate NetBox inventory via passively-gathered network traffic metadata (basic functionality, work in progress) (#135)

* Enhancements
    - use .tar.xz instead of .tar.gz for packaging Malcolm docker images for better compression (and smaller ISO file size)
    - Malcolm documentation edits (#204)
    - add option to enable SSH via password in hedgehog's configure-interfaces.py script (#158)
    - updated "Network Traffic Analysis with Malcolm" slides
    - use an init container in Kubernetes container startup to ensure necessary directories get created under PersistentVolume objects before startup
    - improvements to identifying source of third-party logs sent via fluent bit
    - don't do unnecessary clone of Zeek plugins, just install using URL
    - parse [bacnet_device_control.log](https://github.com/cisagov/icsnpp-bacnet/#device-control-log-bacnet_device_controllog) produced by the icsnpp-bacnet parser for Zeek

* Bug fixes
    - maxlogins value includes tmux sessions, can lock user out of SSH (#214)
    - curl rc file for connecting to external OpenSearch without auth enabled causes logstash startup to fail (#209)
    - failure to parse some suricata alerts due to integer type which should be indexed as long (#206)
    - netbox-restore doesn't work in Kubernetes (#202)
    - PCAP File with no `-` in pcapng Fails to Upload (cisagov#265)
    - disable NetBox telemetry

* Component version updates
    - Alpine (docker container image base) to [v3.18.0](https://www.alpinelinux.org/posts/Alpine-3.18.0-released.html)
    - Arkime to [v4.3.2](https://github.com/arkime/arkime/blob/8bd9d1ccaf3214eeb07da910c45d6172f9ff4ca8/CHANGELOG#L40-L55)
    - capa to [v6.0.0](https://github.com/mandiant/capa/releases/tag/v6.0.0)
    - filebeat to [v8.8.2](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.8.2.html)
    - NetBox to [v3.5.4](https://github.com/netbox-community/netbox/releases/tag/v3.5.4)
    - OpenSearch and OpenSearch Dashboards to [v2.8.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.8.0.md)
    - Supercronic to [v0.2.25](https://github.com/aptible/supercronic/releases/tag/v0.2.25)
    - YARA to [v4.3.2](https://github.com/VirusTotal/yara/releases/tag/v4.3.2)
    - Zeek to [v5.2.2](https://github.com/zeek/zeek/releases/tag/v5.2.2)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from [https://malcolm.fyi/](https://malcolm.fyi/docs/download.html).
@mmguero mmguero moved this from Done to Released in Malcolm Jul 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working logstash Relating to Malcolm's use of Logstash opensearch Relating to Malcolm's use of OpenSearch
Projects
Status: Released
Development

No branches or pull requests

1 participant