Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SFTP upload broken due to dollar sign(s) in openssl-encrypted password #110

Closed
mmguero opened this issue Aug 1, 2022 · 1 comment · Fixed by #112 or cisagov/Malcolm#212
Closed

SFTP upload broken due to dollar sign(s) in openssl-encrypted password #110

mmguero opened this issue Aug 1, 2022 · 1 comment · Fixed by #112 or cisagov/Malcolm#212
Assignees
@mmguero
Labels
bug Something isn't working docker Relating to docker and docker-compose as used by Malcolm upload Relating to PCAP and/or Zeek log ingestion

Comments

@mmguero
Copy link
Collaborator

mmguero commented Aug 1, 2022

I just discovered an issue with the way docker interprets environment variables passed in from .env files.

For the SFTP PCAP upload, the file-upload/docker-entrypoint.sh script sets the user for the sftp upload using the MALCOLM_USERNAME and MALCOLM_PASSWORD environment variables.

However, openssl generates an encrypted password that looks something like $1$o74qn1nt$PRB/XhSybr4k98Y7Fqy9l/. This environment variable doesn't come across "cleanly" to the upload service container, so the SFTP login fails.

To fix this, the auth_setup script (control.py) is going to base64-encode the value of this environment variable, then the entrypoint for the upload container will decode it.
@mmguero mmguero added bug Something isn't working docker Relating to docker and docker-compose as used by Malcolm upload Relating to PCAP and/or Zeek log ingestion labels Aug 1, 2022
@mmguero mmguero self-assigned this Aug 1, 2022
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Aug 1, 2022
@mmguero
fix idaholab#110, sftp upload broken due to dollar sign(s) in openssl…
305a266 
…-encrypted password
@mmguero
Copy link
Collaborator Author

mmguero commented Aug 1, 2022

Fixed for v6.2.0

@mmguero mmguero closed this as completed Aug 1, 2022
This was referenced Aug 3, 2022
Merged
Merged
mmguero added a commit that referenced this issue Aug 3, 2022
@mmguero
Merge pull request #112 from idaholab/v6.2.0_merge_idaholab
c2c3df0 
merge v6.2.0 development into main

* Improvements
    - #109: break Zeek/Suricata into two containers: one for "live" capture and one for uploaded PCAPs
        + give option to disable capture interface hardware offloading and adjust ring buffer sizes for standalone Malcolm capture
        + Zeek and Suricata images are now configured to not drop privileges at init (in order to be able to set permissions for network capture), but supervisord will drop privileges for all of its child processes before they execute to maintain security posture
        + include headers needed to build Zeek af_packet plugin in Zeek docker container
        + updated README to describe methods for capturing local traffic with standalone Malcolm
        + same images will be used for `zeek` and `zeek-live` containers, as well as for `suricata` and `suricata-live` containers, respectively
        + use the same scripts `zeekdeploy.sh` to configure and run Zeek on both Hedgehog and in the Malcolm `zeek` docker images
        + prevent "live" and "non-live" Zeek containers from both trying to update intel indicators at the same time
    - Speed up build time by getting official Debian suricata packages from backports rather than building from source
    - Added Suricata rule update cron jobs
    - Added documentation (in the form of comments) to all docker-compose file variables

* Bugs
    - Fix #107: expand action/result meaning in DNP3 (and other?) dashboards
        - Clean up some `Nul` values that could appear in Zeek logs
        - improve mapping of BACnet actions
    - Fix #108: export PCAP not working from Arkime sessions without "Arkime Sessions"
    - Fix #110: SFTP upload broken due to dollar sign(s) in openssl-encrypted password
        + prompt in `install.py --configure` whether or not to expose this port to external hosts
    - Fix an issue that could prevent some zeek logs from being parsed correctly if they contained non-ASCII charactters
mmguero added a commit to cisagov/Malcolm that referenced this issue Aug 3, 2022
@mmguero
Merge pull request #212 from cisagov/v6.2.0_merge_cisagov
28ee931 
merge v6.2.0 into main

* Improvements
    - idaholab#109: break Zeek/Suricata into two containers: one for "live" capture and one for uploaded PCAPs
        + give option to disable capture interface hardware offloading and adjust ring buffer sizes for standalone Malcolm capture
        + Zeek and Suricata images are now configured to not drop privileges at init (in order to be able to set permissions for network capture), but supervisord will drop privileges for all of its child processes before they execute to maintain security posture
        + include headers needed to build Zeek af_packet plugin in Zeek docker container
        + updated README to describe methods for capturing local traffic with standalone Malcolm
        + same images will be used for `zeek` and `zeek-live` containers, as well as for `suricata` and `suricata-live` containers, respectively
        + use the same scripts `zeekdeploy.sh` to configure and run Zeek on both Hedgehog and in the Malcolm `zeek` docker images
        + prevent "live" and "non-live" Zeek containers from both trying to update intel indicators at the same time
    - Speed up build time by getting official Debian suricata packages from backports rather than building from source
    - Added Suricata rule update cron jobs
    - Added documentation (in the form of comments) to all docker-compose file variables

* Bugs
    - Fix idaholab#107: expand action/result meaning in DNP3 (and other?) dashboards
        - Clean up some `Nul` values that could appear in Zeek logs
        - improve mapping of BACnet actions
    - Fix idaholab#108: export PCAP not working from Arkime sessions without "Arkime Sessions"
    - Fix idaholab#110: SFTP upload broken due to dollar sign(s) in openssl-encrypted password
        + prompt in `install.py --configure` whether or not to expose this port to external hosts
    - Fix an issue that could prevent some zeek logs from being parsed correctly if they contained non-ASCII charactters
This was referenced Sep 7, 2022
Merged
Merged
mmguero added a commit that referenced this issue Sep 7, 2022
@mmguero
Merge pull request #118 from idaholab/v6.3.0_merge_idaholab
cc83826 
* New Features
    * Support remote OpenSearch instance/cluster as alternative to local containerized instance (#110)
    * Documentation and convenience scripts for configuring ingestion of third-party logs, and basic parsing/normalizing of Fluent-Bit's Windows event logs
    * S7comm Plus support and replaced Amazon S7comm parser with icsnpp-s7comm (#99)

* Version Bumps
    * OpenSearch and OpenSearch Dashboards to v2.2.1
    * Zeek to v5.0.1
    * Spicy to v1.5.1
    * spicy-plugin to v1.3.17
    * YARA to v4.2.3
    * Capa to v4.0.1

* Improvements
    * Major improvements to OPC UA Binary parser and supporting dashboards
    * Ensure that all containers are provided the same information about trusted CA certificates
    * changed list of "sensitive countries" to match U.S. Department of Energy Sensitive Country List
    * Increased maximum fields from 3,000 to 5,000
    * Standardized configuration and authentication for primary and secondary remote OpenSearch instances, and make sure that index templates are created on secondary remote OpenSearch instances
    * Expand and fix normalization of [network.direction](https://www.elastic.co/guide/en/ecs/current/ecs-network.html#field-network-direction) in lieu of using tags
    * Various tweaks and improvements to the `install.py` script for enabling/disabling some features

* Bugs Fixed
    * fields could be missing in Arkime due to a large number of concurrent requests (#115)
    * mapper_parsing_exception, TCP flag parsing problem (cisagov#214)
mmguero added a commit to cisagov/Malcolm that referenced this issue Sep 7, 2022
@mmguero
Merge pull request #218 from cisagov/v6.3.0_merge_cisagov
a749daf 
* New Features
    * Support remote OpenSearch instance/cluster as alternative to local containerized instance (idaholab#110)
    * Documentation and convenience scripts for configuring ingestion of third-party logs, and basic parsing/normalizing of Fluent-Bit's Windows event logs
    * S7comm Plus support and replaced Amazon S7comm parser with icsnpp-s7comm (idaholab#99)

* Version Bumps
    * OpenSearch and OpenSearch Dashboards to v2.2.1
    * Zeek to v5.0.1
    * Spicy to v1.5.1
    * spicy-plugin to v1.3.17
    * YARA to v4.2.3
    * Capa to v4.0.1

* Improvements
    * Major improvements to OPC UA Binary parser and supporting dashboards
    * Ensure that all containers are provided the same information about trusted CA certificates
    * changed list of "sensitive countries" to match U.S. Department of Energy Sensitive Country List
    * Increased maximum fields from 3,000 to 5,000
    * Standardized configuration and authentication for primary and secondary remote OpenSearch instances, and make sure that index templates are created on secondary remote OpenSearch instances
    * Expand and fix normalization of [network.direction](https://www.elastic.co/guide/en/ecs/current/ecs-network.html#field-network-direction) in lieu of using tags
    * Various tweaks and improvements to the `install.py` script for enabling/disabling some features

* Bugs Fixed
    * fields could be missing in Arkime due to a large number of concurrent requests (idaholab#115)
    * mapper_parsing_exception, TCP flag parsing problem (#214)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mmguero mmguero

Labels
bug Something isn't working docker Relating to docker and docker-compose as used by Malcolm upload Relating to PCAP and/or Zeek log ingestion
Projects
Malcolm
Status: Released
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

v6.2.0 development idaholab/Malcolm
V6.2.0 merge cisagov cisagov/Malcolm
1 participant
@mmguero