Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

time sync from hedgehog to Malcolm opensearch instance not working #324

Closed
mmguero opened this issue Jan 5, 2024 · 2 comments
Closed

time sync from hedgehog to Malcolm opensearch instance not working #324

mmguero opened this issue Jan 5, 2024 · 2 comments
Assignees
@mmguero
Labels
bug Something isn't working iso relating to the ISO-installed environment for Malcolm and/or Hedgehog regression It worked at one point... sensor For issues dealing with the Hedgehog OS capture sensor
Milestone
v24.01.0

Comments

@mmguero
Copy link
Collaborator

mmguero commented Jan 5, 2024

For some reason the htpdate process on Hedgehog Linux isn't getting the date/time correctly from a Malcolm instance's opensearch server at port 9200 like it used to be able to do.
@mmguero mmguero added bug Something isn't working regression It worked at one point... sensor For issues dealing with the Hedgehog OS capture sensor labels Jan 5, 2024
@mmguero mmguero added this to the v24.01.0 milestone Jan 5, 2024
@mmguero
Copy link
Collaborator Author

mmguero commented Jan 5, 2024

A temporary workaround if NTP is not available is to use 1.1.1.1 port 80 if you have internet access.

@mmguero
Copy link
Collaborator Author

mmguero commented Jan 5, 2024

Found by a colleague:

It's Debian's fault it looks like. I cloned https://salsa.debian.org/debian/htpdate and ran make https now ./htpdate -4 -a -d https://x.x.x.x:9200/ (that's my malcolm instance) works! Noticed after compiling it locally that ldd reports that libssl and libcrypto are requested where the version from the repos doesn't request them and htpdate isn't statically compiled.

@mmguero mmguero added the iso relating to the ISO-installed environment for Malcolm and/or Hedgehog label Jan 5, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jan 5, 2024
@mmguero
Fix idaholab#324 (time sync from hedgehog to Malcolm opensearch insta…
e0784e9 
…nce not working) by building htpdate from source with `make https` for ISO installers rather than installing version from the package repositories

Signed-off-by: Seth Grover <mero.mero.guero@gmail.com>
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jan 5, 2024
@mmguero
idaholab#324 github workflow update
3b4baf3
@mmguero mmguero self-assigned this Jan 5, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jan 9, 2024
@mmguero
update prompt to reflect URL is allow for htpdate sync (idaholab#324)
93bf0a4
@mmguero mmguero closed this as completed Jan 9, 2024
This was referenced Jan 17, 2024
Merged
Merged
mmguero added a commit that referenced this issue Jan 17, 2024
@mmguero
Merge pull request #383 from idaholab/v24.01.0_merge_idaholab
c78a043 
Malcolm v24.01.0 contains new features, improvements, bug fixes and component version updates.

v23.12.1...v24.0.1

* Features and enhancements
    + new Malcolm instance landing page (#252)
    + file carve download with password-protected .zip file (#288)
    + new "all files exept common plain text files" option for Malcolm's file carving to match Hedgehog capability (#290)
    + allow customizing indexes for logs written to OpenSearch/Elasticsearch (#313)
    + more consistently differentiate between uploaded and live-captured traffic (#321)
    + make download extracted file context item from Arkime smarter (#330)
    + improve netbox device type library import by using "official" import script (#384)
* Component version updates
    + Alpine Linux to [v3.19](https://alpinelinux.org/posts/Alpine-3.19.0-released.html) as the base for some Docker images
    + Fluent Bit to [v2.2.2](https://github.com/fluent/fluent-bit/releases/tag/v2.2.2)
    + Beats to [v8.11.4](https://www.elastic.co/guide/en/beats/libbeat/8.11/release-notes-8.11.4.html)
    + LogStash to [v8.11.4](https://www.elastic.co/guide/en/logstash/current/logstash-8-11-4.html)
* Bug fixes
    + Suricata Alerts dashboard "Alerts - Tags" visualization is useless (#314)
    + third party logs are not parsed correctly from fluentbit -> fluentd aggregator -> Malcolm (#318)
    + update document lookup APIs to search either network or host data (#322)
    + suricata rule update is broken (#323)
    + time sync from hedgehog to Malcolm opensearch instance not working (#324)
    + fix issue specifying database mode via command-line
    + have pruning of OpenSearch indices (based on size) include "other" Malcolm indices as well (e.g., nginx logs, system resources, third-party logs, etc.)
* Configuration changes (in [environment variables](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/idaholab/Malcolm/tree/v24.0.1/config))
    + added the following variables with relation to #313
        - added `ARKIME_ROTATE_INDEX` to [`arkime.env`](https://github.com/idaholab/Malcolm/tree/v24.0.1/arkime.env.example) with default value of `daily` (see [Arkime docs on rotateIndex](https://arkime.com/settings#rotateIndex))
        - added the following variables and defaults to [`opensearch.env`](https://github.com/idaholab/Malcolm/tree/v24.0.1/opensearch.env.example):
        ```
        # OpenSearch index patterns and timestamp fields
        # Index pattern for network traffic logs written via Logstash (e.g., Zeek logs, Suricata alerts)
        MALCOLM_NETWORK_INDEX_PATTERN=arkime_sessions3-*
        # Default time field to use for network traffic logs in Logstash and Dashboards
        MALCOLM_NETWORK_INDEX_TIME_FIELD=firstPacket
        # Suffix used to create index to which network traffic logs are written (supports Ruby strftime strings in %{})
        MALCOLM_NETWORK_INDEX_SUFFIX=%{%y%m%d}
        # Index pattern for other logs written via Logstash (e.g., nginx, beats, fluent-bit, etc.)
        MALCOLM_OTHER_INDEX_PATTERN=malcolm_beats_*
        # Default time field to use for other logs in Logstash and Dashboards
        MALCOLM_OTHER_INDEX_TIME_FIELD=@timestamp
        # Suffix used to create index to which other logs are written (supports Ruby strftime strings in %{})
        MALCOLM_OTHER_INDEX_SUFFIX=%{%y%m%d}
        # Index pattern used specifically by Arkime (will probably match MALCOLM_NETWORK_INDEX_PATTERN, should probably be arkime_sessions3-*)
        ARKIME_NETWORK_INDEX_PATTERN=arkime_sessions3-*
        # Default time field used by for sessions in Arkime viewer
        ARKIME_NETWORK_INDEX_TIME_FIELD=firstPacket
        ```
    + changed default for `EXTRACTED_FILE_HTTP_SERVER_KEY` to `infected` in [`zeek-secret.env`](https://github.com/idaholab/Malcolm/tree/v24.0.1/zeek-secret.env.example)
    + added `EXTRACTED_FILE_HTTP_SERVER_ZIP` with default value of `false` in [`zeek.env`](https://github.com/idaholab/Malcolm/tree/v24.0.1/zeek.env.example), see (#288)
mmguero added a commit to cisagov/Malcolm that referenced this issue Jan 17, 2024
@mmguero
Merge pull request #295 from cisagov/v24.01.0_merge_cisagov
2862d43 
Malcolm v24.01.0 contains new features, improvements, bug fixes and component version updates.

v23.12.1...v24.0.1

* Features and enhancements
    + new Malcolm instance landing page (idaholab#252)
    + file carve download with password-protected .zip file (idaholab#288)
    + new "all files exept common plain text files" option for Malcolm's file carving to match Hedgehog capability (idaholab#290)
    + allow customizing indexes for logs written to OpenSearch/Elasticsearch (idaholab#313)
    + more consistently differentiate between uploaded and live-captured traffic (idaholab#321)
    + make download extracted file context item from Arkime smarter (idaholab#330)
    + improve netbox device type library import by using "official" import script (idaholab#384)
* Component version updates
    + Alpine Linux to [v3.19](https://alpinelinux.org/posts/Alpine-3.19.0-released.html) as the base for some Docker images
    + Fluent Bit to [v2.2.2](https://github.com/fluent/fluent-bit/releases/tag/v2.2.2)
    + Beats to [v8.11.4](https://www.elastic.co/guide/en/beats/libbeat/8.11/release-notes-8.11.4.html)
    + LogStash to [v8.11.4](https://www.elastic.co/guide/en/logstash/current/logstash-8-11-4.html)
* Bug fixes
    + Suricata Alerts dashboard "Alerts - Tags" visualization is useless (idaholab#314)
    + third party logs are not parsed correctly from fluentbit -> fluentd aggregator -> Malcolm (idaholab#318)
    + update document lookup APIs to search either network or host data (idaholab#322)
    + suricata rule update is broken (idaholab#323)
    + time sync from hedgehog to Malcolm opensearch instance not working (idaholab#324)
    + fix issue specifying database mode via command-line
    + have pruning of OpenSearch indices (based on size) include "other" Malcolm indices as well (e.g., nginx logs, system resources, third-party logs, etc.)
* Configuration changes (in [environment variables](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/cisagov/Malcolm/tree/v24.0.1/config))
    + added the following variables with relation to idaholab#313
        - added `ARKIME_ROTATE_INDEX` to [`arkime.env`](https://github.com/cisagov/Malcolm/tree/v24.0.1/arkime.env.example) with default value of `daily` (see [Arkime docs on rotateIndex](https://arkime.com/settings#rotateIndex))
        - added the following variables and defaults to [`opensearch.env`](https://github.com/cisagov/Malcolm/tree/v24.0.1/opensearch.env.example):
        ```
        # OpenSearch index patterns and timestamp fields
        # Index pattern for network traffic logs written via Logstash (e.g., Zeek logs, Suricata alerts)
        MALCOLM_NETWORK_INDEX_PATTERN=arkime_sessions3-*
        # Default time field to use for network traffic logs in Logstash and Dashboards
        MALCOLM_NETWORK_INDEX_TIME_FIELD=firstPacket
        # Suffix used to create index to which network traffic logs are written (supports Ruby strftime strings in %{})
        MALCOLM_NETWORK_INDEX_SUFFIX=%{%y%m%d}
        # Index pattern for other logs written via Logstash (e.g., nginx, beats, fluent-bit, etc.)
        MALCOLM_OTHER_INDEX_PATTERN=malcolm_beats_*
        # Default time field to use for other logs in Logstash and Dashboards
        MALCOLM_OTHER_INDEX_TIME_FIELD=@timestamp
        # Suffix used to create index to which other logs are written (supports Ruby strftime strings in %{})
        MALCOLM_OTHER_INDEX_SUFFIX=%{%y%m%d}
        # Index pattern used specifically by Arkime (will probably match MALCOLM_NETWORK_INDEX_PATTERN, should probably be arkime_sessions3-*)
        ARKIME_NETWORK_INDEX_PATTERN=arkime_sessions3-*
        # Default time field used by for sessions in Arkime viewer
        ARKIME_NETWORK_INDEX_TIME_FIELD=firstPacket
        ```
    + changed default for `EXTRACTED_FILE_HTTP_SERVER_KEY` to `infected` in [`zeek-secret.env`](https://github.com/cisagov/Malcolm/tree/v24.0.1/zeek-secret.env.example)
    + added `EXTRACTED_FILE_HTTP_SERVER_ZIP` with default value of `false` in [`zeek.env`](https://github.com/cisagov/Malcolm/tree/v24.0.1/zeek.env.example), see (idaholab#288)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mmguero mmguero

Labels
bug Something isn't working iso relating to the ISO-installed environment for Malcolm and/or Hedgehog regression It worked at one point... sensor For issues dealing with the Hedgehog OS capture sensor
Projects
Malcolm
Status: Released
Milestone
v24.01.0
Development

No branches or pull requests
1 participant
@mmguero