Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kubernetes deployment - opensearch keystore #176

Closed
mmguero opened this issue Mar 29, 2023 · 2 comments
Closed

kubernetes deployment - opensearch keystore #176

mmguero opened this issue Mar 29, 2023 · 2 comments
Assignees
@mmguero
Labels
cloud Relating to deployment of Malcolm in the cloud and/or with Kubernetes opensearch Relating to Malcolm's use of OpenSearch
Milestone
v23.05.0

Comments

@mmguero
Copy link
Collaborator

mmguero commented Mar 29, 2023

Sub-issue of #149

in order to store creds for an email alerter, opensearch.keystore is created and populated by control.py at startup. It then has to be mounted into the container. I think this would work okay with a configmap but what about the initial creation? Today that's done inside of a container (either by docker run when Malcolm isn't started up yet or docker-compose exec if it is). Is there some way we can pass these secrets in to the container itself for the keystore to be created at the entrypoint, or do we still have to have them create it beforehand?
@mmguero mmguero added opensearch Relating to Malcolm's use of OpenSearch cloud Relating to deployment of Malcolm in the cloud and/or with Kubernetes labels Mar 29, 2023
@mmguero mmguero self-assigned this Apr 13, 2023
mmguero added a commit that referenced this issue Apr 13, 2023
@mmguero
for #176, bootstrap opensearch keystore
c4ef50b
mmguero added a commit that referenced this issue Apr 14, 2023
@mmguero
for #176, bootstrap opensearch keystore
db35047
mmguero added a commit that referenced this issue Apr 14, 2023
@mmguero
for #176, bootstrap opensearch keystore
ef69bdc
mmguero added a commit that referenced this issue Apr 14, 2023
@mmguero
for #176, bootstrap opensearch keystore
f4934f2
mmguero added a commit that referenced this issue Apr 14, 2023
@mmguero
for #176, bootstrap opensearch keystore
8e13221
@mmguero
Copy link
Collaborator Author

mmguero commented Apr 14, 2023

This will work very similar to how nginx-proxy/htadmin do it with htpasswd (see #169) :

# make sure the keystore file used by the tool (e.g., foobar) is copied or created
# into the correct location before the tool. starts up.
#
# example:
#   - /usr/share/foobar/config/persist/foobar.keystore
#       - "real" location of keystore file used by foobar
#   - /usr/share/foobar/config/bootstrap/foobar.keystore
#       - if "persist" keystore does't exist, but "bootstrap" keystore
#         does, then copy "bootstrap" keystore to "persist" keystore
#   - /usr/share/foobar/config/foobar.keystore
#       - symlinks to "persist" keystore (foobar requires this
#         to be the file it actually looks at)
#

These volume mounts and [volumes](https://github.com/idaholab/Malcolm/blob/8e13221081df87a318f2a81bc7c4bb06239c2699/kubernetes/02-opensearch.yml#L80-L85] will make sure that what's initially in the local Malcolm directory (if anything) gets put into ./config/bootstrap via configMap and then moved to ./config/persist/, else a new empty keystore will get created on startup and live from there in persist. Starting up again in subsequent runs, if something's already in ./config/persist that will take precedence.

@mmguero
Copy link
Collaborator Author

mmguero commented Apr 14, 2023

There is still the issue of initial population without docker, but from a kubernetes point of view this is working with the configmaps. There will be some overlap in #172 but for now I'm going to close this issue and worry about the creation part of it there.

@mmguero mmguero closed this as completed Apr 14, 2023
@mmguero mmguero added this to the v23.05.0 milestone Apr 26, 2023
This was referenced May 1, 2023
Merged
Merged
mmguero added a commit that referenced this issue May 1, 2023
@mmguero
Merge pull request #191 from idaholab/v23.05.0_merge_idaholab
3438f9c 
Malcolm v23.05.0

Malcolm v23.05.0 is a major release with new features, enhancements, component version updates and bug fixes.

IMPORTANT NOTE: Malcolm v23.05.0 has completely changed the way it manages its settings: rather than using environment variables found at the top of the `docker-compose.yml` file, it uses environment variables in `.env` files inside of the `config` directory. The locations of a number of configuration files have also changed. It's not recommended to update to Malcolm v23.05.0 from a previous version of Malcolm. Instead, shut down Malcolm, rename your old Malcolm installation directory to something else, and reconfigure Malcolm using `./scripts/configure` and `./scripts/auth_setup`.

v23.04.0...v23.05.0

* New features
    - integrate [ICSNPP-Synchrophasor](https://github.com/cisagov/icsnpp-synchrophasor/) parser (#190)
    - [End-to-end Malcolm and Hedgehog Linux ISO Installation]((https://idaholab.github.io/Malcolm/docs/malcolm-hedgehog-e2e-iso-install.html#InstallationExample) document (#181)
    - support Malcolm deployment with Kubernetes (#149)
        + see [Deploying Malcolm with Kubernetes](https://idaholab.github.io/Malcolm/docs/kubernetes.html#Kubernetes)
        + This could be considered a "beta" release for Malcolm deployment with Kubernetes, as there is still [some work](https://github.com/idaholab/Malcolm/issues?q=is%3Aissue+is%3Aopen+kubernetes) to be done in this area. Please let us know what issues or suggestions you have via the [issue tracker](https://github.com/idaholab/Malcolm/issues) or via email to [malcolm@inl.gov](mailto:malcolm@inl.gov?subject=Malcolm).
        + contributing issues:
            * inotify issue (#168)
            * htadmin/nginx and htpasswd (#169)
            * opensearch (#170)
            * uploading large PCAP files (#171)
            * script consolidation (#172)
            * documentation (#173)
            * user-defined persistent volumes (#174)
            * opensearch keystore (#176)
            * expose other TCP services (#183)
            * provide with filebeat access to nginx access and error logs (#186)
            * use Secrets for some environment variables instead of ConfigMaps (#189)

* Enhancements and fixes
    * remove name-map-ui container (#165) in favor of using NetBox for asset identification
    * Python script refactoring, consolidation and cleanup
    * standardization of Docker container entrypoints
    * create `./scripts/configure` alias for `./scripts/install.py --configure`

* Component version updates
    - Arkime to [v4.3.0](https://github.com/arkime/arkime/blob/fa0db2415bdc109be7a4dd8ee2c2838673980b5f/CHANGELOG#L33-L72)
    - Capa to [v5.1.0](https://github.com/mandiant/capa/releases/tag/v5.1.0)
    - Fluent Bit to [v2.1.2](https://fluentbit.io/announcements/v2.1.2/)
    - NetBox to [v3.5.0](https://github.com/netbox-community/netbox/releases/tag/v3.5.0)
    - NGINX to [v1.22.1](http://nginx.org/en/CHANGES-1.22)
    - Supercronic to [v0.2.24](https://github.com/aptible/supercronic/releases/tag/v0.2.24)
    - Suricata to [v6.0.10](https://suricata.io/2023/01/31/suricata-6-0-10-released/)
    - Yara to [v4.3.0](https://github.com/VirusTotal/yara/releases/tag/v4.3.0)
    - Zeek to [v5.2.1](https://github.com/zeek/zeek/releases/tag/v5.2.1)
mmguero added a commit to cisagov/Malcolm that referenced this issue May 1, 2023
@mmguero
Merge pull request #249 from cisagov/v23.05.0_merge_cisagov
c02a6e9 
Malcolm v23.05.0

Malcolm v23.05.0 is a major release with new features, enhancements, component version updates and bug fixes.

IMPORTANT NOTE: Malcolm v23.05.0 has completely changed the way it manages its settings: rather than using environment variables found at the top of the `docker-compose.yml` file, it uses environment variables in `.env` files inside of the `config` directory. The locations of a number of configuration files have also changed. It's not recommended to update to Malcolm v23.05.0 from a previous version of Malcolm. Instead, shut down Malcolm, rename your old Malcolm installation directory to something else, and reconfigure Malcolm using `./scripts/configure` and `./scripts/auth_setup`.

v23.04.0...v23.05.0

* New features
    - integrate [ICSNPP-Synchrophasor](https://github.com/cisagov/icsnpp-synchrophasor/) parser (idaholab#190)
    - [End-to-end Malcolm and Hedgehog Linux ISO Installation((cisagov://cisagov.github.io/Malcolm/docs/malcolm-hedgehog-e2e-iso-install.html#InstallationExample) document (idaholab#181)
    - support Malcolm deployment with Kubernetes (idaholab#149)
        + see [Deploying Malcolm with Kubernetes](https://idaholab.github.io/Malcolm/docs/kubernetes.html#Kubernetes)
        + This could be considered a "beta" release for Malcolm deployment with Kubernetes, as there is still [some work](https://github.com/idaholab/Malcolm/issues?q=is%3Aissue+is%3Aopen+kubernetes) to be done in this area. Please let us know what issues or suggestions you have via the [issue tracker](https://github.com/idaholab/Malcolm/issues) or via email to [malcolm@inl.gov](mailto:malcolm@inl.gov?subject=Malcolm).
        + contributing issues:
            * inotify issue (idaholab#168)
            * htadmin/nginx and htpasswd (idaholab#169)
            * opensearch (idaholab#170)
            * uploading large PCAP files (idaholab#171)
            * script consolidation (idaholab#172)
            * documentation (idaholab#173)
            * user-defined persistent volumes (idaholab#174)
            * opensearch keystore (idaholab#176)
            * expose other TCP services (idaholab#183)
            * provide with filebeat access to nginx access and error logs (idaholab#186)
            * use Secrets for some environment variables instead of ConfigMaps (idaholab#189)

* Enhancements and fixes
    * remove name-map-ui container (idaholab#165) in favor of using NetBox for asset identification
    * Python script refactoring, consolidation and cleanup
    * standardization of Docker container entrypoints
    * create `./scripts/configure` alias for `./scripts/install.py --configure`

* Component version updates
    - Arkime to [v4.3.0](https://github.com/arkime/arkime/blob/fa0db2415bdc109be7a4dd8ee2c2838673980b5f/CHANGELOG#L33-L72)
    - Capa to [v5.1.0](https://github.com/mandiant/capa/releases/tag/v5.1.0)
    - Fluent Bit to [v2.1.2](https://fluentbit.io/announcements/v2.1.2/)
    - NetBox to [v3.5.0](https://github.com/netbox-community/netbox/releases/tag/v3.5.0)
    - NGINX to [v1.22.1](http://nginx.org/en/CHANGES-1.22)
    - Supercronic to [v0.2.24](https://github.com/aptible/supercronic/releases/tag/v0.2.24)
    - Suricata to [v6.0.10](https://suricata.io/2023/01/31/suricata-6-0-10-released/)
    - Yara to [v4.3.0](https://github.com/VirusTotal/yara/releases/tag/v4.3.0)
    - Zeek to [v5.2.1](https://github.com/zeek/zeek/releases/tag/v5.2.1)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mmguero mmguero

Labels
cloud Relating to deployment of Malcolm in the cloud and/or with Kubernetes opensearch Relating to Malcolm's use of OpenSearch
Projects
Malcolm
Status: Released
Milestone
v23.05.0
Development

No branches or pull requests
1 participant
@mmguero