migrate to open distro for elasticsearch #15
Labels
docker
Relating to docker and docker-compose as used by Malcolm
enhancement
New feature or request
opensearch
Relating to Malcolm's use of OpenSearch
Projects
Comments
mmguero
added
docker
|
This could potentially cover what ElastAlert does as well. OpenDistro comes with free Alerting plugin for Kibana! |
mmguero
changed the title
|
Tracking updating elastic components from 7.6.2 to 7.9.x that open distro tracks. The unstable/development branch these commits are being done on before integration into the main branch can be found at https://github.com/mmguero-dev/Malcolm/tree/topic/elastic_new
|
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Feb 5, 2021
mmguero
added a commit
to cisagov/Malcolm
that referenced
this issue
Feb 16, 2021
mmguero
added a commit
to cisagov/Malcolm
that referenced
this issue
Feb 16, 2021
* bring idaholab/Malcolm up-to-date with cisagov/Malcolm * Restore malcolm.fyi web page generation * ISO sha256sum placeholders * working on idaholab#15, migrating to open distro for elastic * bump yara to 4.0.5 * update SHA256 sums * don't show comments from config file in settings appl7 * use mmguero-dev/zeek-community-id so it will build with docker 3.0.12 * somehow the extracted file http server stuff didn't make it into this branch * expose HTTP file server port * added filemon virtual_host * added depends for nginx upstream * set hostname to file-monitor for filemon * Allow overriding cloned version of a plugin with a specific branch, and for corelight/bro-xor-exe-plugin use release 1.2 (pre commit corelight/zeek-xor-exe-plugin@a86f595) for Zeek 3.0.x and the latest release for Zeek 4.x * use corelight/zeek-community-id build 3.0.0 for zeek 3.0.x * put URLs in quotes for zeek clone * temporarily remove the 'Top Actions and Results by Service' visualization (of type kbn_sankey) as it is not compatible with the current version of Kibana being used. tracking externally at uniberg/kbn_sankey_vis#15, at which point we can add it back in * updates for open distro-based images for using non-1000 UIDs * updates for open distro-based images for using non-1000 UIDs * tweaks to iso for 3.0.0 * remove unnecessary package * updated dashboards for new Kibana for Malcolm 3.0.0 * fix wording of prompt for email account inforrmation for alerts * fix some dashboards where i lost a comma in the JSON * updates to Arkime startup to try to fix race condition with fields getting created * update auditbeat.yml rules to ignore firefox caching stuff (reduce noise) * fix spacing in WISE field definition * fixed a few dashboards (tag cloud visualizations didn't come over correctly) * continuation of the idea for commit 045d7e6, for Logstash the STDIN doesn't work if we are dropping privs, which we don't need to do anyway. So differentiate between elastic and logstash for storing keystore values when dropping privileges * minor tweaks to cloud visualizations * updated local.zeek for sensor image to include ICSNPP parsers * Revert "updated local.zeek for sensor image to include ICSNPP parsers" This reverts commit dc19d83. * updated local.zeek for sensor image to include ICSNPP parsers * Updates to documentation and comments in preparation for 3.0.0 release
mmguero
added a commit
to cisagov/Malcolm
that referenced
this issue
Feb 16, 2021
- Change base for Elasticsearch and Kibana Docker images (version 7.6.2) from Elastic.co to Open Distro for Elastic (based on Elastic 7.10.0); see idaholab#15. This is a major change which **breaks backwards compatibility** for several features (listed below). If you are using these features, you will need to back up the data and/or configuration associated with them and migrate them manually to the new tools. No automatic migration or upgrade of these features is performed. It's recommended that you re-run `install.py --configure` (see [System configuration and tuning](https://github.com/cisagov/Malcolm#ConfigAndTuning)) prior to running Malcolm v3.0.0. - Kibana [comments](https://github.com/gwintzer/kibana-comments-app-plugin) replaced with [Notebooks](https://opendistro.github.io/for-elasticsearch-docs/docs/notebooks/) - Kibana [elastalert](https://github.com/nsano-rururu/elastalert-kibana-plugin) plugin replaced with [Alerting](https://opendistro.github.io/for-elasticsearch-docs/docs/alerting/) plugin - Elasticsearch [curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html) replaced with [Index Management](https://opendistro.github.io/for-elasticsearch-docs/docs/ism/) plugin - The third-party [Sankey visualization plugin](https://github.com/mmguero-dev/kbn_sankey_vis] has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see uniberg/kbn_sankey_vis#15) - The third-party [Kibana drill-down plugin](https://github.com/mmguero-dev/kibana-plugin-drilldownmenu/) providing Kibana-to-Moloch pivoting has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see goodlabs-studio/kibana-plugin-drilldownmenu#5) - In addition to those replacements, the Real Time Anomaly Detection feature is now available: - [Real Time Anomaly Detection in Open Distro for Elasticsearch](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/real-time-anomaly-detection-in-open-distro-for-elasticsearch/) blog announcement - [Anomaly Detection](https://opendistro.github.io/for-elasticsearch-docs/docs/ad/) documentation and source code for [Elasticsearch](https://github.com/opendistro-for-elasticsearch/anomaly-detection) and [Kibana](https://github.com/opendistro-for-elasticsearch/anomaly-detection-kibana-plugin) components - [Random Cut Forests writeup](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/random-cut-forests/) - Malcolm startup time (especially the Logstash container) has been reduced drastically - Improvements to Malcolm's prebuilt Kibana dashboards - Improvements to build scripts - Minor tweaks and bugfixes for ISO-installed environments for Malcolm and Hedgehog Linux - Minor other bug fixes and performance improvements - Version bump - Yara [v4.0.5](https://github.com/VirusTotal/yara/releases/tag/v4.0.5)
mmguero
added a commit
that referenced
this issue
Feb 16, 2021
- Change base for Elasticsearch and Kibana Docker images (version 7.6.2) from Elastic.co to Open Distro for Elastic (based on Elastic 7.10.0); see #15. This is a major change which **breaks backwards compatibility** for several features (listed below). If you are using these features, you will need to back up the data and/or configuration associated with them and migrate them manually to the new tools. No automatic migration or upgrade of these features is performed. It's recommended that you re-run `install.py --configure` (see [System configuration and tuning](https://github.com/cisagov/Malcolm#ConfigAndTuning)) prior to running Malcolm v3.0.0. - Kibana [comments](https://github.com/gwintzer/kibana-comments-app-plugin) replaced with [Notebooks](https://opendistro.github.io/for-elasticsearch-docs/docs/notebooks/) - Kibana [elastalert](https://github.com/nsano-rururu/elastalert-kibana-plugin) plugin replaced with [Alerting](https://opendistro.github.io/for-elasticsearch-docs/docs/alerting/) plugin - Elasticsearch [curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html) replaced with [Index Management](https://opendistro.github.io/for-elasticsearch-docs/docs/ism/) plugin - The third-party [Sankey visualization plugin](https://github.com/mmguero-dev/kbn_sankey_vis] has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see uniberg/kbn_sankey_vis#15) - The third-party [Kibana drill-down plugin](https://github.com/mmguero-dev/kibana-plugin-drilldownmenu/) providing Kibana-to-Moloch pivoting has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see goodlabs-studio/kibana-plugin-drilldownmenu#5) - In addition to those replacements, the Real Time Anomaly Detection feature is now available: - [Real Time Anomaly Detection in Open Distro for Elasticsearch](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/real-time-anomaly-detection-in-open-distro-for-elasticsearch/) blog announcement - [Anomaly Detection](https://opendistro.github.io/for-elasticsearch-docs/docs/ad/) documentation and source code for [Elasticsearch](https://github.com/opendistro-for-elasticsearch/anomaly-detection) and [Kibana](https://github.com/opendistro-for-elasticsearch/anomaly-detection-kibana-plugin) components - [Random Cut Forests writeup](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/random-cut-forests/) - Malcolm startup time (especially the Logstash container) has been reduced drastically - Improvements to Malcolm's prebuilt Kibana dashboards - Improvements to build scripts - Minor tweaks and bugfixes for ISO-installed environments for Malcolm and Hedgehog Linux - Minor other bug fixes and performance improvements - Version bump - Yara [v4.0.5](https://github.com/VirusTotal/yara/releases/tag/v4.0.5)
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Feb 16, 2021
- Change base for Elasticsearch and Kibana Docker images (version 7.6.2) from Elastic.co to Open Distro for Elastic (based on Elastic 7.10.0); see idaholab#15. This is a major change which **breaks backwards compatibility** for several features (listed below). If you are using these features, you will need to back up the data and/or configuration associated with them and migrate them manually to the new tools. No automatic migration or upgrade of these features is performed. It's recommended that you re-run `install.py --configure` (see [System configuration and tuning](https://github.com/cisagov/Malcolm#ConfigAndTuning)) prior to running Malcolm v3.0.0. - Kibana [comments](https://github.com/gwintzer/kibana-comments-app-plugin) replaced with [Notebooks](https://opendistro.github.io/for-elasticsearch-docs/docs/notebooks/) - Kibana [elastalert](https://github.com/nsano-rururu/elastalert-kibana-plugin) plugin replaced with [Alerting](https://opendistro.github.io/for-elasticsearch-docs/docs/alerting/) plugin - Elasticsearch [curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html) replaced with [Index Management](https://opendistro.github.io/for-elasticsearch-docs/docs/ism/) plugin - The third-party [Sankey visualization plugin](https://github.com/mmguero-dev/kbn_sankey_vis] has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see uniberg/kbn_sankey_vis#15) - The third-party [Kibana drill-down plugin](https://github.com/mmguero-dev/kibana-plugin-drilldownmenu/) providing Kibana-to-Moloch pivoting has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see goodlabs-studio/kibana-plugin-drilldownmenu#5) - In addition to those replacements, the Real Time Anomaly Detection feature is now available: - [Real Time Anomaly Detection in Open Distro for Elasticsearch](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/real-time-anomaly-detection-in-open-distro-for-elasticsearch/) blog announcement - [Anomaly Detection](https://opendistro.github.io/for-elasticsearch-docs/docs/ad/) documentation and source code for [Elasticsearch](https://github.com/opendistro-for-elasticsearch/anomaly-detection) and [Kibana](https://github.com/opendistro-for-elasticsearch/anomaly-detection-kibana-plugin) components - [Random Cut Forests writeup](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/random-cut-forests/) - Malcolm startup time (especially the Logstash container) has been reduced drastically - Improvements to Malcolm's prebuilt Kibana dashboards - Improvements to build scripts - Minor tweaks and bugfixes for ISO-installed environments for Malcolm and Hedgehog Linux - Minor other bug fixes and performance improvements - Version bump - Yara [v4.0.5](https://github.com/VirusTotal/yara/releases/tag/v4.0.5)
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Feb 16, 2021
- Change base for Elasticsearch and Kibana Docker images (version 7.6.2) from Elastic.co to Open Distro for Elastic (based on Elastic 7.10.0); see idaholab#15. This is a major change which **breaks backwards compatibility** for several features (listed below). If you are using these features, you will need to back up the data and/or configuration associated with them and migrate them manually to the new tools. No automatic migration or upgrade of these features is performed. It's recommended that you re-run `install.py --configure` (see [System configuration and tuning](https://github.com/cisagov/Malcolm#ConfigAndTuning)) prior to running Malcolm v3.0.0. - Kibana [comments](https://github.com/gwintzer/kibana-comments-app-plugin) replaced with [Notebooks](https://opendistro.github.io/for-elasticsearch-docs/docs/notebooks/) - Kibana [elastalert](https://github.com/nsano-rururu/elastalert-kibana-plugin) plugin replaced with [Alerting](https://opendistro.github.io/for-elasticsearch-docs/docs/alerting/) plugin - Elasticsearch [curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html) replaced with [Index Management](https://opendistro.github.io/for-elasticsearch-docs/docs/ism/) plugin - The third-party [Sankey visualization plugin](https://github.com/mmguero-dev/kbn_sankey_vis] has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see uniberg/kbn_sankey_vis#15) - The third-party [Kibana drill-down plugin](https://github.com/mmguero-dev/kibana-plugin-drilldownmenu/) providing Kibana-to-Moloch pivoting has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see goodlabs-studio/kibana-plugin-drilldownmenu#5) - In addition to those replacements, the Real Time Anomaly Detection feature is now available: - [Real Time Anomaly Detection in Open Distro for Elasticsearch](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/real-time-anomaly-detection-in-open-distro-for-elasticsearch/) blog announcement - [Anomaly Detection](https://opendistro.github.io/for-elasticsearch-docs/docs/ad/) documentation and source code for [Elasticsearch](https://github.com/opendistro-for-elasticsearch/anomaly-detection) and [Kibana](https://github.com/opendistro-for-elasticsearch/anomaly-detection-kibana-plugin) components - [Random Cut Forests writeup](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/random-cut-forests/) - Malcolm startup time (especially the Logstash container) has been reduced drastically - Improvements to Malcolm's prebuilt Kibana dashboards - Improvements to build scripts - Minor tweaks and bugfixes for ISO-installed environments for Malcolm and Hedgehog Linux - Minor other bug fixes and performance improvements - Version bump - Yara [v4.0.5](https://github.com/VirusTotal/yara/releases/tag/v4.0.5)
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Feb 16, 2021
- Change base for Elasticsearch and Kibana Docker images (version 7.6.2) from Elastic.co to Open Distro for Elastic (based on Elastic 7.10.0); see idaholab#15. This is a major change which **breaks backwards compatibility** for several features (listed below). If you are using these features, you will need to back up the data and/or configuration associated with them and migrate them manually to the new tools. No automatic migration or upgrade of these features is performed. It's recommended that you re-run `install.py --configure` (see [System configuration and tuning](https://github.com/cisagov/Malcolm#ConfigAndTuning)) prior to running Malcolm v3.0.0. - Kibana [comments](https://github.com/gwintzer/kibana-comments-app-plugin) replaced with [Notebooks](https://opendistro.github.io/for-elasticsearch-docs/docs/notebooks/) - Kibana [elastalert](https://github.com/nsano-rururu/elastalert-kibana-plugin) plugin replaced with [Alerting](https://opendistro.github.io/for-elasticsearch-docs/docs/alerting/) plugin - Elasticsearch [curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html) replaced with [Index Management](https://opendistro.github.io/for-elasticsearch-docs/docs/ism/) plugin - The third-party [Sankey visualization plugin](https://github.com/mmguero-dev/kbn_sankey_vis] has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see uniberg/kbn_sankey_vis#15) - The third-party [Kibana drill-down plugin](https://github.com/mmguero-dev/kibana-plugin-drilldownmenu/) providing Kibana-to-Moloch pivoting has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see goodlabs-studio/kibana-plugin-drilldownmenu#5) - In addition to those replacements, the Real Time Anomaly Detection feature is now available: - [Real Time Anomaly Detection in Open Distro for Elasticsearch](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/real-time-anomaly-detection-in-open-distro-for-elasticsearch/) blog announcement - [Anomaly Detection](https://opendistro.github.io/for-elasticsearch-docs/docs/ad/) documentation and source code for [Elasticsearch](https://github.com/opendistro-for-elasticsearch/anomaly-detection) and [Kibana](https://github.com/opendistro-for-elasticsearch/anomaly-detection-kibana-plugin) components - [Random Cut Forests writeup](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/random-cut-forests/) - Malcolm startup time (especially the Logstash container) has been reduced drastically - Improvements to Malcolm's prebuilt Kibana dashboards - Improvements to build scripts - Minor tweaks and bugfixes for ISO-installed environments for Malcolm and Hedgehog Linux - Minor other bug fixes and performance improvements - Version bump - Yara [v4.0.5](https://github.com/VirusTotal/yara/releases/tag/v4.0.5)
mmguero
added a commit
that referenced
this issue
Mar 3, 2021
- Change base for Elasticsearch and Kibana Docker images (version 7.6.2) from Elastic.co to Open Distro for Elastic (based on Elastic 7.10.0); see #15. This is a major change which **breaks backwards compatibility** for several features (listed below). If you are using these features, you will need to back up the data and/or configuration associated with them and migrate them manually to the new tools. No automatic migration or upgrade of these features is performed. It's recommended that you re-run `install.py --configure` (see [System configuration and tuning](https://github.com/cisagov/Malcolm#ConfigAndTuning)) prior to running Malcolm v3.0.0. - Kibana [comments](https://github.com/gwintzer/kibana-comments-app-plugin) replaced with [Notebooks](https://opendistro.github.io/for-elasticsearch-docs/docs/notebooks/) - Kibana [elastalert](https://github.com/nsano-rururu/elastalert-kibana-plugin) plugin replaced with [Alerting](https://opendistro.github.io/for-elasticsearch-docs/docs/alerting/) plugin - Elasticsearch [curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html) replaced with [Index Management](https://opendistro.github.io/for-elasticsearch-docs/docs/ism/) plugin - The third-party [Sankey visualization plugin](https://github.com/mmguero-dev/kbn_sankey_vis] has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see uniberg/kbn_sankey_vis#15) - The third-party [Kibana drill-down plugin](https://github.com/mmguero-dev/kibana-plugin-drilldownmenu/) providing Kibana-to-Moloch pivoting has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see goodlabs-studio/kibana-plugin-drilldownmenu#5) - In addition to those replacements, the Real Time Anomaly Detection feature is now available: - [Real Time Anomaly Detection in Open Distro for Elasticsearch](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/real-time-anomaly-detection-in-open-distro-for-elasticsearch/) blog announcement - [Anomaly Detection](https://opendistro.github.io/for-elasticsearch-docs/docs/ad/) documentation and source code for [Elasticsearch](https://github.com/opendistro-for-elasticsearch/anomaly-detection) and [Kibana](https://github.com/opendistro-for-elasticsearch/anomaly-detection-kibana-plugin) components - [Random Cut Forests writeup](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/random-cut-forests/) - Malcolm startup time (especially the Logstash container) has been reduced drastically - Improvements to Malcolm's prebuilt Kibana dashboards - Improvements to build scripts - Minor tweaks and bugfixes for ISO-installed environments for Malcolm and Hedgehog Linux - Minor other bug fixes and performance improvements - Version bump - Yara [v4.0.5](https://github.com/VirusTotal/yara/releases/tag/v4.0.5)
mmguero
added a commit
that referenced
this issue
Mar 3, 2021
- Change base for Elasticsearch and Kibana Docker images (version 7.6.2) from Elastic.co to Open Distro for Elastic (based on Elastic 7.10.0); see #15. This is a major change which **breaks backwards compatibility** for several features (listed below). If you are using these features, you will need to back up the data and/or configuration associated with them and migrate them manually to the new tools. No automatic migration or upgrade of these features is performed. It's recommended that you re-run `install.py --configure` (see [System configuration and tuning](https://github.com/cisagov/Malcolm#ConfigAndTuning)) prior to running Malcolm v3.0.0. - Kibana [comments](https://github.com/gwintzer/kibana-comments-app-plugin) replaced with [Notebooks](https://opendistro.github.io/for-elasticsearch-docs/docs/notebooks/) - Kibana [elastalert](https://github.com/nsano-rururu/elastalert-kibana-plugin) plugin replaced with [Alerting](https://opendistro.github.io/for-elasticsearch-docs/docs/alerting/) plugin - Elasticsearch [curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html) replaced with [Index Management](https://opendistro.github.io/for-elasticsearch-docs/docs/ism/) plugin - The third-party [Sankey visualization plugin](https://github.com/mmguero-dev/kbn_sankey_vis] has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see uniberg/kbn_sankey_vis#15) - The third-party [Kibana drill-down plugin](https://github.com/mmguero-dev/kibana-plugin-drilldownmenu/) providing Kibana-to-Moloch pivoting has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see goodlabs-studio/kibana-plugin-drilldownmenu#5) - In addition to those replacements, the Real Time Anomaly Detection feature is now available: - [Real Time Anomaly Detection in Open Distro for Elasticsearch](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/real-time-anomaly-detection-in-open-distro-for-elasticsearch/) blog announcement - [Anomaly Detection](https://opendistro.github.io/for-elasticsearch-docs/docs/ad/) documentation and source code for [Elasticsearch](https://github.com/opendistro-for-elasticsearch/anomaly-detection) and [Kibana](https://github.com/opendistro-for-elasticsearch/anomaly-detection-kibana-plugin) components - [Random Cut Forests writeup](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/random-cut-forests/) - Malcolm startup time (especially the Logstash container) has been reduced drastically - Improvements to Malcolm's prebuilt Kibana dashboards - Improvements to build scripts - Minor tweaks and bugfixes for ISO-installed environments for Malcolm and Hedgehog Linux - Minor other bug fixes and performance improvements - Version bump - Yara [v4.0.5](https://github.com/VirusTotal/yara/releases/tag/v4.0.5)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
From Malcolm created by mmguero: cisagov#70
https://opendistro.github.io/for-elasticsearch/
This looks like it might be a good choice for the default ES images used for Malcolm
The text was updated successfully, but these errors were encountered: