You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Malcolm's is notorious for pushing the boundaries of Arkime in terms of the number of custom fields we create (like 3k+). This occasionally causes issues in Arkime surrounding creating and retrieving those fields. Totally not bagging on Arkime, they're fantastic and always helpful to find the solution, it's just the nature of integrating with another tool.
Recently I discovered that some fields weren't getting created correctly on the initial statup of Arkime. Andy Wick helped point in the direction of the debug I needed to do and I saw this:
malcolm-arkime-1 | Aug 11 14:46:40 config.c:115 moloch_config_section_str(): custom-fields.zeek.rdp.security_protocol=db:zeek.rdp.security_protocol;group:zeek_rdp;kind:termfield;friendly:Security Protocol;help:Security Protocol
malcolm-arkime-1 | Aug 11 14:46:40 field.c:131 moloch_field_define_text_full(): Parsing db:zeek.rdp.security_protocol;group:zeek_rdp;kind:termfield;friendly:Security Protocol;help:Security Protocol
malcolm-arkime-1 | Aug 11 14:46:40 config.c:115 moloch_config_section_str(): custom-fields.zeek.rdp.client_channels=db:zeek.rdp.client_channels;group:zeek_rdp;kind:termfield;friendly:Channel;help:Channel
malcolm-arkime-1 | Aug 11 14:46:40 field.c:131 moloch_field_define_text_full(): Parsing db:zeek.rdp.client_channels;group:zeek_rdp;kind:termfield;friendly:Channel;help:Channel
malcolm-arkime-1 | Aug 11 14:46:40 http.c:763 moloch_http_schedule(): ERROR - Dropping request (https://arkime.com/faq#error-dropping-request) /arkime_fields/_doc/zeek.rdp.client_channels of size 128 queue 1001 is WAY too big
The (temporary, at least) fix is to set the requests for field creation in Arkime to be higher priority (see mmguero-dev/Malcolm@3ad2f9c). Andy says he's going to look at better solutions in Arkime in the future.
The text was updated successfully, but these errors were encountered:
* New Features
* Support remote OpenSearch instance/cluster as alternative to local containerized instance (#110)
* Documentation and convenience scripts for configuring ingestion of third-party logs, and basic parsing/normalizing of Fluent-Bit's Windows event logs
* S7comm Plus support and replaced Amazon S7comm parser with icsnpp-s7comm (#99)
* Version Bumps
* OpenSearch and OpenSearch Dashboards to v2.2.1
* Zeek to v5.0.1
* Spicy to v1.5.1
* spicy-plugin to v1.3.17
* YARA to v4.2.3
* Capa to v4.0.1
* Improvements
* Major improvements to OPC UA Binary parser and supporting dashboards
* Ensure that all containers are provided the same information about trusted CA certificates
* changed list of "sensitive countries" to match U.S. Department of Energy Sensitive Country List
* Increased maximum fields from 3,000 to 5,000
* Standardized configuration and authentication for primary and secondary remote OpenSearch instances, and make sure that index templates are created on secondary remote OpenSearch instances
* Expand and fix normalization of [network.direction](https://www.elastic.co/guide/en/ecs/current/ecs-network.html#field-network-direction) in lieu of using tags
* Various tweaks and improvements to the `install.py` script for enabling/disabling some features
* Bugs Fixed
* fields could be missing in Arkime due to a large number of concurrent requests (#115)
* mapper_parsing_exception, TCP flag parsing problem (cisagov#214)
mmguero
added a commit
to cisagov/Malcolm
that referenced
this issue
Sep 7, 2022
* New Features
* Support remote OpenSearch instance/cluster as alternative to local containerized instance (idaholab#110)
* Documentation and convenience scripts for configuring ingestion of third-party logs, and basic parsing/normalizing of Fluent-Bit's Windows event logs
* S7comm Plus support and replaced Amazon S7comm parser with icsnpp-s7comm (idaholab#99)
* Version Bumps
* OpenSearch and OpenSearch Dashboards to v2.2.1
* Zeek to v5.0.1
* Spicy to v1.5.1
* spicy-plugin to v1.3.17
* YARA to v4.2.3
* Capa to v4.0.1
* Improvements
* Major improvements to OPC UA Binary parser and supporting dashboards
* Ensure that all containers are provided the same information about trusted CA certificates
* changed list of "sensitive countries" to match U.S. Department of Energy Sensitive Country List
* Increased maximum fields from 3,000 to 5,000
* Standardized configuration and authentication for primary and secondary remote OpenSearch instances, and make sure that index templates are created on secondary remote OpenSearch instances
* Expand and fix normalization of [network.direction](https://www.elastic.co/guide/en/ecs/current/ecs-network.html#field-network-direction) in lieu of using tags
* Various tweaks and improvements to the `install.py` script for enabling/disabling some features
* Bugs Fixed
* fields could be missing in Arkime due to a large number of concurrent requests (idaholab#115)
* mapper_parsing_exception, TCP flag parsing problem (#214)
Malcolm's is notorious for pushing the boundaries of Arkime in terms of the number of custom fields we create (like 3k+). This occasionally causes issues in Arkime surrounding creating and retrieving those fields. Totally not bagging on Arkime, they're fantastic and always helpful to find the solution, it's just the nature of integrating with another tool.
Recently I discovered that some fields weren't getting created correctly on the initial statup of Arkime. Andy Wick helped point in the direction of the debug I needed to do and I saw this:
The (temporary, at least) fix is to set the requests for field creation in Arkime to be higher priority (see mmguero-dev/Malcolm@3ad2f9c). Andy says he's going to look at better solutions in Arkime in the future.
The text was updated successfully, but these errors were encountered: