Update Kafka client to 2.7.0 #10146
Merged
Update Kafka client to 2.7.0 #10146
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Workaround a bug in the jackson BOM.
It doesn't contain a managed dependency for
jackson-module-scala_2.13
This makes it being resolved through kafka to a newer
version, which prevents Jersey to start.
- Fix use of wrong props variable - Use a shortened node-id because the full node-id plus input-id is just too long to be readable
Those are not necessary anymore Fixes #9927
This can be removed, if FasterXML/jackson-bom#38 gets merged, or we move to a newer jackson release.
This was referenced Feb 24, 2021
|
Thanks so much for this!! Happy to smoke test this in our development environment -- will get back to you within the week. |
|
i'm unable to package the jar locally could you package this and upload it as a github release? |
|
@jpdstan that artifact should be on maven central: I'm about to push an update to that PR. So bear with me. |
This includes missing dependencies Refs https://github.com/Graylog2/graylog-shaded/pull/4
|
hey @mpfz0r i'm still having trouble building the jar locally, getting some yarn errors now, perhaps i'm using the wrong version (1.22.4). would you mind just packaging the release for me? apologies - I have not setup this Graylog environment locally before and I can't find any instructions on it |
|
Would you mind also test this scenario to see if the consumer rejoins the after broker restart with the upgraded kafka client? Essentially leave your graylog kafka input running, brings down you kakfa brokers. Restart your kafka brokers, put some messages into the topic, and see if the graylog input can consume those messages. From my current testing of Graylog kafka input, I need to restart my Graylog input in order to pick up message again. I am using Graylog 4.0.2. Thanks. |
|
@yangzaizi That seems to work with the new client. (But I've never had this issue with the old one either, so...) |
|
Been running this all weekend with the following parameters:
....and so far so good! Every time i've checked the consumer groups have been in a Stable state, no more "permanently stuck rebalancing" anymore. Will let you know if anything changes. Thanks so much for your work on this @mpfz0r !!! |
|
Thanks for testing @jpdstan Much appreciated! |
mpfz0r
added a commit
to Graylog2/graylog-docker
that referenced
this pull request
Mar 1, 2021
mpfz0r
added a commit
to Graylog2/fpm-recipes
that referenced
this pull request
Mar 1, 2021
mpfz0r
added a commit
to Graylog2/fpm-recipes
that referenced
this pull request
Mar 1, 2021
|
I deployed this to my setup and didn't see any problems. 👍 |
The log level was meant for the new consumer library. We don't want to log the entire consumer config into our server log. It may contain sensitive data, like passwords.
mpfz0r
added a commit
to Graylog2/graylog-docker
that referenced
this pull request
Mar 2, 2021
mpfz0r
added a commit
to Graylog2/fpm-recipes
that referenced
this pull request
Mar 2, 2021
mpfz0r
added a commit
that referenced
this pull request
Mar 2, 2021
* Use shaded kafka 0.9.0.1 version for disk journal
* Update scala to 2.13.4
Workaround a bug in the jackson BOM.
It doesn't contain a managed dependency for
jackson-module-scala_2.13
This makes it being resolved through kafka to a newer
version, which prevents Jersey to start.
* Switch to current kafka client
* Update log4j2 filter for shaded kafka
* Bump kafka to version -5 which should now be on mavencentral
* Keep Kafka legacy mode using the shaded client version
* Use a better readable client.id
- Fix use of wrong props variable
- Use a shortened node-id because the full node-id plus input-id is
just too long to be readable
* Remove workarounds for bugs in the old kafka client
Those are not necessary anymore
Fixes #9927
* Mention workaround for incomplete jackson BOM
This can be removed, if
FasterXML/jackson-bom#38
gets merged, or we move to a newer jackson release.
* Bump shaded kafka version -6
This includes missing dependencies
Refs https://github.com/Graylog2/graylog-shaded/pull/4
* Keep Kafka ConsumerConfig log level package unshaded
The log level was meant for the new consumer library.
We don't want to log the entire consumer config into our server log.
It may contain sensitive data, like passwords.
(cherry picked from commit 2cd3e2b)
thll
pushed a commit
that referenced
this pull request
Mar 8, 2021
* Use shaded kafka 0.9.0.1 version for disk journal
* Update scala to 2.13.4
Workaround a bug in the jackson BOM.
It doesn't contain a managed dependency for
jackson-module-scala_2.13
This makes it being resolved through kafka to a newer
version, which prevents Jersey to start.
* Switch to current kafka client
* Update log4j2 filter for shaded kafka
* Bump kafka to version -5 which should now be on mavencentral
* Keep Kafka legacy mode using the shaded client version
* Use a better readable client.id
- Fix use of wrong props variable
- Use a shortened node-id because the full node-id plus input-id is
just too long to be readable
* Remove workarounds for bugs in the old kafka client
Those are not necessary anymore
Fixes #9927
* Mention workaround for incomplete jackson BOM
This can be removed, if
FasterXML/jackson-bom#38
gets merged, or we move to a newer jackson release.
* Bump shaded kafka version -6
This includes missing dependencies
Refs https://github.com/Graylog2/graylog-shaded/pull/4
* Keep Kafka ConsumerConfig log level package unshaded
The log level was meant for the new consumer library.
We don't want to log the entire consumer config into our server log.
It may contain sensitive data, like passwords.
(cherry picked from commit 2cd3e2b)
malcyon
pushed a commit
to Graylog2/fpm-recipes
that referenced
this pull request
Mar 9, 2021
malcyon
pushed a commit
to Graylog2/graylog-docker
that referenced
this pull request
Mar 9, 2021
danotorrey
pushed a commit
that referenced
this pull request
Apr 13, 2021
* Fix order of tabs in dashboard after install (#9812) ## Motivation Prior to this change, a dashboard installed by a content pack had a tabs in a random order. Since we cannot change the order of the tabs this is quite inconvinient for the user. ## Description This change will do mutiple steps to fix that: - Use a LinkedHashMap to ensure the order stays fix during content pack creation. - Instead of using the serach QueryIds we use the views QueryIds which we previously fixed in the order. - When loading a view and selecting the activeQuery we also use the view.state queryids instead of the search ones. Fixes #8264 * Show timezone setting to the user (#9814) * Display the timezone settings to the user ## Motivation Prior to this change, the user was not able to change his own timezone settings since they were hidden to him. ## Description This change will show the settings to the user so he can edit them. * Fix settings section test * Bump urijs to 1.19.5 (#9874) This provides a fix for the following security issue: GHSA-3329-pjwv-fjpg Backport of #9857 for 4.0 * Fix pattern used to identify parameters in search/dashboard query strings. (#9877) (#9887) * Consolidating backend-agnostic query parsing/metadata creation. * Fixing test. * Adding test cases for bug. * Adjusting regex for query string parser. * Removing unused parameter. Co-authored-by: Dennis Oelkers <dennis@graylog.com> * Fix deserialization of incomplete AlertReceivers objects (#9785) (#9910) The new paginated endpoint in StreamResource is using mongojack to deserialize the database objects. In some setups the AlertReceivers object in the database might not be complete. Change AlertReceivers to use defaults for missing values. Fixes #9637 (cherry picked from commit 1e456af) * Fix page size change selector in Content Pack list (#9875) (#9924) The `PageSizeSelect` component passes an event as argument to its `onChange` function but but we were expecting a string containing a number. This change fixes that by extracting the value from the received event. Fixes #9629 Co-authored-by: Edmundo Alvarez <edmundo@graylog.com> * Do not allow selecting same pipeline stage twice (#9916) * Avoid using a stage number that is already used Check if selected stage number already exists in pipeline, displaying an error in that case and not allowing to submit the form. This will prevent users of accidentally overriding one of their existing stages. Additionally improve the initial suggestion for new stages. Before we always displayed `0` when creating a stage, now we base that value in the existing stages for the pipeline. Fixes #6806 * Do not submit form when using an existing stage * Retry read timeouts for Elasticsearch (#9922) * Retry read timeouts for Elasticsearch. Before this change, the retry strategy used for ES requests did not retry for instances of `SocketTimeoutException`, raised in case of read timeouts. This resulted in exceptions being raised when individual nodes were not responding and node discovery was not used. This was e.g. being noticable when performing searches and 1 of 3 nodes was failing, so every 3rd search request was timing out and returning an error instead of being retried and successful and just taking longer. This change is adding this class to the exceptions being retried, so any request that goes out to a node which is not responding on time is retried. * Removing duplicate license header. * Fix date format specifiers in index mapping templates (#9948) * Fix date format specifiers in index mapping templates for ES7. Before this change, the pre-ES7 date format specifiers were reused for ES7 to define the format of date fields in index mapping templates. Unfortunately, this results in warnings being returned in responses to searches operating on date fields, due to a [breaking change introduced in ES7](https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html#_joda_based_date_formatters_are_replaced_with_java_ones). This change is now splitting up the date format specifiers and uses joda syntax for ES6 and `java.time` syntax for ES7. In addition, the overly lax exposition of the date format specifiers from the `Tools` class was locked down to prevent them leaking into new places. Fixes #9690. * Adjusting tests/expected results. * Adjusting single date format in GIM Mapping for v7 too. * Prefixing date format for ES6 to disable deprecation notice. As we are using a deprecated formatter according to the [ES breaking changes](https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html#_joda_based_date_formatters_are_replaced_with_java_ones), this commit is adding the `8` prefix to prevent a warning. * Prefixing legacy date formats in GIM mapping. * Adjusting tests to use `8yyyy`. * Fixing stream rule form state update on input change. (#9968) (#9975) Before this change we were mutating this.state on input changes directly which resulted in a strange behaviour. One result was that the stream rule form did not reset after creating a stream rule, even though the component is unmounting correctly. Have a look at #9956 for more information. * Fix SourceCode editor combined with pipeline rules (4.0) (#9976) Co-authored-by: Gaya <gaya@theclevernode.com> * Retrying bulk indexing if target alias has invalid target(s) (#9978) * Adding test to check if indexing retries for invalid alias targets. * Adding indexing listener to bulk indexing to improve tests. * Suppressing specific warnings. * Introducing `InvalidWriteTargetException`. * Retrying bulk indexing upon `InvalidWriteTargetException`. * Relaxing ES exception regex. * Throw specific exception if parsed exception indices invalid write target. * Fixing error parsing for ESS6, extracting target for exception. * Reverting changes to `ElasticsearchException`. * Adding related test case for ES exception parser. * Fix implicit mutation of retryer builder when listener is present. Fixes #9955 Fixes #9962 * Bump package.json version to 4.0.2 * [graylog2-server] prepare release 4.0.2 * Bump package.json version to 4.0.3-SNAPSHOT * [graylog2-server] prepare for next development iteration * Fix export to dashboard when graylog is behind a proxy (#9991) ## Motivation Prior to this change, when loading the new dashboard containing the previous search we used a not qualified route, which did not take the prefix in account. ## Description This change will use `Routes` to set the new pathname for history.push. * Retry failed deployments to sonatype.org (#9987) * Provide default position in case the widget isn't found in positions map (#10007) * Provide default position in case the widget isn't found in positions map (#9841) * Provide default position in case the widget isn't found in positions map ## Motivation Prior to this change, a widget copied from a search was unable to be copied to a different page since the position was missing in the positions map. ## Description This change will add a safty net in case the position is missing we provide the widgets default size. * Use AddNewWidgetsToPosition for CopyWidgetToDashboard That way we can saftly assume that future copied widgets will not have a missing position The NaN values in the test can be ignored, since the widgets are not existing anymore. * Fix linter warning * Fix license header * Clear positions from stale entries * Provide defaultHeight/Width to test * Update snapshots * Refactor position calculation for new widgets * Fix linter warnings * Create new id when copying a widget * Reuse old sizes if possible * Rename last leftover import * Fix linter * Don't mix responsibilities * Update snapshots * Fix merge errors * Add missing flow annotation * Remove unused files * Fix tests * Remove development grants migration (#10044) (#10046) This shouldn't have been released and can cause a high load on MongoDB. Fixes #10043 (cherry picked from commit 40fbb82) * Bump package.json version to 4.0.3 * [graylog2-server] prepare release 4.0.3 * Bump package.json version to 4.0.4-SNAPSHOT * [graylog2-server] prepare for next development iteration * Fixing counting of messages in index for ES7. (4.0) (#10094) * Creating integration test for `Indices#numberOfMessages`. * Correcting json path, adding safeguard if not present. * Add missing GRN type for outputs (#10089) (#10108) * Add missing GRN type for outputs Fixes #10088 * Add output permissions to the capabilities Without this, the dependency check doesn't run properly because it filters out any permission that isn't handled via grants. That's another bug we should be fixing.. * Add TODO comment * Hide some dependencies that are not relevant for sharing entities * Use Set instead of List for ignored dependcies lookup (cherry picked from commit 8eec5db) * Provide a timeout for request when optimizing index in ES7. (4.0) (#10098) * Use custom timeout when optimizing index for ES7. * Actually return value, use correct import. * Validating config setting to make sure it can be safely casted to int. * Actually use 24 days as maximum. * Bump package.json version to 4.0.4 * [graylog2-server] prepare release 4.0.4 * Bump package.json version to 4.0.5-SNAPSHOT * [graylog2-server] prepare for next development iteration * Prevent NPE in index optimization job. (4.0) (#10119) * Adding test case to check if index optimization throws NPE. * If request config is `null`, create new one instead of copying. * Check it `requestOptions` is `null`. * Bump package.json version to 4.0.5 * [graylog2-server] prepare release 4.0.5 * Bump package.json version to 4.0.6-SNAPSHOT * [graylog2-server] prepare for next development iteration * Fix rendering of trend color in report generation (#10154) * Fix rendering of trend color in report generation ## Motivation The report generation is using the chrome print to pdf functionallity which will turn all backgrounds in css to white. ## Description We add an !important to the css rule (and explain why) and add the magical webkit rule which tells chrome to keep the background as it is. * Fix trend forground color Before the forground color of the rendered pdf was black. Now its WHITE :D * Fix style-lint * Fix type annotation * Fix merge of trend background * Update Kafka client to 2.7.0 (#10146) (#10184) * Use shaded kafka 0.9.0.1 version for disk journal * Update scala to 2.13.4 Workaround a bug in the jackson BOM. It doesn't contain a managed dependency for jackson-module-scala_2.13 This makes it being resolved through kafka to a newer version, which prevents Jersey to start. * Switch to current kafka client * Update log4j2 filter for shaded kafka * Bump kafka to version -5 which should now be on mavencentral * Keep Kafka legacy mode using the shaded client version * Use a better readable client.id - Fix use of wrong props variable - Use a shortened node-id because the full node-id plus input-id is just too long to be readable * Remove workarounds for bugs in the old kafka client Those are not necessary anymore Fixes #9927 * Mention workaround for incomplete jackson BOM This can be removed, if FasterXML/jackson-bom#38 gets merged, or we move to a newer jackson release. * Bump shaded kafka version -6 This includes missing dependencies Refs https://github.com/Graylog2/graylog-shaded/pull/4 * Keep Kafka ConsumerConfig log level package unshaded The log level was meant for the new consumer library. We don't want to log the entire consumer config into our server log. It may contain sensitive data, like passwords. (cherry picked from commit 2cd3e2b) * Fix content-pack auto installation (#10237) This fixes the automatic content pack installation when using the "content_packs_auto_install" config option. The ContentPackService#installContentPack expects a username as third argument, not a user ID. We've been using "local:admin" which is the local admin's pseudo user ID. UserService#load has a special case for the local admin user but is expecting Configuration#getRootUsername to be used and not "local:admin". So the user service tried to find a user with name "local:admin" in the database instead of returning the special admin user object. That fails with: Caused by: java.lang.IllegalStateException: Cannot load user <local:admin> from db The ContentPackLoaderPeriodical is now using Configuration#getRootUsername instead of hard coding "local:admin". Fixes Graylog2/graylog-docker#153 Co-authored-by: Bernd Ahlers <bernd@graylog.com> * Initialize ES7 client and sniffer lazily. (4.0) (#10292) * Initialize ES7 client and sniffer lazily. (#10264) This change is avoiding eager initialization of the ES7 client (and sniffer) in its provider class. Instead, it is initializing a memoized supplier, which returns the same instance upon repeated invocation. (cherry picked from commit aecb2f3) * Removed 4.1 - functionality Co-authored-by: Dennis Oelkers <dennis@graylog.com> * Fix permission check for notifcations in event definitions (#10380) ## Motivation Prior to this change, the permissions check was broken for notification when displaying the event definition summary or edit it. ## Description This change will fix the template string which was missing a colon. ## Notes Special kudos to @mpfz0r (cherry picked from commit b49ce7b) * Bump package.json version to 4.0.6 * [graylog2-server] prepare release 4.0.6 * Bump package.json version to 4.0.7-SNAPSHOT * [graylog2-server] prepare for next development iteration Co-authored-by: Konrad Merz <konrad@graylog.com> Co-authored-by: Dennis Oelkers <dennis@graylog.com> Co-authored-by: Bernd Ahlers <bernd@users.noreply.github.com> Co-authored-by: Linus Pahl <linus@graylog.com> Co-authored-by: Gaya <gaya@theclevernode.com> Co-authored-by: Jenkins <garybot2@torch.sh> Co-authored-by: Donald Morton <donald.morton@graylog.com> Co-authored-by: Marco Pfatschbacher <marco@graylog.com> Co-authored-by: Bernd Ahlers <bernd@graylog.com> Co-authored-by: Jan Heise <jan@janheise.de>
ousmaneo
pushed a commit
that referenced
this pull request
May 10, 2021
* Fix order of tabs in dashboard after install (#9812) ## Motivation Prior to this change, a dashboard installed by a content pack had a tabs in a random order. Since we cannot change the order of the tabs this is quite inconvinient for the user. ## Description This change will do mutiple steps to fix that: - Use a LinkedHashMap to ensure the order stays fix during content pack creation. - Instead of using the serach QueryIds we use the views QueryIds which we previously fixed in the order. - When loading a view and selecting the activeQuery we also use the view.state queryids instead of the search ones. Fixes #8264 * Show timezone setting to the user (#9814) * Display the timezone settings to the user ## Motivation Prior to this change, the user was not able to change his own timezone settings since they were hidden to him. ## Description This change will show the settings to the user so he can edit them. * Fix settings section test * Bump urijs to 1.19.5 (#9874) This provides a fix for the following security issue: GHSA-3329-pjwv-fjpg Backport of #9857 for 4.0 * Fix pattern used to identify parameters in search/dashboard query strings. (#9877) (#9887) * Consolidating backend-agnostic query parsing/metadata creation. * Fixing test. * Adding test cases for bug. * Adjusting regex for query string parser. * Removing unused parameter. Co-authored-by: Dennis Oelkers <dennis@graylog.com> * Fix deserialization of incomplete AlertReceivers objects (#9785) (#9910) The new paginated endpoint in StreamResource is using mongojack to deserialize the database objects. In some setups the AlertReceivers object in the database might not be complete. Change AlertReceivers to use defaults for missing values. Fixes #9637 (cherry picked from commit 1e456af) * Fix page size change selector in Content Pack list (#9875) (#9924) The `PageSizeSelect` component passes an event as argument to its `onChange` function but but we were expecting a string containing a number. This change fixes that by extracting the value from the received event. Fixes #9629 Co-authored-by: Edmundo Alvarez <edmundo@graylog.com> * Do not allow selecting same pipeline stage twice (#9916) * Avoid using a stage number that is already used Check if selected stage number already exists in pipeline, displaying an error in that case and not allowing to submit the form. This will prevent users of accidentally overriding one of their existing stages. Additionally improve the initial suggestion for new stages. Before we always displayed `0` when creating a stage, now we base that value in the existing stages for the pipeline. Fixes #6806 * Do not submit form when using an existing stage * Retry read timeouts for Elasticsearch (#9922) * Retry read timeouts for Elasticsearch. Before this change, the retry strategy used for ES requests did not retry for instances of `SocketTimeoutException`, raised in case of read timeouts. This resulted in exceptions being raised when individual nodes were not responding and node discovery was not used. This was e.g. being noticable when performing searches and 1 of 3 nodes was failing, so every 3rd search request was timing out and returning an error instead of being retried and successful and just taking longer. This change is adding this class to the exceptions being retried, so any request that goes out to a node which is not responding on time is retried. * Removing duplicate license header. * Fix date format specifiers in index mapping templates (#9948) * Fix date format specifiers in index mapping templates for ES7. Before this change, the pre-ES7 date format specifiers were reused for ES7 to define the format of date fields in index mapping templates. Unfortunately, this results in warnings being returned in responses to searches operating on date fields, due to a [breaking change introduced in ES7](https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html#_joda_based_date_formatters_are_replaced_with_java_ones). This change is now splitting up the date format specifiers and uses joda syntax for ES6 and `java.time` syntax for ES7. In addition, the overly lax exposition of the date format specifiers from the `Tools` class was locked down to prevent them leaking into new places. Fixes #9690. * Adjusting tests/expected results. * Adjusting single date format in GIM Mapping for v7 too. * Prefixing date format for ES6 to disable deprecation notice. As we are using a deprecated formatter according to the [ES breaking changes](https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html#_joda_based_date_formatters_are_replaced_with_java_ones), this commit is adding the `8` prefix to prevent a warning. * Prefixing legacy date formats in GIM mapping. * Adjusting tests to use `8yyyy`. * Fixing stream rule form state update on input change. (#9968) (#9975) Before this change we were mutating this.state on input changes directly which resulted in a strange behaviour. One result was that the stream rule form did not reset after creating a stream rule, even though the component is unmounting correctly. Have a look at #9956 for more information. * Fix SourceCode editor combined with pipeline rules (4.0) (#9976) Co-authored-by: Gaya <gaya@theclevernode.com> * Retrying bulk indexing if target alias has invalid target(s) (#9978) * Adding test to check if indexing retries for invalid alias targets. * Adding indexing listener to bulk indexing to improve tests. * Suppressing specific warnings. * Introducing `InvalidWriteTargetException`. * Retrying bulk indexing upon `InvalidWriteTargetException`. * Relaxing ES exception regex. * Throw specific exception if parsed exception indices invalid write target. * Fixing error parsing for ESS6, extracting target for exception. * Reverting changes to `ElasticsearchException`. * Adding related test case for ES exception parser. * Fix implicit mutation of retryer builder when listener is present. Fixes #9955 Fixes #9962 * Bump package.json version to 4.0.2 * [graylog2-server] prepare release 4.0.2 * Bump package.json version to 4.0.3-SNAPSHOT * [graylog2-server] prepare for next development iteration * Fix export to dashboard when graylog is behind a proxy (#9991) ## Motivation Prior to this change, when loading the new dashboard containing the previous search we used a not qualified route, which did not take the prefix in account. ## Description This change will use `Routes` to set the new pathname for history.push. * Retry failed deployments to sonatype.org (#9987) * Provide default position in case the widget isn't found in positions map (#10007) * Provide default position in case the widget isn't found in positions map (#9841) * Provide default position in case the widget isn't found in positions map ## Motivation Prior to this change, a widget copied from a search was unable to be copied to a different page since the position was missing in the positions map. ## Description This change will add a safty net in case the position is missing we provide the widgets default size. * Use AddNewWidgetsToPosition for CopyWidgetToDashboard That way we can saftly assume that future copied widgets will not have a missing position The NaN values in the test can be ignored, since the widgets are not existing anymore. * Fix linter warning * Fix license header * Clear positions from stale entries * Provide defaultHeight/Width to test * Update snapshots * Refactor position calculation for new widgets * Fix linter warnings * Create new id when copying a widget * Reuse old sizes if possible * Rename last leftover import * Fix linter * Don't mix responsibilities * Update snapshots * Fix merge errors * Add missing flow annotation * Remove unused files * Fix tests * Remove development grants migration (#10044) (#10046) This shouldn't have been released and can cause a high load on MongoDB. Fixes #10043 (cherry picked from commit 40fbb82) * Bump package.json version to 4.0.3 * [graylog2-server] prepare release 4.0.3 * Bump package.json version to 4.0.4-SNAPSHOT * [graylog2-server] prepare for next development iteration * Fixing counting of messages in index for ES7. (4.0) (#10094) * Creating integration test for `Indices#numberOfMessages`. * Correcting json path, adding safeguard if not present. * Add missing GRN type for outputs (#10089) (#10108) * Add missing GRN type for outputs Fixes #10088 * Add output permissions to the capabilities Without this, the dependency check doesn't run properly because it filters out any permission that isn't handled via grants. That's another bug we should be fixing.. * Add TODO comment * Hide some dependencies that are not relevant for sharing entities * Use Set instead of List for ignored dependcies lookup (cherry picked from commit 8eec5db) * Provide a timeout for request when optimizing index in ES7. (4.0) (#10098) * Use custom timeout when optimizing index for ES7. * Actually return value, use correct import. * Validating config setting to make sure it can be safely casted to int. * Actually use 24 days as maximum. * Bump package.json version to 4.0.4 * [graylog2-server] prepare release 4.0.4 * Bump package.json version to 4.0.5-SNAPSHOT * [graylog2-server] prepare for next development iteration * Prevent NPE in index optimization job. (4.0) (#10119) * Adding test case to check if index optimization throws NPE. * If request config is `null`, create new one instead of copying. * Check it `requestOptions` is `null`. * Bump package.json version to 4.0.5 * [graylog2-server] prepare release 4.0.5 * Bump package.json version to 4.0.6-SNAPSHOT * [graylog2-server] prepare for next development iteration * Fix rendering of trend color in report generation (#10154) * Fix rendering of trend color in report generation ## Motivation The report generation is using the chrome print to pdf functionallity which will turn all backgrounds in css to white. ## Description We add an !important to the css rule (and explain why) and add the magical webkit rule which tells chrome to keep the background as it is. * Fix trend forground color Before the forground color of the rendered pdf was black. Now its WHITE :D * Fix style-lint * Fix type annotation * Fix merge of trend background * Update Kafka client to 2.7.0 (#10146) (#10184) * Use shaded kafka 0.9.0.1 version for disk journal * Update scala to 2.13.4 Workaround a bug in the jackson BOM. It doesn't contain a managed dependency for jackson-module-scala_2.13 This makes it being resolved through kafka to a newer version, which prevents Jersey to start. * Switch to current kafka client * Update log4j2 filter for shaded kafka * Bump kafka to version -5 which should now be on mavencentral * Keep Kafka legacy mode using the shaded client version * Use a better readable client.id - Fix use of wrong props variable - Use a shortened node-id because the full node-id plus input-id is just too long to be readable * Remove workarounds for bugs in the old kafka client Those are not necessary anymore Fixes #9927 * Mention workaround for incomplete jackson BOM This can be removed, if FasterXML/jackson-bom#38 gets merged, or we move to a newer jackson release. * Bump shaded kafka version -6 This includes missing dependencies Refs Graylog2/graylog-shaded#4 * Keep Kafka ConsumerConfig log level package unshaded The log level was meant for the new consumer library. We don't want to log the entire consumer config into our server log. It may contain sensitive data, like passwords. (cherry picked from commit 2cd3e2b) * Fix content-pack auto installation (#10237) This fixes the automatic content pack installation when using the "content_packs_auto_install" config option. The ContentPackService#installContentPack expects a username as third argument, not a user ID. We've been using "local:admin" which is the local admin's pseudo user ID. UserService#load has a special case for the local admin user but is expecting Configuration#getRootUsername to be used and not "local:admin". So the user service tried to find a user with name "local:admin" in the database instead of returning the special admin user object. That fails with: Caused by: java.lang.IllegalStateException: Cannot load user <local:admin> from db The ContentPackLoaderPeriodical is now using Configuration#getRootUsername instead of hard coding "local:admin". Fixes Graylog2/graylog-docker#153 Co-authored-by: Bernd Ahlers <bernd@graylog.com> * Initialize ES7 client and sniffer lazily. (4.0) (#10292) * Initialize ES7 client and sniffer lazily. (#10264) This change is avoiding eager initialization of the ES7 client (and sniffer) in its provider class. Instead, it is initializing a memoized supplier, which returns the same instance upon repeated invocation. (cherry picked from commit aecb2f3) * Removed 4.1 - functionality Co-authored-by: Dennis Oelkers <dennis@graylog.com> * Fix permission check for notifcations in event definitions (#10380) ## Motivation Prior to this change, the permissions check was broken for notification when displaying the event definition summary or edit it. ## Description This change will fix the template string which was missing a colon. ## Notes Special kudos to @mpfz0r (cherry picked from commit b49ce7b) * Bump package.json version to 4.0.6 * [graylog2-server] prepare release 4.0.6 * Bump package.json version to 4.0.7-SNAPSHOT * [graylog2-server] prepare for next development iteration * Log failed messages and full stack trace on processing errors (#10421) (#10433) Previously there was no way to find out which messages failed in processing and where they failed. This change adjusts the log levels in ProcessBufferProcessor to allow stack trace and message content logging on DEBUG level. Refs #10319 (cherry picked from commit 6f38abf) * Fix case inconsistencies with ES cluster health status (#10428) (#10431) * Use consistent case for ES cluster health status The cluster health status on ES6 was a lowercase string but on ES7 is uppercase. This change transforms the status in ES7 into a lowercase string. * Make IndexerClusterHealthSummary case independent The component made comparisons of ES cluster health status in a case dependent way, causing #10407. This change ensures all comparisons are done in a case independent way. Fixes #10407 (cherry picked from commit b8f9cad) * Adjust entity dependency resolver to ignore outputs for dashboards (#10443) (#10446) We previously added an exception to ignore outputs when sharing streams. This is also required for sharing dashboards because output dependencies might get pulled in transitively via streams. Fixes Graylog2/graylog-plugin-enterprise#2131 (cherry picked from commit 5606787) Co-authored-by: Bernd Ahlers <bernd@users.noreply.github.com> * Fix rendering of WorldMapVisualization in reports (#10450) (#10473) ## Motivation Prior to this change, we used a prop to pass the interactive context value to the MapVisualization component. But this prop was never set so the defaultProp (true) was used. ## Description This change will remove the prop and use the InteractiveContext.Consumer directly just like in GenericPlot, to ensure interactive is always set correctly. * Allow using the same LDAP attribute for different config settings (#10437) (#10462) Some users want to use the same attribute (e.g. "cn") for different config settings. (e.g. name and full name) Also don't skip the unique ID attribute when collecting user attributes. This is an issue when the same attribute is used for the unique attribute and another config setting (note recommended) because the attribute will be missing from user details. Adjust the LDAP login test output to use a special key for the unique ID attribute to avoid confusion because of the base64 encoded value. Fixes #10069 (cherry picked from commit 7e64e6a) * Adjust ADAuthServiceBackend login test output (#10489) Adjust the AD login test output to use a special key for the unique ID attribute to avoid confusion because of the base64 encoded value. Similar to the change in #10437. Fixes #10069 (cherry picked from commit fbadce9) Co-authored-by: Bernd Ahlers <bernd@users.noreply.github.com> * Support + in index set prefixes. (4.0) (#10507) * URL-encode template name so special characters are quoted. * Quote special characters in index prefixes before creating regex. * Adding test case. Co-authored-by: Dennis Oelkers <dennis@graylog.com> * Cleaning up tests, adding test case for issue. * Do not quote index match pattern. * Encode usernames used in URLs (#10567) Usernames may contain some URL reserved characters that may break API calls. This PR fixes those instances where we were not yet encoding usernames. Fixes #10530, backport of #10566 * Bump package.json version to 4.0.7 * [graylog2-server] prepare release 4.0.7 * Bump package.json version to 4.0.8-SNAPSHOT * [graylog2-server] prepare for next development iteration Co-authored-by: Konrad Merz <konrad@graylog.com> Co-authored-by: Edmundo Alvarez <edmundo@graylog.com> Co-authored-by: Dennis Oelkers <dennis@graylog.com> Co-authored-by: Bernd Ahlers <bernd@users.noreply.github.com> Co-authored-by: Linus Pahl <linus@graylog.com> Co-authored-by: Gaya <gaya@theclevernode.com> Co-authored-by: Jenkins <garybot2@torch.sh> Co-authored-by: Donald Morton <donald.morton@graylog.com> Co-authored-by: Marco Pfatschbacher <marco@graylog.com> Co-authored-by: Bernd Ahlers <bernd@graylog.com> Co-authored-by: Jan Heise <jan@janheise.de>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Using a shaded dependency of the old Kafka 0.9 client, allows us to update to a current Kafka version,
without breaking backwards compatibility on our on-disk Kafka journal.
The disk-journal and the Kafka legacy input keeps on using the old library.
Motivation and Context
The old client has several bugs.
It is known to hang on high loads and shows problems when partitions are re-balanced.
How Has This Been Tested?
Tested against multiple Kafka versions:
2.7.0,2.4.0,0.10.2.2(the latter also with legacy mode)I've extended partitions for a topic:
bash-4.4# kafka-topics.sh --topic=rsyslog --bootstrap-server=localhost:9092 --alter --partitions=5They get rebalanced just fine:
Screenshots (if appropriate):
Types of changes
Fixes #7101
Fixes #7783 (tested with SASL_PLAIN and SCRAM-SHA-512)
Fixes #9927