New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Waiting for data adapter/cache creation in bundle importer. #4197

Merged
merged 4 commits into from Oct 4, 2017

Conversation

Projects
None yet
2 participants
@dennisoelkers
Member

dennisoelkers commented Sep 29, 2017

Description

Motivation and Context

Before this change, lookup data adapters, caches and tables were created
sequentially during content pack import, leading to errors when lookup
tables were created before caches or data adapters were successfully
created and started. Imported lookup tables were usable only after the
server was restarted at least once after a content pack import.

After this change, the bundle importer waits for the successful creation
of data adapters/caches before creating lookup tables.

Fixes Graylog2/graylog-plugin-threatintel#57.

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.
Waiting for data adapter/cache creation in bundle importer.
Before this change, lookup data adapters, caches and tables were created
sequentially during content pack import, leading to errors when lookup
tables were created before caches or data adapters were successfully
created and started. Imported lookup tables were usable only after the
server was restarted at least once after a content pack import.

After this change, the bundle importer waits for the successful creation
of data adapters/caches before creating lookup tables.

Fixes Graylog2/graylog-plugin-threatintel#57.

@dennisoelkers dennisoelkers added the bug label Sep 29, 2017

@dennisoelkers dennisoelkers added this to the 2.4.0 milestone Sep 29, 2017

final CountDownLatch latch = new CountDownLatch(caches.size());
caches.forEach(c -> c.addListener(new LatchUpdaterListener(latch), scheduler));
latch.await(30, TimeUnit.SECONDS);

This comment has been minimized.

@joschi

joschi Oct 2, 2017

Contributor

We should at least log a short message on either INFO or WARN if the timeout expired (which means that not all caches were refreshed).

final CountDownLatch latch = new CountDownLatch(dataAdapters.size());
dataAdapters.forEach(da -> da.addListener(new LatchUpdaterListener(latch), scheduler));
latch.await(30, TimeUnit.SECONDS);

This comment has been minimized.

@joschi

joschi Oct 2, 2017

Contributor

We should at least log a short message on either INFO or WARN if the timeout expired (which means that not all caches were refreshed).

This comment has been minimized.

@dennisoelkers
try {
latch.await(30, TimeUnit.SECONDS);
} catch (InterruptedException e) {
LOG.warn("Starting imported Lookup Table Caches did not finish within 30 seconds. A server restart might be required for imported Lookup Tables to function.");

This comment has been minimized.

@joschi

joschi Oct 4, 2017

Contributor

The InterruptedException is not thrown when the latch's timeout expires but only when the thread is being interrupted.

If the timeout expired, CountDownLatch#await() will return false.

This comment has been minimized.

@dennisoelkers

dennisoelkers Oct 4, 2017

Member

(Should read javadocs properly)

try {
latch.await(30, TimeUnit.SECONDS);
} catch (InterruptedException e) {
LOG.warn("Starting imported Lookup Table Data Adapters did not finish within 30 seconds. A server restart might be required for imported Lookup Tables to function.");

This comment has been minimized.

@joschi

joschi Oct 4, 2017

Contributor

The InterruptedException is not thrown when the latch's timeout expires but only when the thread is being interrupted.

If the timeout expired, CountDownLatch#await() will return false.

This comment has been minimized.

@dennisoelkers

dennisoelkers Oct 4, 2017

Member

(Should read javadocs properly)

@joschi

joschi approved these changes Oct 4, 2017

@joschi joschi merged commit 3493509 into 2.4 Oct 4, 2017

5 checks passed

ci-web-linter Jenkins build graylog-pr-linter-check 1974 has succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
graylog-project/pr Jenkins build graylog-project-pr-snapshot 551 has succeeded
Details
license/cla Contributor License Agreement is signed.
Details

@joschi joschi deleted the awaiting-data-adapter-and-cache-creation-in-bundle-importer branch Oct 4, 2017

joschi added a commit that referenced this pull request Oct 4, 2017

Waiting for data adapter/cache creation in bundle importer
Before this change, lookup data adapters, caches and tables were created
sequentially during content pack import, leading to errors when lookup
tables were created before caches or data adapters were successfully
created and started. Imported lookup tables were usable only after the
server was restarted at least once after a content pack import.

After this change, the bundle importer waits for the successful creation
of data adapters/caches before creating lookup tables.

Fixes Graylog2/graylog-plugin-threatintel#57
Refs #4197
(cherry picked from commit 3493509)

dennisoelkers added a commit that referenced this pull request Oct 4, 2017

Waiting for data adapter/cache creation in bundle importer (#4206)
Before this change, lookup data adapters, caches and tables were created
sequentially during content pack import, leading to errors when lookup
tables were created before caches or data adapters were successfully
created and started. Imported lookup tables were usable only after the
server was restarted at least once after a content pack import.

After this change, the bundle importer waits for the successful creation
of data adapters/caches before creating lookup tables.

Fixes Graylog2/graylog-plugin-threatintel#57
Refs #4197
(cherry picked from commit 3493509)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment