New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor lifecycle for lookup adapters and caches #3873

Merged
merged 16 commits into from Jun 12, 2017

Conversation

Projects
2 participants
@kroepke
Member

kroepke commented May 29, 2017

we use auxiliary servicemanagers to start lookup tables and use listeners to maintain the dependent services' state

Graylog2/graylog-plugin-map-widget#48 needs to be merged once this one is in.

@kroepke kroepke added the in progress label May 29, 2017

@kroepke kroepke added this to the 2.3.0 milestone May 29, 2017

@kroepke kroepke requested a review from bernd May 29, 2017

@bernd bernd self-assigned this May 29, 2017

@bernd bernd added this to In Progress in Lookup Tables Jun 7, 2017

@bernd bernd force-pushed the cache-lifecycle branch from bdca888 to cc9dca9 Jun 9, 2017

@bernd bernd force-pushed the cache-lifecycle branch from d52a6f8 to 9d61bbb Jun 10, 2017

@bernd bernd added ready-for-review and removed in progress labels Jun 12, 2017

@bernd bernd changed the title from [WIP] add initial lifecycle to caches and make them services to Refactor lifecycle for lookup adapters and caches Jun 12, 2017

kroepke and others added some commits May 29, 2017

add initial lifecycle to caches and make them services
we use auxiliary servicemanagers to start lookup tables and use listeners to maintain the dependent services' state
refactoring of lookup table entity lifecycle
caches and data adapters now run on their own, without being lazily loaded when a lookup table uses them
this greatly simplfies the update and creation logic and makes it trivial to query the system state

not all update/delete events are filled in the resource yet
add error states to caches (no users yet)
expose errors in errorstate api endpoint
actually start caches/adapters when updating
avoid race condition by waiting for a running service before re-creating lookup tables
Fall back to a DB check in LookupTableService#hasTable()
The live tables might not be populated yet.
improve service logging by adding the object identity
fix wrong state in logging service listener
convert caches from loading to manual caches
since we are sharing instances across lookup tables, the caches and adapters no longer have a back reference to the table
this means the lookup table needs to orchestrate loading and caching

only loading caches can use refreshAfterWrite, so we had to remove that option from the config

make sure restarting caches and adapters is properly synchronized by waiting for all instances to be running
Create a LookupCacheKey to be used as key for the lookup caches
The LookupCacheKey consists of the actual key object and a prefix String
to make it possible to cache the same key value for a different data
adapter. Since cache instances can be shared between data adapters,
cache values might be overwritten for different data adapters otherwise.
Move data adapter refresh handling into LookupDataAdapterRefreshService
This decouples the data adapters from the caches and handles the refresh
logic outside of the data adapter.

@bernd bernd force-pushed the cache-lifecycle branch from 1e56cdb to f94127d Jun 12, 2017

@bernd

This comment has been minimized.

Member

bernd commented Jun 12, 2017

@kroepke I would like to merge this PR so it doesn't get bigger and bigger.

Please review the two commits I added to move the refresh handling out of the adapters.

@kroepke

This comment has been minimized.

Do we need this to be @JsonCreator in case we want to serialize it ever?

This comment has been minimized.

Member

bernd replied Jun 12, 2017

I thought we didn't need it because we do not deser it yet. But I will add @JsonCreator and @JsonProperty annotations to make sure it gets deser correctly. 👍

This comment has been minimized.

Member

bernd replied Jun 12, 2017

@kroepke Added in 4fd2020

@kroepke

This comment has been minimized.

Member

kroepke commented Jun 12, 2017

lgtm!

@bernd

bernd approved these changes Jun 12, 2017

@bernd bernd merged commit 51157da into master Jun 12, 2017

1 of 4 checks passed

graylog-project/pr Jenkins build graylog-project-pr-snapshot 209 has failed
Details
continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
continuous-integration/travis-ci/push The Travis CI build is in progress
Details
ci-web-linter Jenkins build graylog-pr-linter-check 1731 has succeeded
Details

@bernd bernd deleted the cache-lifecycle branch Jun 12, 2017

@bernd bernd removed the ready-for-review label Jun 12, 2017

@bernd bernd moved this from In Progress to Done in Lookup Tables Jun 14, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment