Lookup tables (#3748)

* WIP on DTOs and domain objects

* Start LookupTableService

* wip data structures/dto services

* fix botched rebase

* add resources for caches, adapters and lookup tables

extends Objectmapper provider to allow registering NamedTypes for subtype resolution in plugins
cache and data adapter configs are polymorphic and require proper binding of config classes

* add missing property annotations for cache api dtos

* add even more property annotations and make ids nullable in REST DTOs

forgetting builder annotations is quickly becoming my favorite bug

* consolidate resources into one

without a single hierarchy getting the cache/adapter types is awkward as it would shadow the GET {idOrName} call

* convert data adapters to factory with descriptors

this simplifies lifecycle management, so we can easily instantiate the descriptor without having to have a complete data adapter

return the information in the `/system/lookups/types/adapters` call

* convert lookup cache metadata to factory/descriptor

in order to get metadata (which cache providers are present, what default config do they have) we need to expose the descriptors
since we don't want to instantiate the entire cache, we break out the metadata into its own class that is provided
the factory can also create instances of the actual caches, based on its configuration

* build lookup tables during start up

this change assembles lookup tables from configured caches and adapters
a GET resource for data is exposed on /system/lookup/data/{name}

moved adapters and caches to their own packages to make easier on the eye
introduced NullCache (type "none") to disable caching

* send update/delete cluster events when lookup tables or underlying objects are mutated

adapter and cache deletion is denied if those are in use by any lookup table in the system

* fix lookup cache and adapter usages

fix wrong ids in warning log messages

* don't recreate cache or adapter when their config hasn't changed

when updating a lookup table we need to take care not to throw away existing caches or stateful adapters
that would make them lose their state, even though nothing has changed

* add generic lookup table support

* guard against nulls

* initial skeleton for lookup table UI

 - lookup table resource can optionally resolve the referenced caches and data adapters on each page (resulting in fewer API calls and simpler UI code)
 - three new pages are available underneath /system/lookuptables

* minor cosmetic fixes

* lookup table detail view skeleton

* cache list and detail page skeletons

* data adapter UI skeleton

* wip making adapter UI pluggable

actually give random number generator config so we can showcase it

* wip data adapter ui

* simplify data management for adapters pages

only the page is loading data and passing it down to components

* remove obsolete page import

* use correct validation annotation for config

* use correct state from store
attach promise to correct action

* Always lookup values through the cache

The cache handles the lookup in the data adapter on a cache miss.

* Add license headers

* the great cache page copy of 2017

* fix routes

* import cache plugin export definitions from core

use correct guava type name in plugin registration

* Lookup now returns a LookupResult instead of Object (#3731)

The LookupResult contains a map so it can hold multiple values for the
key lookup. It also provides a cache-ttl value to allow custom
expiration for individual results.

* expose simple guava cache configuration

* Remove dataAdapter field from cache, it's accessible through the table

* Always get the lookup table when the actual lookup is executed

This way we avoid keeping a reference to the lookup table for too long.
That would be an issue when the lookup table gets updated.

* wip lookup table ui

added create/edit form for lookup tables
not possible to save them yet, neither to pick adapters and caches

don't load pagination twice for adapters and caches

extract some common css

* Refactor LookupTableService

- Rename createTable() to updateTable()
- Remove duplication in initialize() and updateTable() methods
- Move table creation into a separate class

* always return promises to minimize accidents by implicit returns

* Add some more debug/warn log statements to LookupTableService#updateTable

* Add lifecycle management to LookupDataAdapter and subclasses

Some data adapters need lifecycle management to handle thread pools,
background jobs etc.

* create/edit form for lookup tables

* Do not return without unlocking

It's safe to check started/stopped before grabbing the lock.

* Support an optional documentation component for adapters and caches

* audit events for create/update/delete lookup operations

* Add a CSV file lookup table data adapter

* Disable react/no-unescaped-entities for csvfile documentation

* Use java.nio API to create file input stream and set a charset

https://www.cloudbees.com/blog/fileinputstream-fileoutputstream-considered-harmful

* Add more information about CSV file requirements

Reformat to look nicer.

* Return deleted tables/caches/data adapters in the response

* Use java.nio.charset.StandardCharsets to please forbidden-apis

* added TimeUnitInput core component

extend guava cache configuration to include
 * refresh keys
 * both expire after options

added documentation to edit/create form
pass down explicit "updateConfig" prop handler to support more complex config updates from plugin fieldsets

* fix wrong attribute name

* remove dummy random data adapter

graylog2-server/src/main/java/org/graylog2/audit/AuditEventTypes.java

@@ -83,6 +83,15 @@ public class AuditEventTypes implements PluginAuditEventTypes {
     public static final String LDAP_GROUP_MAPPING_UPDATE = PREFIX + "ldap_group_mapping:update";
     public static final String LOAD_BALANCER_STATUS_UPDATE = PREFIX + "load_balancer_status:update";
     public static final String LOG_LEVEL_UPDATE = PREFIX + "log_level:update";
+    public static final String LOOKUP_ADAPTER_CREATE = PREFIX + "lut_adapter:create";
+    public static final String LOOKUP_ADAPTER_DELETE = PREFIX + "lut_adapter:delete";
+    public static final String LOOKUP_ADAPTER_UPDATE = PREFIX + "lut_adapter:update";
+    public static final String LOOKUP_CACHE_CREATE = PREFIX + "lut_cache:create";
+    public static final String LOOKUP_CACHE_DELETE = PREFIX + "lut_cache:delete";
+    public static final String LOOKUP_CACHE_UPDATE = PREFIX + "lut_cache:update";
+    public static final String LOOKUP_TABLE_CREATE = PREFIX + "lut_table:create";
+    public static final String LOOKUP_TABLE_DELETE = PREFIX + "lut_table:delete";
+    public static final String LOOKUP_TABLE_UPDATE = PREFIX + "lut_table:update";
     public static final String MESSAGE_DECORATOR_CREATE = PREFIX + "message_decorator:create";
     public static final String MESSAGE_DECORATOR_DELETE = PREFIX + "message_decorator:delete";
     public static final String MESSAGE_DECORATOR_UPDATE = PREFIX + "message_decorator:update";
@@ -201,6 +210,15 @@ public class AuditEventTypes implements PluginAuditEventTypes {
             .add(LDAP_GROUP_MAPPING_UPDATE)
             .add(LOAD_BALANCER_STATUS_UPDATE)
             .add(LOG_LEVEL_UPDATE)
+            .add(LOOKUP_ADAPTER_CREATE)
+            .add(LOOKUP_ADAPTER_DELETE)
+            .add(LOOKUP_ADAPTER_UPDATE)
+            .add(LOOKUP_CACHE_CREATE)
+            .add(LOOKUP_CACHE_DELETE)
+            .add(LOOKUP_CACHE_UPDATE)
+            .add(LOOKUP_TABLE_CREATE)
+            .add(LOOKUP_TABLE_DELETE)
+            .add(LOOKUP_TABLE_UPDATE)
             .add(MESSAGE_DECORATOR_CREATE)
             .add(MESSAGE_DECORATOR_DELETE)
             .add(MESSAGE_DECORATOR_UPDATE)

graylog2-server/src/main/java/org/graylog2/bindings/ServerBindings.java

@@ -16,10 +16,12 @@
  */
 package org.graylog2.bindings;
 
-import com.floreysoft.jmte.Engine;
 import com.google.inject.Scopes;
 import com.google.inject.assistedinject.FactoryModuleBuilder;
 import com.google.inject.multibindings.Multibinder;
+
+import com.floreysoft.jmte.Engine;
+
 import org.apache.shiro.mgt.DefaultSecurityManager;
 import org.elasticsearch.client.Client;
 import org.elasticsearch.node.Node;
@@ -60,6 +62,7 @@
 import org.graylog2.inputs.InputEventListener;
 import org.graylog2.inputs.InputStateListener;
 import org.graylog2.inputs.PersistedInputsImpl;
+import org.graylog2.lookup.LookupModule;
 import org.graylog2.plugin.RulesEngine;
 import org.graylog2.plugin.cluster.ClusterConfigService;
 import org.graylog2.plugin.inject.Graylog2Module;
@@ -120,6 +123,7 @@ protected void configure() {
         install(new AuthenticatingRealmModule());
         bindSearchResponseDecorators();
         install(new GrokModule());
+        install(new LookupModule());
     }
 
     private void bindProviders() {

graylog2-server/src/main/java/org/graylog2/bindings/providers/MongoJackObjectMapperProvider.java

@@ -24,6 +24,7 @@
 import com.fasterxml.jackson.databind.jsontype.TypeIdResolver;
 import com.fasterxml.jackson.databind.module.SimpleModule;
 import com.fasterxml.jackson.databind.type.SimpleType;
+
 import org.graylog2.indexer.retention.strategies.UnknownRetentionStrategyConfig;
 import org.graylog2.jackson.MongoJodaDateTimeDeserializer;
 import org.graylog2.jackson.MongoJodaDateTimeSerializer;
@@ -33,11 +34,14 @@
 import org.joda.time.DateTime;
 import org.mongojack.internal.MongoJackModule;
 
-import javax.inject.Inject;
-import javax.inject.Provider;
 import java.io.IOException;
 import java.time.ZonedDateTime;
 
+import javax.inject.Inject;
+import javax.inject.Provider;
+import javax.inject.Singleton;
+
+@Singleton
 public class MongoJackObjectMapperProvider implements Provider<ObjectMapper> {
     private final ObjectMapper objectMapper;
 

graylog2-server/src/main/java/org/graylog2/lookup/DtoLoader.java

@@ -0,0 +1,66 @@
+/**
+ * This file is part of Graylog.
+ *
+ * Graylog is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * Graylog is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with Graylog.  If not, see <http://www.gnu.org/licenses/>.
+ */
+package org.graylog2.lookup;
+
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Maps;
+import com.google.inject.assistedinject.Assisted;
+import org.graylog2.lookup.dto.CacheDto;
+import org.graylog2.lookup.dto.DataAdapterDto;
+import org.graylog2.lookup.dto.LookupTableDto;
+
+import javax.inject.Inject;
+import java.util.Collection;
+import java.util.Map;
+import java.util.Optional;
+
+/**
+ * Loads and caches lookup table related DTOs to avoid hammering the database when loading all lookup tables for the
+ * first time during server startup.
+ */
+public class DtoLoader {
+    interface Factory {
+        DtoLoader create(Collection<LookupTableDto> lookupTableDtos);
+    }
+
+    private final Map<String, CacheDto> caches;
+    private final Map<String, DataAdapterDto> dataAdapters;
+
+    @Inject
+    DtoLoader(@Assisted Collection<LookupTableDto> lookupTableDtos,
+              MongoLutCacheService cacheService,
+              MongoLutDataAdapterService dataAdapterService) {
+        final ImmutableSet.Builder<String> cacheIds = ImmutableSet.builder();
+        final ImmutableSet.Builder<String> adapterIds = ImmutableSet.builder();
+
+        lookupTableDtos.forEach(dto -> {
+            cacheIds.add(dto.cacheId());
+            adapterIds.add(dto.dataAdapterId());
+        });
+
+        this.caches = Maps.uniqueIndex(cacheService.findByIds(cacheIds.build()), CacheDto::id);
+        this.dataAdapters = Maps.uniqueIndex(dataAdapterService.findByIds(adapterIds.build()), DataAdapterDto::id);
+    }
+
+    Optional<CacheDto> getCacheDto(String id) {
+        return Optional.ofNullable(caches.get(id));
+    }
+
+    Optional<DataAdapterDto> getDataAdapterDto(String id) {
+        return Optional.ofNullable(dataAdapters.get(id));
+    }
+}

graylog2-server/src/main/java/org/graylog2/lookup/LookupModule.java

@@ -0,0 +1,51 @@
+/**
+ * This file is part of Graylog.
+ *
+ * Graylog is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * Graylog is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with Graylog.  If not, see <http://www.gnu.org/licenses/>.
+ */
+package org.graylog2.lookup;
+
+import com.google.inject.assistedinject.FactoryModuleBuilder;
+
+import org.graylog2.lookup.adapters.CSVFileDataAdapter;
+import org.graylog2.lookup.caches.GuavaLookupCache;
+import org.graylog2.lookup.caches.NullCache;
+import org.graylog2.plugin.inject.Graylog2Module;
+
+public class LookupModule extends Graylog2Module {
+
+    @Override
+    protected void configure() {
+        bind(LookupTableService.class).asEagerSingleton();
+
+        install(new FactoryModuleBuilder().build(DtoLoader.Factory.class));
+        install(new FactoryModuleBuilder().build(LookupTableCreator.Factory.class));
+
+        installLookupCache(NullCache.NAME,
+                NullCache.class,
+                NullCache.Factory.class,
+                NullCache.Config.class);
+
+        installLookupCache(GuavaLookupCache.NAME,
+                GuavaLookupCache.class,
+                GuavaLookupCache.Factory.class,
+                GuavaLookupCache.Config.class);
+
+        installLookupDataAdapter(CSVFileDataAdapter.NAME,
+                CSVFileDataAdapter.class,
+                CSVFileDataAdapter.Factory.class,
+                CSVFileDataAdapter.Config.class);
+    }
+
+}

graylog2-server/src/main/java/org/graylog2/lookup/LookupTable.java

@@ -0,0 +1,70 @@
+/**
+ * This file is part of Graylog.
+ *
+ * Graylog is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * Graylog is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with Graylog.  If not, see <http://www.gnu.org/licenses/>.
+ */
+package org.graylog2.lookup;
+
+import com.google.auto.value.AutoValue;
+import org.graylog.autovalue.WithBeanGetter;
+import org.graylog2.plugin.lookup.LookupCache;
+import org.graylog2.plugin.lookup.LookupDataAdapter;
+import org.graylog2.plugin.lookup.LookupResult;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+@AutoValue
+@WithBeanGetter
+public abstract class LookupTable {
+
+    @Nullable
+    public abstract String id();
+
+    public abstract String title();
+
+    public abstract String description();
+
+    public abstract String name();
+
+    public abstract LookupCache cache();
+
+    public abstract LookupDataAdapter dataAdapter();
+
+    public static Builder builder() {
+        return new AutoValue_LookupTable.Builder();
+    }
+
+    @Nullable
+    public LookupResult lookup(@Nonnull Object key) {
+        return cache().get(key);
+    }
+
+    @AutoValue.Builder
+    public abstract static class Builder {
+        public abstract Builder id(String id);
+
+        public abstract Builder title(String title);
+
+        public abstract Builder description(String description);
+
+        public abstract Builder name(String name);
+
+        public abstract Builder cache(LookupCache cache);
+
+        public abstract Builder dataAdapter(LookupDataAdapter dataAdapter);
+
+        public abstract LookupTable build();
+    }
+}