Trim all String field values #186

lennartkoopmann · 2013-08-20T22:52:47Z

No description provided.

* Unify navigation across all pages The navigation that was changing on pages was immensely confusing me. Like already done for other parts of the web interface that are following a similar pattern, this change addresses this by always having the same navigation items and highlighting the currently active one. * Remove now duplicate button to simulator This helps a little with the large amount of buttons on the page. * Fix typo * Warn when a pipeline isn't connected to streams Show a prominent warning on the pipeline details page in case the pipeline is not connected to any streams. * Improve "No stream connection" warning Include a note that you do not have to connect a pipeline to a stream if you intend to use it for decorators only. Also make clear that pipelines that are connected to a stream will process incoming messages.

@test

* Call `finishProcessing` in the right place We should call `InterpreterListener.finishProcessing()` only once, but before that was not necessarily true, as it was being called inside a loop. Instead, now we call it in the same method as `startProcessing()`, after all processing has been finished. Fixes #51 * Adapt to changed decorators interface (#43) * Providing a message decorator that uses pipelines. * Making decorator configurable. * Allow adding new messages by pipeline decorator. * Adding changes related due to introduced listener. * Adapt to naming changes, using easier forEach idiom. * Changing decorator to work on SearchResponse instead of message list. * Improve error message on regex() * Update evalError test rule Since #32, the behaviour of `to_ip()` changed and now it doesn't raise an exception in those circumstances. Now we use `regex()` to throw an exception instead. * Add sample Decorator preset. (#52) * Providing a message decorator that uses pipelines. * Making decorator configurable. * Allow adding new messages by pipeline decorator. * Adding changes related due to introduced listener. * Adapt to naming changes, using easier forEach idiom. * Changing decorator to work on SearchResponse instead of message list. * Adding decoration stats for pipeline processor decorator. * Add uppercase decorator using pipelines interpreter with preset. * Decorators don't need to generate decoration stats on their own anymore. * Bumping versions to 2.1.0-beta.1 / 1.1.0-beta.1 * [graylog-plugin-pipeline-processor-release] prepare release 1.1.0-beta.1 * [graylog-plugin-pipeline-processor-release] prepare for next development iteration * Bumping graylog dependency version to next development iteration. * Making pipeline decorator more robust if removed pipelines are ref'd. * Fix permissions on system navigation route * Update npm dependencies Refs #2544 * Fix cancel button on new pipeline form Fixes #57 * Update Travis CI configuration Disable building the web-part of this project because it would require a full checkout of the Graylog web interface. * Add missing license headers * Move -Dskip.web.build=true into install/script commands (Travis CI) * Fix Travis CI badge in README.md [ci skip] * Run on Trusty build environment (Travis CI) * Fix issues with app prefix (#66) * Use core Routes instead of literals This allows us to prefix routes if needed. * Make plugin aware of __webpack_public_path__ setting Refs #2564 * Support "only named captures" for pipeline grok function (#65) The server cache is necessary because the named captures support needs a separately compiled regex. So far the cache is only used by the grok function in the pipeline processor Closes #59 * Make conversion functions more consistent (#64) * there was a bug with to_string returning null instead of its default value (refs #63) * all core conversion functions now return their "default empty" value if the value is `null` - String: "" - bool: false - double: 0d - long 0L - IP: V4 ANY (0.0.0.0) * adds test cases for all cases, including the edge cases * Add type hint for Travis CI's Java compiler to be happy * Update Maven plugins * Sync dependencies with Graylog 2.1.0-beta.2-SNAPSHOT * Disable failing tests in FunctionsSnippetsTest Refs #64 * Fix FunctionsSnippetsTest after conversion function unification (#70) `evalError()` can no longer trigger the error tested for and was removed. `evalErrorSuppressed()` now tests an illegal default value in `to_ip()`. Fixes #64 * Remove wildcard type cast to make IDEA 2016.2 happy (#71) This change should not affect `javac` at all, but intellij flags the collect call with having two errors. * Bumping versions to 2.1.0-beta.2 / 1.1.0-beta.2 * [graylog-plugin-pipeline-processor-release] prepare release 1.1.0-beta.2 * [graylog-plugin-pipeline-processor-release] prepare for next development iteration * Bumping graylog dependency version to next development iteration. * Remove upper case decorator (#73) There is no real use case for this besides being a good test case for the decorator system development. Fixes #2588 * do not preprocess arguments of the error function fixes #25 * unwrap JsonNode values (#72) e.g. strings would be double quoted without this fixes #68 * add optional prefix/suffix to set_fields functions (#75) fixes #74 * Add key-value parsing function (#77) Fixes #38 * Allow selection of an input ID for the simulation message (#78) * Invert equals() call to avoid possible null pointer exception Fixes #2610 * Allow selection of an input ID for the simulation message Pipeline rules may use the `from_input` function in a condition so this is needed to make the simulation work. Refs #2610 * Bumping versions to 2.1.0-beta.3 / 1.1.0-beta.3 * [graylog-plugin-pipeline-processor-release] prepare release 1.1.0-beta.3 * [graylog-plugin-pipeline-processor-release] prepare for next development iteration * Bump server dependency to 2.1.0-beta.4-SNAPSHOT * Unregister PipelineInterpreter from event bus (#79) Message decorators and the pipeline simulator create new instances of PipelineInterpreters that never get garbage collected, as they are still registered in the event bus. These changes add a simple workaround for that. We should probably refactor the lifecycle of the PipelineInterpreter, but this is probably not the best time to do it. * Pipeline UI improvements (#83) * Make changes summary the default view on simulator * Improve message when there are no pipeline connections Include links to rules and pipelines if there are no pipelines available. * Display pipelines using a certain rule On the edit rule page, show pipelines that are using that rule, including a link to them. * Fix add new pipeline button position * Improve navigation options Change some of the options in the top right navigation to adapt better to the workflow. * Disable message actions in simulator This is a left over of when we used real messages to test the pipelines. Fixes #2683 * Dynamic function list (#89) * add resource to access entire function registry * add descriptions to Functions and Parameters add basic UI for displaying function descriptors not searchable yet * descriptions for conversion functions * descriptions for date functions * descriptions for hash functions * description for cidrmatch * description for json functions * descriptions for message functions * descriptions for null/notnull functions * descriptions for from_input * descriptions for string functions * tweak table widths #lolcss * Use find in the regex function (#88) Do not force regular expressions passed to `regex()` to match the whole string. Fixes #35 * Bumping versions to 2.1.0-beta.4 / 1.1.0-beta.4 * [graylog-plugin-pipeline-processor-release] prepare release 1.1.0-beta.4 * [graylog-plugin-pipeline-processor-release] prepare for next development iteration * Bumping graylog dependency version to next development iteration. * Support DateTime comparison (#92) Add support for comparison operators on DateTime objects. Fixes #86 * Make some small UI changes around RuleHelper (#90) - Update text descriptions - Hide page selector input * use shared classloader so other plugins can contribute functions (#94) fixes #81 * Fix loading issue by removing <includes> from resource * Rename graylog2.{plugin-dir,version} to graylog.* * Add missing Graylog-Plugin-Properties-Path manifest entry * Integrate audit log (#96) * Integrate audit log * Replace edit with update action on pipeline connections * Add license * add parse error handler for precompute args failures (#93) * add parse error handler for precompute args failures allow lowercase timezone ids * make error message look nicer * don't hit undo before commit… * Fix license check * Bumping versions to 2.1.0-rc.1 / 1.1.0-rc.1 * [graylog-plugin-pipeline-processor-release] prepare release 1.1.0-rc.1 * [graylog-plugin-pipeline-processor-release] prepare for next development iteration * Bumping graylog dependency version to next development iteration. * Fix page size in function list (#97) `slice()` doesn't include the end element in the array that returns, so we were rendering 9 functions instead of 10. * Add '@test' annotation to keyValue test Refs #77 * Correct trim value argument description * Update version to 1.2.0-SNAPSHOT * use case insensitive lookup for timezone IDs (#102) * use case insensitive lookup for timezone IDs simply upper-casing timezone IDs failed for strings like 'Europe/Moscow'. Unfortunately the forID function is case sensitive. fixes #100 * override the millis provider to stabilize test * remove unused statement and fix import * Display boolean values in pipeline simulator (#99) Convert field values into strings for displaying. Fixes #54 (cherry picked from commit 18a31177aedb47a14fd6db7c7400e6e9d5a3c1c0) * Simplify pipeline processor UI (#106) Fixes #104 * Use pipelines overview page as entry point * Add connected streams to pipeline overview page * Move pipeline connection lists to its own component * Add API method to connect streams to a pipeline Migrating pipeline connections to the pipeline page also means that we need to create connections in the reverse order: several streams connected to a pipeline. I added a new API method for it, as I think the old method can still be useful for someone using the API programatically. * Format PipelineConnectionsActions * Set connections from pipeline details view * Use nicer word break in timeline * Cleanup after moving connections to pipeline view * Add link to manage rules to rule details page * Adapt pipeline simulator to recent changes Now the stream selection needs to be done in the simulate page, as we don't know before which stream to use. * Implement arithmetic operators (#111) Refs #16 * Create benchmark harness for file-based rules/pipelines (#107) * initial wip version of a parameterized jmh runner for rules and pipelines * implement in-memory rule service to be able to use guice in the benchmarks, so we can reuse the production bindings, we need in-memory variants of the services we have in order to avoid benchmarking database access the mongodb backed services have been renamed and are bound by default * [wip] adding in-memory rule service use in-memory grok service from #2914 add some license headers some in-memory services are still missing and the pom packaging needs work * add in-memory pipeline and stream connections service dummy stream service implementation because upstream will change this anyway in the near future use in-memory module instead of anonymous module * make in-memory services singletons so we can load data into them actually read benchmarks from disk and populate the stores etc see `match_all_rule` `benchmark.toml` for an example of how to set up a benchmark * split pipeline plugin into parent, benchmarks and actual plugin lots of refactoring needed because the benchmarks need to be build as a separate module due to jmh requirements explicit mention of jmh annotation processing necessary because something overrides the setting, making classpath autodetection barf this requires an updated manifest to work with graylog-project, will push separately slight tweaks to the benchmark class * support no-fork mode (-Dprofile) and running a single benchark (-Dbenchmark.name) for debugging/profiling add yourkit profiling commands as comments (useful to enable when digging into why a benchmark is slow) (manually include the yourkit -redist jar to classpath to use) add another multi-stage benchmark as baseline for stage performance * make build work again another web-parent artifact: we need to disable the globally enabled frontend plugin * add benchmark * move misplaced files due to rebasing madness * move eslint file * remove old files left by improper rebase * Use webplugin babel6 (#112) * update to web-plugin and babel6, requires graylog2-server branch update-babel-packages-and-config * "unskip" web plugin the parent has to skip the execution of the frontend maven plugin, but the actual plugin needs to enable it * Adjust to new parent structure * bump to version 2.2.0-alpha.1 * bump parent version to 2.2.0-alpha.1 * add distributionManagement * fix versions * Correcting submodule versions. * Making graylog dependency version dynamic. * Binding version to project parent version. * Revert "Binding version to project parent version." This reverts commit a977672. * Revert "Making graylog dependency version dynamic." This reverts commit b80529d. * Bumping graylog2-server version, adding dependency mgmt clause. * [maven-release-plugin] prepare release 2.2.0-alpha.1 * [maven-release-plugin] prepare for next development iteration * bump to snapshot server version * update benchmark pom after alpha.1 * failing test for bug #2952 * Revert "failing test for bug #2952" This reverts commit fcdf628. * Plugin infrastructure changes - Add `<groupId/>` to make versions plugin work correctly - Enable maven deploy - Add `<repositories/>` to make building on travis work - Finish switch to `graylog-plugin.properties` - Use dependency versions from parent properties if possible - Use plugin versions from parent if possible * [graylog-plugin-pipeline-processor] prepare release 2.2.0-alpha.2 * [graylog-plugin-pipeline-processor] prepare release 2.2.0-alpha.3 * [graylog-plugin-pipeline-processor] prepare for next development iteration * Fix build by enabling the frontend-maven-plugin again * create tarball for benchmark package, to avoid having to peel out benchmark files from the jar. also makes it easier to run new benchmarks * don't try to use 512 gb..., turn off verbose flag for git maven plugin * Do not use when building with webpack * cleanup pom, bump to auto value snapshot for memoized extension * include git describe in benchmark artifact * tweak benchmark invocation * fix graylog.version property * don't look up pipelines twice during interpreter run * memoize Pipeline#hashCode * reuse meters in pipelines/stages/rules this prevents spending unnecessary time in recreating the names for the metrics, which saves a considerable amount of time * rewrite StageIterator to do less on-the-fly work when the iterator is created the entire set of stages is known, don't try to be clever and pre-compute the set instead of having to do computation to solve it "elegantly" * write CSV result files * replace TreeSet usage by ImmutableSortedSet, which is just a list underneath * [graylog-plugin-pipeline-processor] prepare release 2.2.0-alpha.4 * [graylog-plugin-pipeline-processor] prepare for next development iteration * lazily initialize rarely used EvaluationContext properties we can save some allocations by defering allocation the maps and lists * bump auto-value to 1.4-rc1 * remove superfluous empty constructor * Messagelist benchmark (#119) Load test messages from csv files or generate 25000 static ones if the benchmark requests that, otherwise fall back on a single message. * extract pipeline state updater from interpreter by separating the state computation and updating from the interpreter we can control what state we want to use for interpretation this allows a cleaner simulation without interfering with current processing it also makes the code a little easier to follow by separating the two different concerns * Split up process* methods for better visibility in profiler/code clarity (#121) * don't doubly negate the value of the expression (#125) fixes #124 * fix wording to be consistent: metrics -> throughput * Use explicit default stream (#123) * Display error message when system has no streams Pipeline simulation currently requires a stream, but we didn't handle the case where a system had no streams. These changes will display a message indicating the situation, and link to the streams page, where a new stream can be created. * Use explicit default stream in REST and UI Remove special handling of default stream in REST resources and web interface. * Use explicit default stream in processor * Add method to delete pipeline connections * Add migration for legacy default stream Move stream connections to the legacy default stream to the new, explicit default stream. * Use explicit default stream in interpreter tests As the default stream behaves like another stream, we need to explicitly add the stream to messages routed to it. * wip codegen * first working version of rule code generation for when conditions function invocation works (pipeline function authors API stayed the same) arguments are pre-transformed (but constant expressions are not hoisted yet) * create code for then method implement Map and Array/List literals assert that then method is properly executed * only resolve functions once * prioritize function call over var reference * track the referenced expression in varref and fieldref for constant checking * ignore varref arguments when precomputing constant args during direct tree interpretation the variables are not yet defined, so we need to skip them code generation doesn't care and won't ever take this code path * hoist constant expressions/statements into rule constructor this includes variable assignements, literal creation as well as function arguments no attempt is made to clean the variables yet, which may be necessary between runs hoisted constant literals are not enforced to be constant yet, however the generated code also never touches them again, so it seems unnecessary to wrap them in Unmodifiable* objects * use correct rule reference, otherwise we are simply measuring a false rule * bump to use latest server version * attach generated rule code and execute it if present * cache stage iterator content for each different pipeline set setting up the stage iterator is by far the most expensive part of interpreting overhead * Revert "cache stage iterator content for each different pipeline set" This reverts commit 49116699c94dde1eeba9e63419cd8543942df3ff. * moved stage iterator config caching into interpreter state configuration option to toggle caching (mostly for benchmarking) * add arithmetics and indexed access to code generator run all rule-based parser tests with and without code generation refactor test functions to work around square/javapoet#526 * when resolving a rule, make an invokable copy with a fresh instance of the generated code if present this avoid having to make the generated code threadsafe, which would severely impact runtime performance * add configuration options to turn off stage caching and code generation refactor classloader passing, so that each state reload reuses a single classloader * remove large message file * fix benchmark * only show generated code on TRACE (#133) * Turn off code generation if no compiler available (#134) * detect whether the environment has a compiler available and complain if we don't this typically happens in a JRE, which has the ToolProvider, but not the tools.jar. we ask to run on a JDK instead fail more gracefully on compilation errors fixes #132 * only warn when code generation was requested so the user can silence the warning * Date arithmetic (#136) * initial date arithmetic support #91 * code generation and interpreter date arithmetic plus tests * remove obsolete comment about operator selection * prevent adding two dates at parse time attempting to add (not subtract) two dates raises an InvalidOperation parse error because there aren't any sane semantics for doing so while subtraction gives the duration between the dates, there's hardly any good reason to add them * add individual function tests add test for subtracting 10000 years * fix benchmark due to signature change in core * [graylog-plugin-pipeline-processor] prepare release 2.2.0-beta.1 * [graylog-plugin-pipeline-processor] prepare for next development iteration * Fix Javadoc errors about self-closing element <br/> * fix NPEs when code generation is off and a function is unresolved (#141) there were two possible NPEs during parse checking and when compiling rules with unresolved functions * replace openhft compiler wrapper non-caching version (#137) * replace openhft compiler wrapper with simpler version that does not cache the caching compiler wrapper prevented class unloading, which we depend on unscientific test that checks for class unloading is added migrate all call sites * Update ConfigurationStateUpdaterTest.java * add to_date() function for easier interaction with $message.timestamp (#140) this function makes the following statement possible: set_field("timestamp", to_date($message.timestamp) + hours(1)); * do not register cache gauges more than once (#139) * do not register cache gauges more than once invalid re-registering of cache gauges caused pipeline state updates to fail the second time refactor the state updating to not use event bus, but instead let the interpreter pull the state directly from the instance it has decorators and simulator push the state into their interpreter instance, making the interpreter immutable itself. * Update ConfigurationStateUpdater.java remove obsolete publishing of state on the event bus users of the state now poll the state * don't remove the initial state reload * Implement split() function (#143) Fixes #98 * Attach Javadoc and Sources artifacts and sign artifacts in "release" profile (#144) Fixes #142 * Disable native code generation until compiler interface is fixed (#145) this change disables the tests for codegen as well as the config option to be a no-op * [graylog-plugin-pipeline-processor] prepare release 2.2.0-beta.2 * [graylog-plugin-pipeline-processor] prepare for next development iteration * Add page titles (#148) Refs #2834 * Remove experimental flag from pages 🎉 (#149) * Implement loadAllWithIndexSet method (#150) Refs #3207 * [graylog-plugin-pipeline-processor] prepare release 2.2.0-beta.3 * [graylog-plugin-pipeline-processor] prepare for next development iteration * Do not use lambdas with gauge metrics (#152) Instead use CacheStatsSet to expose Guava cache stats to metric registry Fixes #146 * Add clone_message() function (#153) * Add clone_message() function Closes #138 * Add $message parameter to clone_message() * Track total pipeline interpreter executionTime as a single metric (#155) Fixes #3124 * [graylog-plugin-pipeline-processor] prepare release 2.2.0-beta.4 * [graylog-plugin-pipeline-processor] prepare for next development iteration * Allow duplicate stream titles in route_to_stream (#154) Cache stream ids and titles to avoid heavy database traffic during function evaluation Fixes #101 * [graylog-plugin-pipeline-processor] prepare release 2.2.0-beta.5 * [graylog-plugin-pipeline-processor] prepare for next development iteration * [graylog-plugin-pipeline-processor] prepare release 2.2.0-beta.6 * [graylog-plugin-pipeline-processor] prepare for next development iteration * [graylog-plugin-pipeline-processor] prepare release 2.2.0-rc.1 * [graylog-plugin-pipeline-processor] prepare for next development iteration * Shrinkwrap JS dependencies for Graylog 2.2.0 (#157) More information in: https://docs.npmjs.com/cli/shrinkwrap * [graylog-plugin-pipeline-processor] prepare release 2.2.0 * [graylog-plugin-pipeline-processor] prepare for next development iteration * Remove npm shrinkwrap * Bump version to 2.3.0-SNAPSHOT (#161) * Update to react-bootstrap 0.30 (#164) * Change onSelect handler signature * Update gitignore Exclude babel cache * Replace deprecated react-bootstrap Inputs Use the new API when only using Input as a wrapper, otherwise using the adaptor Input component. * Add id to Tab to improve accessibility * Use uppercase timezone in TimezoneAwareFunction and fix default value (#169) Fixes #168 * Lookup tables (#177) * add generic lookup table support * Use the new LookupResult return type in the lookup function (#176) * [graylog-plugin-pipeline-processor] prepare release 2.3.0-alpha.1 * [graylog-plugin-pipeline-processor] prepare for next development iteration * Adjust lookup function to single value change in server (#178) Also add a lookup_value function. * Adjust to upcoming changes in LookupResult (#179) * [graylog-plugin-pipeline-processor] prepare release 2.3.0-alpha.2 * [graylog-plugin-pipeline-processor] prepare for next development iteration * Spelling: safety (#181) [ci skip] * Add support for custom locale in parse_date() (#184) Closes #183 * [graylog-plugin-pipeline-processor] prepare release 2.3.0-alpha.3 * [graylog-plugin-pipeline-processor] prepare for next development iteration * Smaller UI and UX changes (#186) * Unify navigation across all pages The navigation that was changing on pages was immensely confusing me. Like already done for other parts of the web interface that are following a similar pattern, this change addresses this by always having the same navigation items and highlighting the currently active one. * Remove now duplicate button to simulator This helps a little with the large amount of buttons on the page. * Fix typo * Warn when a pipeline isn't connected to streams Show a prominent warning on the pipeline details page in case the pipeline is not connected to any streams. * Improve "No stream connection" warning Include a note that you do not have to connect a pipeline to a stream if you intend to use it for decorators only. Also make clear that pipelines that are connected to a stream will process incoming messages. * New function: debug() (#188) * New function: debug() Added a new function that will log the string represntation of any value that has been passed to it. I just found this immensely helpful when working with a lookup table result. (cherry picked from commit fbe8cfb81e8c90f4b299c54cae8594642a57bec0) * add missing license header * Allow snake-case access to bean objects (#189) If the given field name does not work, the code now tries to convert the field name to a camel case value. Fixes Graylog2/graylog-plugin-map-widget#47 * Improve lookup functions (#191) - Enable default parameter - Add a description for function parameters - Return null instead of an optional if the default is not set * [graylog-plugin-pipeline-processor] prepare release 2.3.0-beta.1 * [graylog-plugin-pipeline-processor] prepare for next development iteration * Prevent ClassCastException with invalid timestamp in clone_message() (#192) * Prevent ClassCastException with invalid timestamp in clone_message() If the "timestamp" field of a message is invalid, `clone_message()` will fail because `Message#getTimestamp()` tries to cast this field to a `DateTime` object. With the changes in this commit, the type of the "timestamp" field will be checked before accessing it and if it is invalid (i. e. not a `DateTime` object), the current date and time will be used and the original content will be stored in the "_original_timestamp" field of the new message. Fixes #3880 * Convert original (invalid) timestamp to string * Log warning if message has invalid timestamp * Rename field to "gl2_original_timestamp" * Fix handling of "match all"/"match either" (#193) The pipeline interpreter had a bug regarding the handling of the "match all" and "match either" statements which caused pipelines containing stages with "match all" to continue processing even if not all rules in the stage were executed. Fixes #3924 * Fix serialization/deserialization of StageSource (#195) * Fix serialization/deserialization of StageSource Some important Jackson annotations were missing from the StageSource class which prevented proper deserialization via the Graylog REST API. Fixes #194 * Add missing "stages" attribute to PipelineSource JsonCreator * Use friendlier error message in case of invalid expressions (#196) Closes #185 * [graylog-plugin-pipeline-processor] prepare release 2.3.0-rc.1 * [graylog-plugin-pipeline-processor] prepare for next development iteration * Bump version to 2.4.0-SNAPSHOT * Make "locale" parameter of "parse_date()" optional The "locale" parameter of the "parse_date()" function processed as optional already, but wasn't marked as such in the function signature. Refs #202 * Add various Base encoding functions (#190) * Sort alphabetically by title in MongoDbRuleService#loadAll() (#208) The previous implementation returned an intransparent sorting of rules to the user. Users expect a stable sorting of rules, so we'll sort them alphabetically by title to get a stable sorting. * parse_json() returns MissingNode if input wasn't valid JSON (#210) The `parse_json()` function is supposed to return a `JsonNode` but returned `null` if the input wasn't valid JSON. This change set changes the return type to `MissingNode` if the input wasn't valid and couldn't be parsed. Closes #209 * Convert to use separate prop-types package Closes #205 * [graylog-plugin-pipeline-processor] prepare release 2.4.0-alpha.1 * [graylog-plugin-pipeline-processor] prepare for next development iteration * Fix plugin author formatting * [graylog-plugin-pipeline-processor] prepare release 2.4.0-alpha.2 * [graylog-plugin-pipeline-processor] prepare for next development iteration * Use Auto Value version from Graylog Plugin Parent * [graylog-plugin-pipeline-processor] prepare release 2.4.0-alpha.3 * [graylog-plugin-pipeline-processor] prepare for next development iteration * Bump version to 3.0.0-SNAPSHOT * Prevent NPE in FunctionArgs#getConstantArgs() if Expression is null (#212) Closes #211 * Set active state in the right prop (#214) * Adapt to changes in Select (#213) - Replace deprecated `onValueChange` prop - Add `required` prop, now supported * Specify type of 'value' (single or multi) for lookup functions (#217) * Link to pipeline processors docs instead of Graylog website (#218) Refs Graylog2/documentation#377 * Fix numeric conversions with to_double()/to_long() (#219) * Fix numeric conversions with to_double()/to_long() The functions for numeric conversions, `to_long()` and `to_double()`, didn't properly support converting from strings or other numeric types. Refs https://community.graylog.org/t/graylog-pipeline-problem/2810 * Add test cases for numeric corner cases (min/max/infinity) * Sort function list on rules edit page alphabetically by name (#222) This makes it much easier to find anything on that page. * Register accidentally forgotten Base64Encode and Base64Decode functions (#223) Refs #190 * Add functions to remove messages from streams (#220) * Add `remove_from_default` boolean option to `route_to_stream()` function This allows remove messages from the default stream like the stream router already does * Add `remove_from_stream()` function This function removes the function from the given streams (via name or id) if the message would be taken off all streams, it is routed back to the default stream to avoid dropping the message accidentally * Add missing Base64Decode and Base64Encode imports * React router update (#225) * Fix history prop deprecation Use global history instead. * Replace deprecated history pushState with push * Add missing Input IDs (#226) * Filter null values when iterating over `select_jsonpath` results (#233) Fixes #232 * Source code editor (#234) * Use SourceCodeEditor component * Use annotations prop to set validations * Add "parse_unix_milliseconds" functions The `parse_unix_milliseconds()` function enables users to parse a UNIX epoch timestamp in milliseconds into a date object. * Remove Netty 3 dependency in CidrMatch (#239) Replace `org.jboss.netty.handler.ipfilter.CIDR` with `org.graylog2.utilities.IpSubnet` to get rid of the Netty 3 dependency in the `cidr_match()` function. Refs #4226 * Handle PipelineInterpreter.State event to avoid warning in the log (#241) Fixes #236 * Stabilize date/time based tests The message timestamp used to be in the system default locale instead of UTC. Additionally, this commit adds tests/examples for accessing individual components of the message timestamp in a pipeline rule. * Move `parse_unix_milliseconds()` tests into separate test case * Add `starts_with` and `ends_with` string functions (#227) * Add comparison functions for all supported types (#237) * Add comparison functions for all supported types * Ensure actions were triggered in FunctionsSnippetsTest#comparisons() * Add support for JsonNode to ´set_fields()` (#228) * Add support for JsonNode to ´set_fields()` Sometimes users might want to parse and merge the JSON payload of a message with the Graylog message without knowing the complete structure of the payload or without having a fixed structure which could be selectively merged by using the `json_path()` method. This commit essentially adds the possiblity to create a pipeline rule emulating the existing JSON extractor: rule "json" when // some condition then let json = parse_json(to_string($message.some_field)); set_fields(json); end * Introduce `to_map` function to convert JsonNode to Map * Fix description of `to_map()` function * Merge Pipeline Processor plugin into Graylog core Closes Graylog2/graylog-plugin-pipeline-processor#216 * Migrate URL class to HttpUrl for more sanity ...and less Forbidden APIs warnings * Fix or suppress Error Prone and Forbidden APIs compile errors * Add PipelineProcessorAuditEventTypes to AuditCoverageTest * Move loading PipelineConfig into Server command Otherwise the named "cached_stageiterators" and "generate_native_code" settings cannot be found. While this *should* work with the `PluginModule` interface and the `PluginModule#getConfigBeans()` method, fixing this would require some refactorings which I'd like to defer to a later point in time * Fix React routes * Remove benchmarks and unused files

* Initial commit * Add AWS SDK v2 dependency * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Add Kinesis SDK * General UI Skeleton (#75) * Simple Clickthrough without API * Clickable Skeleton All Steps * Feedback * Lint * AWS Cloudwatch backend API calls Add initial backend API calls for CloudWatch integration: getRegions, getLogGroups, getStreams, retrieveKinesisMessages, healthCheck * AWS Cloud Watch services and resources (#24) Adds beginnings of API endpoint and Kinesis/Cloudwatch services Includes a structure that we will continue to build from. * Rework organization of classes for unified structure The goal is to establish some structure that we can implement AWS API calls within. There's now one resource (for API calls), one service (for business logic), and one AWSClient (for all AWS/API SDK interactions). * Cleanup and add comments * Cleanup code around log message auto-detection * Aws cloudwatch client (#31) * Add CloudWatchService class * Add AWSConfigSettings class * Add UserCredentials class * Add temporary Main class * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Update AWSConfigSettings class * Update CloudWatchService class * Update temporary Main class * Fix inject error in AWSResource * Fix merge conflicts * Restructure Resources and Service classes * Add getRegion into AWSResource (#43) * Add getRegion into AWSResource * Update api paths * Increase max retry limit for stream get to 1000 Runaway looping Stopping at 100 is probably to small. * Use underscores instead of camel case for json names It is a general project standard to use underscores. * Add paging functionality to getLogGroupNames (#53) * Add paging functionality to getLogGroupNames * Add unit test for CloudWatch log groups * Add comments in log group name unit test * Code clean up and remove unneeded code * Revert unintended change from unit test commit * Add retrievelogs for Kinesis Healthcheck (#76) * Refactor getKinesisStreams in KinesisService * Add validCredentials method in AWSService class * Temporary Main class added to test putting records into a Kinesis stream * Add retrieveKinesisLogs in KinesisService * Update temporary Main class * Update retrieveKinesisLogs to loop through shard iterators * Update Main class * Update pom file * Update validateCredentials in AWSService class * Add createKinesisClient method in KinesisService class * Add testGetStreamCredentials and update testGetStreams * Healthcheck merge (#81) * empty commit to push branch * Add Kinesis Healthcheck (#45) * Improve organization for Flow Log message detection * Improve Flow Log test It now tests for a message with too many and too few spaces. * Add TODOs for healthCheck method logic * Add beginnings of Kinesis healthChecker This will pull a establish a Kinesis subscription and pull a single message from a Kinesis stream. * Fix failing unit test * Continue developing KinesisHealthCheck - Remove unneeded metric tracking - Remove extra parsing logic (this object should just hand back payload and not do any parsing) - Improve application name handling - Add comments * Add detection logic for raw vs. CloudWatch logs * Remove KinesisHealthCheck class The KinesisConsumer does not work well for the health check (designed for realtime processing, takes a long time to start, cannot detect empty stream, and is really hard to use in a quick API request). Now, we're planning to directly retrieve the messages using the Kinesis client. This is the most straight-forward thing We might revisit this later. * Fix JSON parsing of Kinesis CloudWatch subscription record Parse the record just as was done in the existing AWS plugin. The logic now includes autodetection of compressed/vs not compressed. Mock Kinesis CloudWatch subscription record included for testing purposes. * Add CloudWatch logs codec and tests from existing AWS plugin * Parse Flow Log message into object * Load appropriate codec during healthCheck process When the message type is detected, load the respective codec for that message type. * Parse message with appropriate codec Once the log message type is detected, then the codec is looked up. The message is then parsed with the codec. * Supply log group name with the response * Improve comments, logging, and error checking The log group name is now also included in the response. * Add Flow Log codec test * Use AutoValue for CloudWatchLogEntry class * Use AutoValue for all remaining CloudWatch value classes * Cleanup merge conflicts after rebasing * Specify constants for all JsonProperty annotations * Delete uneeded KinesisDTO All data will be stored in the input * Establish a base AWSRequest JSON class * Fix Guice injection error for KinesisService * Add sample cURL command for healthCheck method A similar cURL command will be used for other methods, so that it is clear how the UI will use them. * Remove unneeded Kinesis Client 1.x dependency * Add formatted message summary in the Health Check response * Cleanup formatting and TODOs * Minor cleanup after rebasing and merging * Fix failing unit tests * Integrate AWS Health Check with Kinesis record retrieval (#86) * Fix incorrect pass of AWS key instead of secret Also improve comments for fake message retrieval with TODOs. * Update and connect retrieveRecords * Add handleCompressedMessages method * Delete temporary main class * Update retrieveRecords to only return sample size * Update KinesisService for healthCheck to function properly * Add unit test for selecting random record * Add unit test for retrieveRecords * Add Available Services API call (#82) * Resolves #50: Add Available Services API call * Add a test * Add missing spaces, change Amazon > AWS * Backend code cleanup (#90) * Pass region and AWS credentials with AWS API requests (#92) * Require POST object containing region and credentials for all requests Specifically adds a POST body requirement for the getKinesisSteams and getLogGroupNames methods. * Use snake_case for paths * Update region api call (#110) * Migrate Regions request from a list to a full response object with total * Update Region API call to include label and value * Reformat code * Aws permissions check (#113) * Add AWSPermissions class and update AWSResource with permission checks * Rename and remove permissions in AWSPermissions * Register AWSPermissions in IntegrationsModule * Remove space between methods * Save AWS input API endpoint (#93) * First-pass structure for saving AWS input * Add more structure for general AWS input - Add a type enum to differentiate the various types of log messages that are possible. - Add metacodec that will eventually differentiate between the types of log messages. - Fill configuration values when saving the input. * Resolve merge conflicts after rebasing over latest aws branch * Consolidate log type detection and input type identification There's no longer a need to use two enums for this. Also added healthCheck tests covering all message types: flow log, raw Cloud Watch and raw Kinesis. * Clean up saveInput request parameters and handling * Fix invalid type specification that was preventing input save * Add unit test for saving input * Fix incorrectly specified arguments * Minor cleanup * Cleanup for PR review * Remove uneeded log statements * Make save AWS input path and description more specific * Indicate that the save request is specifically for Kinesis In the future, each type of AWS input will likely require it's own request object and endpoint due to the fact that unique fields will probably be required for each. * AWSMessageType cleanup * Remove uneeded isFlowLog, isRaw methods. * Remove invalid AWSMessageType.Source.CLOUD_WATCH enum value. Messages are always read from Kinesis, and therefore the source is always Kinesis. Source is meant to differentiate messages from Kinesis and S3 for example. * Improve comments for AWSMessageType.Source enum class and method. * Remove typo * Generify the create AWS integration endpoint The naming, description, and comment now reflect that a generic AWS input is being created. * Return InputSummary response entity for AWS input creation request Also remove unneeded AWSResourceTest * More cleanup of healthCheck after input creation changes - Remove unneeded log_group field for health_check request. Resolves #108 - Add Kinesis stream name as a field in both raw and CloudWatch messages This change is lumped in with the other changes related to saving the input, since lots of healthCheck changes were already made there. This fixes some problems, so might as well have these improvements included with the review. * Clarify that AWSMetaCodec is a general AWS codec This class no longer erroneously extends AbstractKinesisCodec, which was only intended for Kinesis-specific codecs. * Use DateTime instead of long in KinesisLogEntry It turns out that Kinesis Record objects do have an arrival ime Instant timestamp. This is now being used instead of just using the date/time when the message was read by Graylog. * Add missing Guice annotation that were preventing server startup * Return JSON message field list in response from Health Check (#114) * Return JSON fields in healthCheck response There is still an issue with serializing the DateTime timestamp (serializing the deep object tree). This will be addressed separately. * Adapt unit test timestamp check to object value Health Check now returns a map of actual field values, so the test needs to check the JSON value. * Fix incorrect codec specified for Flow Logs, add tests * Fix incorrectly named field * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn * Advanced Kinesis Options * Cleanup * currently busted toggle * redo state handling * Remove artifact * removing commented code * removing unnecessary functions * pr feedback * Rebase conflicts * Fix several issues from PR review * "Final Review" layout (#83) * General Content * Review view layout complete * Cleanup * Update utils file name and export * Change LogFlow > Flow Log * default_values.js files * Route constants (#111) * Route constants * Fixing conflicts * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn Route constants Fixing conflicts * Correct import path * Changing route object * Adjust Kinesis setup wording * Remove the unneeded * Fixing up rebase artifacts from squash of irrelevant commits * AWS lockfile * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * Setting up necessary hooks * Removing input validation steps to reduce PR * Rebasing * Rename StyledForm * DEFAULT_VALUES * cleanup hook returns * Moving some reducers to state and breaking apart the rest * Remove functions from Input values * Steps function cleanup * Remove commented import * FormDataProvider and Context * Steps Context & provider * log and advancedoptions moved to context and provider * FormWrap functional component * renaming things * lint error * PR feedback * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * error and dirty states * Error on submit * Update validation to new state hooks * remove unneeded defaults * Output error messages * text key instead of password * Remove field validation * Update comment blocks (#137) * Update comment blocks * Remove minor typos * Update version to 3.1.0-beta.2-SNAPSHOT * Implement AWS codec selector with tests (#116) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Remove erroneous comment * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump package.json version to 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * Authorize and Streams API * All API calls except Save * reverting to hooks for api calls * Regions hook * Streams and Logs * streams useFetch * log sample * generic loading states * move regions * adding some documentation * Adding all the files * removing uneeded * cleanup * ApiRoutes * import proper path * Changing up useFetch * need the values... * lint * Removed unneeded inline cURL API documentation (#143) No need for this docs duplication. The System > Nodes > API Browser already documents them. * Use standard field names for AWS key and secret (#144) * Use standard naming for AWS secret and access key ids for Save Input Use these standard names for AWS key and secret from the Save Input AWS endpoint (From the AWSRequest interface). All other AWS API requests should be using these already. `aws_access_key_id` `aws_secret_access_key` * Use central constant for Region also * Fix incorrectly set Region property name * Fix access and secret key method names to match the AWSRequest interface * rename useFetch file * better var naming in kinesisstreams: * Update useFetch props * AWS transport selector with tests (#117) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Remove duplicate AWSTestingUtils class * Add missing permissions for sample AWS policy (#122) * Migrate AWS policy to autovalue object This avoids embedding JSON in a Java constant, which is error prone. * Add missing permissions Added: - kinesis:ListStreams - logs:DescribeLogGroups * Define JSON policy field order that matches AWS examples * Return the policy JSON string to the UI as string This is consistent with how this was done before. The UI can then format and display this to the user on the Available Services page. This is the policy that the user will create the AWS user for the AWS integration with. * Specify the correct AWS policy version * Change AWS Policy to a list * Add missing permissions Based on this [sample KCL code](aws/aws-sdk-java-v2#1214 (comment)) * Update version to 3.1.0-beta.2-SNAPSHOT * Remove uneeded import * Add Available Services API response error documentation * Throw InternalServerErrorException instead of JSON exception This is a bit cleaner from the API consumption side. A nice short error message is now returned instead of an obscure JSON error. * API Save Input (#145) * Save after Review and redirect to Inputs * Save functional * Adding match function to pluginmanifest * custom component * Update version to 3.1.0-beta.3-SNAPSHOT * Update version to 3.1.0-beta.3-SNAPSHOT * Aws kinesis newstream (#129) * Add create new Kinesis stream method * Add API call for createNewKinesisStream and KinesisNewStream classes * Update createNewKinesisStream * Update version to 3.1.0-beta.2-SNAPSHOT * Add error logging to createNewKinesisStream method * Add unit test for createNewKinesisStream method * Update error logging for createNewKinesisStream * Update testCreateNewKinesisStream * Remove shard count variable and set default value * Update testCreateNewKinesisStream * Code clean up * Update api comment blocks * Code clean up * Update error handling in createNewKinesisStream * Update unit test testCreateNewKinesisStream * Unload Confirmation Dialog (#148) * Playing with onunload * use ConfirmLeaveDialog common component * Question * Update version to 3.1.0-beta.3-SNAPSHOT * lint * cleanup * Don't alert on last step. Save call failed if desc wasn't available * Remove emotion and use styled-components (#157) * Remove emotion and use styled-components * import styled-components * Move title and description to FormWrap for consistent layout * Handle validation per-step * fix aws key h4x * styled validatedinput * enable health check and review forms * Starting Error Handling * Output API Errors * remove some more debugging * Validate API Key & Secret and output styled errors * Error message if stream doesn't contain any logs * Return a 4xx response for an unsuccessful health check * Save API Error Handling * Remove `success` flag from HealthCheckResponse Failure is now tracked by throwing a `BadRequestException` resulting in an `ApiError` json response * Throw `BadRequestException` if no Kinesis streams are found * Throw `BadRequestException` if no Kinesis CloudWatch groups are found * Error message styling * wrap up api errors and adding styles * rebasing and cleanup * trim input values * remove defaults * Create input with togglable masking * Include stream name in error * Port of existing KinesisTransport (#140) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Remove unused imports and fix formatting * Update version to 3.1.0-beta.3-SNAPSHOT * Merge branch 'aws' into aws-transport-kinesis and resolve conflicts # Conflicts: # src/main/java/org/graylog/integrations/aws/service/KinesisService.java * Upgrade to latest Kinesis Client version (#151) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Update Kinesis Client version * Add Kinesis client library v2 Consumer sample * Migrate KinesisConsumer to Kinesis Client Library v2 * Remove unused imports and fix formatting * Finish migration of Kinesis client to new version * Complete upgrade to Kinesis Client Library v2 * Bump KCL version * Migrate shard processor to its own class * Bump version to 3.1.0-beta.3-SNAPSHOT * Add batch size limit * Code cleanup * Remove unneeded throttle time limit reached shutdown * Remove kinesis_max_throttled_wait_ms save input field This field is no longer needed, since the new Kinesis Consumer appears to correctly handle longer throttling and pausing in `processRecords` without making the consumer unhealthy. * Temporarily remove Assume Role Arn auth This will be added back later. See #29 * Adjust logging levels * Remove sample class * Ignore unmapped properties due to removed max_throttled_wait field See Graylog2/graylog-plugin-integrations#156 * Update Throttling advanced option (#169) * Update Throttling advanced option * messages/ms * Remove throttle number * Update label * Shells/mocks for Kinesis Auto setup API endpoints (#167) * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Update version to 3.1.0-beta.4-SNAPSHOT * Code cleanup - Remove unneeded imports - Replace hyphens with underscores * Add `aws` to root URL path for consistency All other AWS routes are prefixed with this, so do the same here. * Change subscription path Change `add_subscription` route to `create_subscription` * Fix linter warning was introduced during merge conflict resolution. * Add RolePermissions for Kinesis stream (#163) * Add checkKinesisStreamStatus and setRolePermissions method * Add getNewRolePermissions and autoKinesisPermissionRequired method * Rename autoKinesisPermissionsRequired method and add error handling * Code clean up * Refactor autoSetup, split up creating a role and attaching a policy * Add error handling for creating a new role * Update autoKinesisPermissionsRequired logic and add comment block * Update version to 3.1.0-beta.4-SNAPSHOT * Delete checkKinesisStreamStatus method * Update error logging messages * Add temporary main class for testing, update error handling logic * Delete temporary main, update autoKinesisPermissionsRequired * Change exceptions thrown and code clean up * Update version to 3.1.0-rc.2-SNAPSHOT * Add addSubscriptionFilter to CloudwatchService (#160) * Add addSubscriptionFilter method to KinesisService * Move addSubscriptionFilter method to CloudWatchService * Code clean up, change exception errors * Bump version to 3.1.0-rc.3-SNAPSHOT * Kinesis Auto-setup (#186) * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump js-yaml from 3.13.0 to 3.13.1 Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 3.13.0 to 3.13.1. - [Release notes](https://github.com/nodeca/js-yaml/releases) - [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md) - [Commits](nodeca/js-yaml@3.13.0...3.13.1) Signed-off-by: dependabot[bot] <support@github.com> * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * [Security] Bump lodash from 4.17.11 to 4.17.14 (#131) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.11 to 4.17.14. **This update includes security fixes.** - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.11...4.17.14) Signed-off-by: dependabot-preview[bot] <support@dependabot.com> * Bump package.json version to 3.1.0-beta.2 * [graylog-plugin-integrations] prepare release 3.1.0-beta.2 * Bump package.json version to 3.1.0-beta.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Bump package.json version to 3.1.0-beta.3 * [graylog-plugin-integrations] prepare release 3.1.0-beta.3 * Bump package.json version to 3.1.0-beta.4-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Render Setup screen * cleanup * Allowing toggle to get from Streams to Setup * Toggle back to Streams & cleanup Group on Region change * Add Kinesis auto-setup routes * Add initial `create_stream` request * Define fetchs for auto Kinesis setup * Finish create_stream request * Add all auto-setup API requests * Specify actual region and request properties * Improve presentation for transition to automated Kinesis setup * Improve on-screen documentation for the automatic Kinesis setup * Bump package.json version to 3.1.0-rc.1 * [graylog-plugin-integrations] prepare release 3.1.0-rc.1 * Bump package.json version to 3.1.0-rc.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Update version to 3.1.0-rc.2-SNAPSHOT * Add KinesisSetupStep shell * Add KinesisSetupSteps container * Update createKinesisStream to return streamArn * Update createKinesisStream and testCreateNewKinesisStream * Add edge case error handling to createNewKinesisStream * Add preliminary code for KinesisFullSetup method * Use await for synchronous auto setup steps * Migrate to map for auto-setup state management * Render auto-setup progress to UI * Include success field and initial delay for testing * Delete KinesisFullSetup method * Migrate Kinesis setup to map to allow setup step iteration * Refactor autoKinesisPermissions parameters and return response * Rework setup logic to use individual state fields Individual state fields are needed for each component to update independently. * Add progress icons * Improve icon and step styles * Add auto-setup error handling * Refactor addSubscriptionFilter to return CreateLogSubscriptionResponse * Update api calls in KinesisFullSetup * Update createNewKinesisStream and createPolicies api call * Add flow control for setup steps * Clean up api methods * Add mocking for UI testing Also remove full setup, since the individual steps will be used. * Sync field mappings between UI and backend * Add auto-setup integration unit test * Add auto-setup test assertions * Rename `explanation` -> `result` * Display result to the user for each auto-setup step * Use unique role ane role policy names Since these are only needed in the auto-setup flow, we can automatically generate and assign them (as long as we provide them to the user in the response). Customizing these names is possible, and can be considered in the future. * Adjustments to get auto-setup working * Add support subscription retries to resolve IAM eventual consistency IAM sometimes takes 10+ seconds to initialize, so retrying allows the subscription attempt to be attempted multiple times, so that once IAM is available, then the subscription will be created successfully. * Cleanup retry logic * Add new page for auto-setup steps * Adjust wording * Add auto-setup TODO for adding warning * TODOs and cleanup * Formatting cleanup * Support ability to continue adding input after auto-setup * Bump package.json version to 3.1.0-rc.2 * [graylog-plugin-integrations] prepare release 3.1.0-rc.2 * Bump package.json version to 3.1.0-rc.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Disable Continue Setup button while auto-setup is in progress * Include warning that messages might take time to arrive in new stream * Require explicit agreement from user before starting auto-setup * Bump version to 3.1.0-rc.3-SNAPSHOT * Adjustments to auto-setup buttons, labels, and formatting * Skip CloudWatch control messages in Health Check Fixes Graylog2/graylog-plugin-integrations#178 * Add comment * Add support for the "flow_log_" prefix Closes Graylog2/graylog-plugin-integrations#120 * Fixing lint errors * Return mock responses * DEV COMMIT * Remove unnecessary Row and Col now that we're rendering a sidebar * sidebar styles * Add checkbox for adding Flow Log field prefix * Auto-Setup Sidebar * lint * Adjust wording * Correct label for AWS Message Type * Consolidate Kinesis payload decompression logic * Agree TOS in Modal * Return actual stream name in create stream mock * Modal Styled * Don't include the policy name in response, since it's inline * Remove DEV COMMIT changes * Update header description * Fix failing unit test * Mock responses * Add default setting for prefix field * Add support for assuming an ARN role (#188) * Add support for assuming an ARN role Fixes Graylog2/graylog-plugin-integrations#29 * Add Assume Role ARN config field * Remove unneeded imports * Fix formatting * Record when users agree to auto-setup (#194) * Add log message recording that a user agreed to auto-setup * Remove unneeded import * Fixing select default value (#199) * Adding pattern to Stream Name input (#200) * Adding pattern to Stream Name input * Don't submit if field has error * Add API request to get permissions (#196) * Add request to retireve Kinesis permissions Includes permissions for both the full setup and the auto-setup. Closes #174 * Alphabetize permissions, add comments, remove unused imports * Add unit test * Make auto-setup unit test pass when mocking enabled * Remove unneeded create log group/stream permissions * Minor code clean up (#201) * Minor code clean up * Fix typo * Bump version to 3.2.0-SNAPSHOT * Linting Fix (#212)

* Initial commit * Add AWS SDK v2 dependency * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Add Kinesis SDK * General UI Skeleton (#75) * Simple Clickthrough without API * Clickable Skeleton All Steps * Feedback * Lint * AWS Cloudwatch backend API calls Add initial backend API calls for CloudWatch integration: getRegions, getLogGroups, getStreams, retrieveKinesisMessages, healthCheck * AWS Cloud Watch services and resources (#24) Adds beginnings of API endpoint and Kinesis/Cloudwatch services Includes a structure that we will continue to build from. * Rework organization of classes for unified structure The goal is to establish some structure that we can implement AWS API calls within. There's now one resource (for API calls), one service (for business logic), and one AWSClient (for all AWS/API SDK interactions). * Cleanup and add comments * Cleanup code around log message auto-detection * Aws cloudwatch client (#31) * Add CloudWatchService class * Add AWSConfigSettings class * Add UserCredentials class * Add temporary Main class * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Update AWSConfigSettings class * Update CloudWatchService class * Update temporary Main class * Fix inject error in AWSResource * Fix merge conflicts * Restructure Resources and Service classes * Add getRegion into AWSResource (#43) * Add getRegion into AWSResource * Update api paths * Increase max retry limit for stream get to 1000 Runaway looping Stopping at 100 is probably to small. * Use underscores instead of camel case for json names It is a general project standard to use underscores. * Add paging functionality to getLogGroupNames (#53) * Add paging functionality to getLogGroupNames * Add unit test for CloudWatch log groups * Add comments in log group name unit test * Code clean up and remove unneeded code * Revert unintended change from unit test commit * Add retrievelogs for Kinesis Healthcheck (#76) * Refactor getKinesisStreams in KinesisService * Add validCredentials method in AWSService class * Temporary Main class added to test putting records into a Kinesis stream * Add retrieveKinesisLogs in KinesisService * Update temporary Main class * Update retrieveKinesisLogs to loop through shard iterators * Update Main class * Update pom file * Update validateCredentials in AWSService class * Add createKinesisClient method in KinesisService class * Add testGetStreamCredentials and update testGetStreams * Healthcheck merge (#81) * empty commit to push branch * Add Kinesis Healthcheck (#45) * Improve organization for Flow Log message detection * Improve Flow Log test It now tests for a message with too many and too few spaces. * Add TODOs for healthCheck method logic * Add beginnings of Kinesis healthChecker This will pull a establish a Kinesis subscription and pull a single message from a Kinesis stream. * Fix failing unit test * Continue developing KinesisHealthCheck - Remove unneeded metric tracking - Remove extra parsing logic (this object should just hand back payload and not do any parsing) - Improve application name handling - Add comments * Add detection logic for raw vs. CloudWatch logs * Remove KinesisHealthCheck class The KinesisConsumer does not work well for the health check (designed for realtime processing, takes a long time to start, cannot detect empty stream, and is really hard to use in a quick API request). Now, we're planning to directly retrieve the messages using the Kinesis client. This is the most straight-forward thing We might revisit this later. * Fix JSON parsing of Kinesis CloudWatch subscription record Parse the record just as was done in the existing AWS plugin. The logic now includes autodetection of compressed/vs not compressed. Mock Kinesis CloudWatch subscription record included for testing purposes. * Add CloudWatch logs codec and tests from existing AWS plugin * Parse Flow Log message into object * Load appropriate codec during healthCheck process When the message type is detected, load the respective codec for that message type. * Parse message with appropriate codec Once the log message type is detected, then the codec is looked up. The message is then parsed with the codec. * Supply log group name with the response * Improve comments, logging, and error checking The log group name is now also included in the response. * Add Flow Log codec test * Use AutoValue for CloudWatchLogEntry class * Use AutoValue for all remaining CloudWatch value classes * Cleanup merge conflicts after rebasing * Specify constants for all JsonProperty annotations * Delete uneeded KinesisDTO All data will be stored in the input * Establish a base AWSRequest JSON class * Fix Guice injection error for KinesisService * Add sample cURL command for healthCheck method A similar cURL command will be used for other methods, so that it is clear how the UI will use them. * Remove unneeded Kinesis Client 1.x dependency * Add formatted message summary in the Health Check response * Cleanup formatting and TODOs * Minor cleanup after rebasing and merging * Fix failing unit tests * Integrate AWS Health Check with Kinesis record retrieval (#86) * Fix incorrect pass of AWS key instead of secret Also improve comments for fake message retrieval with TODOs. * Update and connect retrieveRecords * Add handleCompressedMessages method * Delete temporary main class * Update retrieveRecords to only return sample size * Update KinesisService for healthCheck to function properly * Add unit test for selecting random record * Add unit test for retrieveRecords * Add Available Services API call (#82) * Resolves #50: Add Available Services API call * Add a test * Add missing spaces, change Amazon > AWS * Backend code cleanup (#90) * Pass region and AWS credentials with AWS API requests (#92) * Require POST object containing region and credentials for all requests Specifically adds a POST body requirement for the getKinesisSteams and getLogGroupNames methods. * Use snake_case for paths * Update region api call (#110) * Migrate Regions request from a list to a full response object with total * Update Region API call to include label and value * Reformat code * Aws permissions check (#113) * Add AWSPermissions class and update AWSResource with permission checks * Rename and remove permissions in AWSPermissions * Register AWSPermissions in IntegrationsModule * Remove space between methods * Save AWS input API endpoint (#93) * First-pass structure for saving AWS input * Add more structure for general AWS input - Add a type enum to differentiate the various types of log messages that are possible. - Add metacodec that will eventually differentiate between the types of log messages. - Fill configuration values when saving the input. * Resolve merge conflicts after rebasing over latest aws branch * Consolidate log type detection and input type identification There's no longer a need to use two enums for this. Also added healthCheck tests covering all message types: flow log, raw Cloud Watch and raw Kinesis. * Clean up saveInput request parameters and handling * Fix invalid type specification that was preventing input save * Add unit test for saving input * Fix incorrectly specified arguments * Minor cleanup * Cleanup for PR review * Remove uneeded log statements * Make save AWS input path and description more specific * Indicate that the save request is specifically for Kinesis In the future, each type of AWS input will likely require it's own request object and endpoint due to the fact that unique fields will probably be required for each. * AWSMessageType cleanup * Remove uneeded isFlowLog, isRaw methods. * Remove invalid AWSMessageType.Source.CLOUD_WATCH enum value. Messages are always read from Kinesis, and therefore the source is always Kinesis. Source is meant to differentiate messages from Kinesis and S3 for example. * Improve comments for AWSMessageType.Source enum class and method. * Remove typo * Generify the create AWS integration endpoint The naming, description, and comment now reflect that a generic AWS input is being created. * Return InputSummary response entity for AWS input creation request Also remove unneeded AWSResourceTest * More cleanup of healthCheck after input creation changes - Remove unneeded log_group field for health_check request. Resolves #108 - Add Kinesis stream name as a field in both raw and CloudWatch messages This change is lumped in with the other changes related to saving the input, since lots of healthCheck changes were already made there. This fixes some problems, so might as well have these improvements included with the review. * Clarify that AWSMetaCodec is a general AWS codec This class no longer erroneously extends AbstractKinesisCodec, which was only intended for Kinesis-specific codecs. * Use DateTime instead of long in KinesisLogEntry It turns out that Kinesis Record objects do have an arrival ime Instant timestamp. This is now being used instead of just using the date/time when the message was read by Graylog. * Add missing Guice annotation that were preventing server startup * Return JSON message field list in response from Health Check (#114) * Return JSON fields in healthCheck response There is still an issue with serializing the DateTime timestamp (serializing the deep object tree). This will be addressed separately. * Adapt unit test timestamp check to object value Health Check now returns a map of actual field values, so the test needs to check the JSON value. * Fix incorrect codec specified for Flow Logs, add tests * Fix incorrectly named field * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn * Advanced Kinesis Options * Cleanup * currently busted toggle * redo state handling * Remove artifact * removing commented code * removing unnecessary functions * pr feedback * Rebase conflicts * Fix several issues from PR review * "Final Review" layout (#83) * General Content * Review view layout complete * Cleanup * Update utils file name and export * Change LogFlow > Flow Log * default_values.js files * Route constants (#111) * Route constants * Fixing conflicts * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn Route constants Fixing conflicts * Correct import path * Changing route object * Adjust Kinesis setup wording * Remove the unneeded * Fixing up rebase artifacts from squash of irrelevant commits * AWS lockfile * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * Setting up necessary hooks * Removing input validation steps to reduce PR * Rebasing * Rename StyledForm * DEFAULT_VALUES * cleanup hook returns * Moving some reducers to state and breaking apart the rest * Remove functions from Input values * Steps function cleanup * Remove commented import * FormDataProvider and Context * Steps Context & provider * log and advancedoptions moved to context and provider * FormWrap functional component * renaming things * lint error * PR feedback * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * error and dirty states * Error on submit * Update validation to new state hooks * remove unneeded defaults * Output error messages * text key instead of password * Remove field validation * Update comment blocks (#137) * Update comment blocks * Remove minor typos * Update version to 3.1.0-beta.2-SNAPSHOT * Implement AWS codec selector with tests (#116) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Remove erroneous comment * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump package.json version to 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * Authorize and Streams API * All API calls except Save * reverting to hooks for api calls * Regions hook * Streams and Logs * streams useFetch * log sample * generic loading states * move regions * adding some documentation * Adding all the files * removing uneeded * cleanup * ApiRoutes * import proper path * Changing up useFetch * need the values... * lint * Removed unneeded inline cURL API documentation (#143) No need for this docs duplication. The System > Nodes > API Browser already documents them. * Use standard field names for AWS key and secret (#144) * Use standard naming for AWS secret and access key ids for Save Input Use these standard names for AWS key and secret from the Save Input AWS endpoint (From the AWSRequest interface). All other AWS API requests should be using these already. `aws_access_key_id` `aws_secret_access_key` * Use central constant for Region also * Fix incorrectly set Region property name * Fix access and secret key method names to match the AWSRequest interface * rename useFetch file * better var naming in kinesisstreams: * Update useFetch props * AWS transport selector with tests (#117) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Remove duplicate AWSTestingUtils class * Add missing permissions for sample AWS policy (#122) * Migrate AWS policy to autovalue object This avoids embedding JSON in a Java constant, which is error prone. * Add missing permissions Added: - kinesis:ListStreams - logs:DescribeLogGroups * Define JSON policy field order that matches AWS examples * Return the policy JSON string to the UI as string This is consistent with how this was done before. The UI can then format and display this to the user on the Available Services page. This is the policy that the user will create the AWS user for the AWS integration with. * Specify the correct AWS policy version * Change AWS Policy to a list * Add missing permissions Based on this [sample KCL code](aws/aws-sdk-java-v2#1214 (comment)) * Update version to 3.1.0-beta.2-SNAPSHOT * Remove uneeded import * Add Available Services API response error documentation * Throw InternalServerErrorException instead of JSON exception This is a bit cleaner from the API consumption side. A nice short error message is now returned instead of an obscure JSON error. * API Save Input (#145) * Save after Review and redirect to Inputs * Save functional * Adding match function to pluginmanifest * custom component * Update version to 3.1.0-beta.3-SNAPSHOT * Update version to 3.1.0-beta.3-SNAPSHOT * Aws kinesis newstream (#129) * Add create new Kinesis stream method * Add API call for createNewKinesisStream and KinesisNewStream classes * Update createNewKinesisStream * Update version to 3.1.0-beta.2-SNAPSHOT * Add error logging to createNewKinesisStream method * Add unit test for createNewKinesisStream method * Update error logging for createNewKinesisStream * Update testCreateNewKinesisStream * Remove shard count variable and set default value * Update testCreateNewKinesisStream * Code clean up * Update api comment blocks * Code clean up * Update error handling in createNewKinesisStream * Update unit test testCreateNewKinesisStream * Unload Confirmation Dialog (#148) * Playing with onunload * use ConfirmLeaveDialog common component * Question * Update version to 3.1.0-beta.3-SNAPSHOT * lint * cleanup * Don't alert on last step. Save call failed if desc wasn't available * Remove emotion and use styled-components (#157) * Remove emotion and use styled-components * import styled-components * Move title and description to FormWrap for consistent layout * Handle validation per-step * fix aws key h4x * styled validatedinput * enable health check and review forms * Starting Error Handling * Output API Errors * remove some more debugging * Validate API Key & Secret and output styled errors * Error message if stream doesn't contain any logs * Return a 4xx response for an unsuccessful health check * Save API Error Handling * Remove `success` flag from HealthCheckResponse Failure is now tracked by throwing a `BadRequestException` resulting in an `ApiError` json response * Throw `BadRequestException` if no Kinesis streams are found * Throw `BadRequestException` if no Kinesis CloudWatch groups are found * Error message styling * wrap up api errors and adding styles * rebasing and cleanup * trim input values * remove defaults * Create input with togglable masking * Include stream name in error * Port of existing KinesisTransport (#140) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Remove unused imports and fix formatting * Update version to 3.1.0-beta.3-SNAPSHOT * Merge branch 'aws' into aws-transport-kinesis and resolve conflicts # Conflicts: # src/main/java/org/graylog/integrations/aws/service/KinesisService.java * Upgrade to latest Kinesis Client version (#151) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Update Kinesis Client version * Add Kinesis client library v2 Consumer sample * Migrate KinesisConsumer to Kinesis Client Library v2 * Remove unused imports and fix formatting * Finish migration of Kinesis client to new version * Complete upgrade to Kinesis Client Library v2 * Bump KCL version * Migrate shard processor to its own class * Bump version to 3.1.0-beta.3-SNAPSHOT * Add batch size limit * Code cleanup * Remove unneeded throttle time limit reached shutdown * Remove kinesis_max_throttled_wait_ms save input field This field is no longer needed, since the new Kinesis Consumer appears to correctly handle longer throttling and pausing in `processRecords` without making the consumer unhealthy. * Temporarily remove Assume Role Arn auth This will be added back later. See #29 * Adjust logging levels * Remove sample class * Ignore unmapped properties due to removed max_throttled_wait field See Graylog2/graylog-plugin-integrations#156 * Update Throttling advanced option (#169) * Update Throttling advanced option * messages/ms * Remove throttle number * Update label * Shells/mocks for Kinesis Auto setup API endpoints (#167) * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Update version to 3.1.0-beta.4-SNAPSHOT * Code cleanup - Remove unneeded imports - Replace hyphens with underscores * Add `aws` to root URL path for consistency All other AWS routes are prefixed with this, so do the same here. * Change subscription path Change `add_subscription` route to `create_subscription` * Fix linter warning was introduced during merge conflict resolution. * Add RolePermissions for Kinesis stream (#163) * Add checkKinesisStreamStatus and setRolePermissions method * Add getNewRolePermissions and autoKinesisPermissionRequired method * Rename autoKinesisPermissionsRequired method and add error handling * Code clean up * Refactor autoSetup, split up creating a role and attaching a policy * Add error handling for creating a new role * Update autoKinesisPermissionsRequired logic and add comment block * Update version to 3.1.0-beta.4-SNAPSHOT * Delete checkKinesisStreamStatus method * Update error logging messages * Add temporary main class for testing, update error handling logic * Delete temporary main, update autoKinesisPermissionsRequired * Change exceptions thrown and code clean up * Update version to 3.1.0-rc.2-SNAPSHOT * Add addSubscriptionFilter to CloudwatchService (#160) * Add addSubscriptionFilter method to KinesisService * Move addSubscriptionFilter method to CloudWatchService * Code clean up, change exception errors * Bump version to 3.1.0-rc.3-SNAPSHOT * Kinesis Auto-setup (#186) * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump js-yaml from 3.13.0 to 3.13.1 Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 3.13.0 to 3.13.1. - [Release notes](https://github.com/nodeca/js-yaml/releases) - [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md) - [Commits](nodeca/js-yaml@3.13.0...3.13.1) Signed-off-by: dependabot[bot] <support@github.com> * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * [Security] Bump lodash from 4.17.11 to 4.17.14 (#131) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.11 to 4.17.14. **This update includes security fixes.** - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.11...4.17.14) Signed-off-by: dependabot-preview[bot] <support@dependabot.com> * Bump package.json version to 3.1.0-beta.2 * [graylog-plugin-integrations] prepare release 3.1.0-beta.2 * Bump package.json version to 3.1.0-beta.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Bump package.json version to 3.1.0-beta.3 * [graylog-plugin-integrations] prepare release 3.1.0-beta.3 * Bump package.json version to 3.1.0-beta.4-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Render Setup screen * cleanup * Allowing toggle to get from Streams to Setup * Toggle back to Streams & cleanup Group on Region change * Add Kinesis auto-setup routes * Add initial `create_stream` request * Define fetchs for auto Kinesis setup * Finish create_stream request * Add all auto-setup API requests * Specify actual region and request properties * Improve presentation for transition to automated Kinesis setup * Improve on-screen documentation for the automatic Kinesis setup * Bump package.json version to 3.1.0-rc.1 * [graylog-plugin-integrations] prepare release 3.1.0-rc.1 * Bump package.json version to 3.1.0-rc.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Update version to 3.1.0-rc.2-SNAPSHOT * Add KinesisSetupStep shell * Add KinesisSetupSteps container * Update createKinesisStream to return streamArn * Update createKinesisStream and testCreateNewKinesisStream * Add edge case error handling to createNewKinesisStream * Add preliminary code for KinesisFullSetup method * Use await for synchronous auto setup steps * Migrate to map for auto-setup state management * Render auto-setup progress to UI * Include success field and initial delay for testing * Delete KinesisFullSetup method * Migrate Kinesis setup to map to allow setup step iteration * Refactor autoKinesisPermissions parameters and return response * Rework setup logic to use individual state fields Individual state fields are needed for each component to update independently. * Add progress icons * Improve icon and step styles * Add auto-setup error handling * Refactor addSubscriptionFilter to return CreateLogSubscriptionResponse * Update api calls in KinesisFullSetup * Update createNewKinesisStream and createPolicies api call * Add flow control for setup steps * Clean up api methods * Add mocking for UI testing Also remove full setup, since the individual steps will be used. * Sync field mappings between UI and backend * Add auto-setup integration unit test * Add auto-setup test assertions * Rename `explanation` -> `result` * Display result to the user for each auto-setup step * Use unique role ane role policy names Since these are only needed in the auto-setup flow, we can automatically generate and assign them (as long as we provide them to the user in the response). Customizing these names is possible, and can be considered in the future. * Adjustments to get auto-setup working * Add support subscription retries to resolve IAM eventual consistency IAM sometimes takes 10+ seconds to initialize, so retrying allows the subscription attempt to be attempted multiple times, so that once IAM is available, then the subscription will be created successfully. * Cleanup retry logic * Add new page for auto-setup steps * Adjust wording * Add auto-setup TODO for adding warning * TODOs and cleanup * Formatting cleanup * Support ability to continue adding input after auto-setup * Bump package.json version to 3.1.0-rc.2 * [graylog-plugin-integrations] prepare release 3.1.0-rc.2 * Bump package.json version to 3.1.0-rc.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Disable Continue Setup button while auto-setup is in progress * Include warning that messages might take time to arrive in new stream * Require explicit agreement from user before starting auto-setup * Bump version to 3.1.0-rc.3-SNAPSHOT * Adjustments to auto-setup buttons, labels, and formatting * Skip CloudWatch control messages in Health Check Fixes Graylog2/graylog-plugin-integrations#178 * Add comment * Add support for the "flow_log_" prefix Closes Graylog2/graylog-plugin-integrations#120 * Fixing lint errors * Return mock responses * DEV COMMIT * Remove unnecessary Row and Col now that we're rendering a sidebar * sidebar styles * Add checkbox for adding Flow Log field prefix * Auto-Setup Sidebar * lint * Adjust wording * Correct label for AWS Message Type * Consolidate Kinesis payload decompression logic * Agree TOS in Modal * Return actual stream name in create stream mock * Modal Styled * Don't include the policy name in response, since it's inline * Remove DEV COMMIT changes * Update header description * Fix failing unit test * Mock responses * Add default setting for prefix field * Add support for assuming an ARN role (#188) * Add support for assuming an ARN role Fixes Graylog2/graylog-plugin-integrations#29 * Add Assume Role ARN config field * Remove unneeded imports * Fix formatting * Record when users agree to auto-setup (#194) * Add log message recording that a user agreed to auto-setup * Remove unneeded import * Adding Permissions component to StepAuthorize * clear sidebar on unmount * Fixing select default value (#199) * Adding pattern to Stream Name input (#200) * Adding pattern to Stream Name input * Don't submit if field has error * Add API request to get permissions (#196) * Add request to retireve Kinesis permissions Includes permissions for both the full setup and the auto-setup. Closes #174 * Alphabetize permissions, add comments, remove unused imports * Add unit test * Make auto-setup unit test pass when mocking enabled * Remove unneeded create log group/stream permissions * Minor code clean up (#201) * Minor code clean up * Fix typo * Make auto-setup unit test pass when mocking enabled * Bump version to 3.2.0-SNAPSHOT * Bump version to 3.2.0-SNAPSHOT * Adding Policies API * bad comment * Adjust permissions wording

* Initial commit * Add AWS SDK v2 dependency * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Add Kinesis SDK * General UI Skeleton (#75) * Simple Clickthrough without API * Clickable Skeleton All Steps * Feedback * Lint * AWS Cloudwatch backend API calls Add initial backend API calls for CloudWatch integration: getRegions, getLogGroups, getStreams, retrieveKinesisMessages, healthCheck * AWS Cloud Watch services and resources (#24) Adds beginnings of API endpoint and Kinesis/Cloudwatch services Includes a structure that we will continue to build from. * Rework organization of classes for unified structure The goal is to establish some structure that we can implement AWS API calls within. There's now one resource (for API calls), one service (for business logic), and one AWSClient (for all AWS/API SDK interactions). * Cleanup and add comments * Cleanup code around log message auto-detection * Aws cloudwatch client (#31) * Add CloudWatchService class * Add AWSConfigSettings class * Add UserCredentials class * Add temporary Main class * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Update AWSConfigSettings class * Update CloudWatchService class * Update temporary Main class * Fix inject error in AWSResource * Fix merge conflicts * Restructure Resources and Service classes * Add getRegion into AWSResource (#43) * Add getRegion into AWSResource * Update api paths * Increase max retry limit for stream get to 1000 Runaway looping Stopping at 100 is probably to small. * Use underscores instead of camel case for json names It is a general project standard to use underscores. * Add paging functionality to getLogGroupNames (#53) * Add paging functionality to getLogGroupNames * Add unit test for CloudWatch log groups * Add comments in log group name unit test * Code clean up and remove unneeded code * Revert unintended change from unit test commit * Add retrievelogs for Kinesis Healthcheck (#76) * Refactor getKinesisStreams in KinesisService * Add validCredentials method in AWSService class * Temporary Main class added to test putting records into a Kinesis stream * Add retrieveKinesisLogs in KinesisService * Update temporary Main class * Update retrieveKinesisLogs to loop through shard iterators * Update Main class * Update pom file * Update validateCredentials in AWSService class * Add createKinesisClient method in KinesisService class * Add testGetStreamCredentials and update testGetStreams * Healthcheck merge (#81) * empty commit to push branch * Add Kinesis Healthcheck (#45) * Improve organization for Flow Log message detection * Improve Flow Log test It now tests for a message with too many and too few spaces. * Add TODOs for healthCheck method logic * Add beginnings of Kinesis healthChecker This will pull a establish a Kinesis subscription and pull a single message from a Kinesis stream. * Fix failing unit test * Continue developing KinesisHealthCheck - Remove unneeded metric tracking - Remove extra parsing logic (this object should just hand back payload and not do any parsing) - Improve application name handling - Add comments * Add detection logic for raw vs. CloudWatch logs * Remove KinesisHealthCheck class The KinesisConsumer does not work well for the health check (designed for realtime processing, takes a long time to start, cannot detect empty stream, and is really hard to use in a quick API request). Now, we're planning to directly retrieve the messages using the Kinesis client. This is the most straight-forward thing We might revisit this later. * Fix JSON parsing of Kinesis CloudWatch subscription record Parse the record just as was done in the existing AWS plugin. The logic now includes autodetection of compressed/vs not compressed. Mock Kinesis CloudWatch subscription record included for testing purposes. * Add CloudWatch logs codec and tests from existing AWS plugin * Parse Flow Log message into object * Load appropriate codec during healthCheck process When the message type is detected, load the respective codec for that message type. * Parse message with appropriate codec Once the log message type is detected, then the codec is looked up. The message is then parsed with the codec. * Supply log group name with the response * Improve comments, logging, and error checking The log group name is now also included in the response. * Add Flow Log codec test * Use AutoValue for CloudWatchLogEntry class * Use AutoValue for all remaining CloudWatch value classes * Cleanup merge conflicts after rebasing * Specify constants for all JsonProperty annotations * Delete uneeded KinesisDTO All data will be stored in the input * Establish a base AWSRequest JSON class * Fix Guice injection error for KinesisService * Add sample cURL command for healthCheck method A similar cURL command will be used for other methods, so that it is clear how the UI will use them. * Remove unneeded Kinesis Client 1.x dependency * Add formatted message summary in the Health Check response * Cleanup formatting and TODOs * Minor cleanup after rebasing and merging * Fix failing unit tests * Integrate AWS Health Check with Kinesis record retrieval (#86) * Fix incorrect pass of AWS key instead of secret Also improve comments for fake message retrieval with TODOs. * Update and connect retrieveRecords * Add handleCompressedMessages method * Delete temporary main class * Update retrieveRecords to only return sample size * Update KinesisService for healthCheck to function properly * Add unit test for selecting random record * Add unit test for retrieveRecords * Add Available Services API call (#82) * Resolves #50: Add Available Services API call * Add a test * Add missing spaces, change Amazon > AWS * Backend code cleanup (#90) * Pass region and AWS credentials with AWS API requests (#92) * Require POST object containing region and credentials for all requests Specifically adds a POST body requirement for the getKinesisSteams and getLogGroupNames methods. * Use snake_case for paths * Update region api call (#110) * Migrate Regions request from a list to a full response object with total * Update Region API call to include label and value * Reformat code * Aws permissions check (#113) * Add AWSPermissions class and update AWSResource with permission checks * Rename and remove permissions in AWSPermissions * Register AWSPermissions in IntegrationsModule * Remove space between methods * Save AWS input API endpoint (#93) * First-pass structure for saving AWS input * Add more structure for general AWS input - Add a type enum to differentiate the various types of log messages that are possible. - Add metacodec that will eventually differentiate between the types of log messages. - Fill configuration values when saving the input. * Resolve merge conflicts after rebasing over latest aws branch * Consolidate log type detection and input type identification There's no longer a need to use two enums for this. Also added healthCheck tests covering all message types: flow log, raw Cloud Watch and raw Kinesis. * Clean up saveInput request parameters and handling * Fix invalid type specification that was preventing input save * Add unit test for saving input * Fix incorrectly specified arguments * Minor cleanup * Cleanup for PR review * Remove uneeded log statements * Make save AWS input path and description more specific * Indicate that the save request is specifically for Kinesis In the future, each type of AWS input will likely require it's own request object and endpoint due to the fact that unique fields will probably be required for each. * AWSMessageType cleanup * Remove uneeded isFlowLog, isRaw methods. * Remove invalid AWSMessageType.Source.CLOUD_WATCH enum value. Messages are always read from Kinesis, and therefore the source is always Kinesis. Source is meant to differentiate messages from Kinesis and S3 for example. * Improve comments for AWSMessageType.Source enum class and method. * Remove typo * Generify the create AWS integration endpoint The naming, description, and comment now reflect that a generic AWS input is being created. * Return InputSummary response entity for AWS input creation request Also remove unneeded AWSResourceTest * More cleanup of healthCheck after input creation changes - Remove unneeded log_group field for health_check request. Resolves #108 - Add Kinesis stream name as a field in both raw and CloudWatch messages This change is lumped in with the other changes related to saving the input, since lots of healthCheck changes were already made there. This fixes some problems, so might as well have these improvements included with the review. * Clarify that AWSMetaCodec is a general AWS codec This class no longer erroneously extends AbstractKinesisCodec, which was only intended for Kinesis-specific codecs. * Use DateTime instead of long in KinesisLogEntry It turns out that Kinesis Record objects do have an arrival ime Instant timestamp. This is now being used instead of just using the date/time when the message was read by Graylog. * Add missing Guice annotation that were preventing server startup * Return JSON message field list in response from Health Check (#114) * Return JSON fields in healthCheck response There is still an issue with serializing the DateTime timestamp (serializing the deep object tree). This will be addressed separately. * Adapt unit test timestamp check to object value Health Check now returns a map of actual field values, so the test needs to check the JSON value. * Fix incorrect codec specified for Flow Logs, add tests * Fix incorrectly named field * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn * Advanced Kinesis Options * Cleanup * currently busted toggle * redo state handling * Remove artifact * removing commented code * removing unnecessary functions * pr feedback * Rebase conflicts * Fix several issues from PR review * "Final Review" layout (#83) * General Content * Review view layout complete * Cleanup * Update utils file name and export * Change LogFlow > Flow Log * default_values.js files * Route constants (#111) * Route constants * Fixing conflicts * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn Route constants Fixing conflicts * Correct import path * Changing route object * Adjust Kinesis setup wording * Remove the unneeded * Fixing up rebase artifacts from squash of irrelevant commits * AWS lockfile * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * Setting up necessary hooks * Removing input validation steps to reduce PR * Rebasing * Rename StyledForm * DEFAULT_VALUES * cleanup hook returns * Moving some reducers to state and breaking apart the rest * Remove functions from Input values * Steps function cleanup * Remove commented import * FormDataProvider and Context * Steps Context & provider * log and advancedoptions moved to context and provider * FormWrap functional component * renaming things * lint error * PR feedback * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * error and dirty states * Error on submit * Update validation to new state hooks * remove unneeded defaults * Output error messages * text key instead of password * Remove field validation * Update comment blocks (#137) * Update comment blocks * Remove minor typos * Update version to 3.1.0-beta.2-SNAPSHOT * Implement AWS codec selector with tests (#116) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Remove erroneous comment * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump package.json version to 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * Authorize and Streams API * All API calls except Save * reverting to hooks for api calls * Regions hook * Streams and Logs * streams useFetch * log sample * generic loading states * move regions * adding some documentation * Adding all the files * removing uneeded * cleanup * ApiRoutes * import proper path * Changing up useFetch * need the values... * lint * Removed unneeded inline cURL API documentation (#143) No need for this docs duplication. The System > Nodes > API Browser already documents them. * Use standard field names for AWS key and secret (#144) * Use standard naming for AWS secret and access key ids for Save Input Use these standard names for AWS key and secret from the Save Input AWS endpoint (From the AWSRequest interface). All other AWS API requests should be using these already. `aws_access_key_id` `aws_secret_access_key` * Use central constant for Region also * Fix incorrectly set Region property name * Fix access and secret key method names to match the AWSRequest interface * rename useFetch file * better var naming in kinesisstreams: * Update useFetch props * AWS transport selector with tests (#117) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Remove duplicate AWSTestingUtils class * Add missing permissions for sample AWS policy (#122) * Migrate AWS policy to autovalue object This avoids embedding JSON in a Java constant, which is error prone. * Add missing permissions Added: - kinesis:ListStreams - logs:DescribeLogGroups * Define JSON policy field order that matches AWS examples * Return the policy JSON string to the UI as string This is consistent with how this was done before. The UI can then format and display this to the user on the Available Services page. This is the policy that the user will create the AWS user for the AWS integration with. * Specify the correct AWS policy version * Change AWS Policy to a list * Add missing permissions Based on this [sample KCL code](aws/aws-sdk-java-v2#1214 (comment)) * Update version to 3.1.0-beta.2-SNAPSHOT * Remove uneeded import * Add Available Services API response error documentation * Throw InternalServerErrorException instead of JSON exception This is a bit cleaner from the API consumption side. A nice short error message is now returned instead of an obscure JSON error. * API Save Input (#145) * Save after Review and redirect to Inputs * Save functional * Adding match function to pluginmanifest * custom component * Update version to 3.1.0-beta.3-SNAPSHOT * Update version to 3.1.0-beta.3-SNAPSHOT * Aws kinesis newstream (#129) * Add create new Kinesis stream method * Add API call for createNewKinesisStream and KinesisNewStream classes * Update createNewKinesisStream * Update version to 3.1.0-beta.2-SNAPSHOT * Add error logging to createNewKinesisStream method * Add unit test for createNewKinesisStream method * Update error logging for createNewKinesisStream * Update testCreateNewKinesisStream * Remove shard count variable and set default value * Update testCreateNewKinesisStream * Code clean up * Update api comment blocks * Code clean up * Update error handling in createNewKinesisStream * Update unit test testCreateNewKinesisStream * Unload Confirmation Dialog (#148) * Playing with onunload * use ConfirmLeaveDialog common component * Question * Update version to 3.1.0-beta.3-SNAPSHOT * lint * cleanup * Don't alert on last step. Save call failed if desc wasn't available * Remove emotion and use styled-components (#157) * Remove emotion and use styled-components * import styled-components * Move title and description to FormWrap for consistent layout * Handle validation per-step * fix aws key h4x * styled validatedinput * enable health check and review forms * Starting Error Handling * Output API Errors * remove some more debugging * Validate API Key & Secret and output styled errors * Error message if stream doesn't contain any logs * Return a 4xx response for an unsuccessful health check * Save API Error Handling * Remove `success` flag from HealthCheckResponse Failure is now tracked by throwing a `BadRequestException` resulting in an `ApiError` json response * Throw `BadRequestException` if no Kinesis streams are found * Throw `BadRequestException` if no Kinesis CloudWatch groups are found * Error message styling * wrap up api errors and adding styles * rebasing and cleanup * trim input values * remove defaults * Create input with togglable masking * Include stream name in error * Port of existing KinesisTransport (#140) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Remove unused imports and fix formatting * Update version to 3.1.0-beta.3-SNAPSHOT * Merge branch 'aws' into aws-transport-kinesis and resolve conflicts # Conflicts: # src/main/java/org/graylog/integrations/aws/service/KinesisService.java * Upgrade to latest Kinesis Client version (#151) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Update Kinesis Client version * Add Kinesis client library v2 Consumer sample * Migrate KinesisConsumer to Kinesis Client Library v2 * Remove unused imports and fix formatting * Finish migration of Kinesis client to new version * Complete upgrade to Kinesis Client Library v2 * Bump KCL version * Migrate shard processor to its own class * Bump version to 3.1.0-beta.3-SNAPSHOT * Add batch size limit * Code cleanup * Remove unneeded throttle time limit reached shutdown * Remove kinesis_max_throttled_wait_ms save input field This field is no longer needed, since the new Kinesis Consumer appears to correctly handle longer throttling and pausing in `processRecords` without making the consumer unhealthy. * Temporarily remove Assume Role Arn auth This will be added back later. See #29 * Adjust logging levels * Remove sample class * Ignore unmapped properties due to removed max_throttled_wait field See Graylog2/graylog-plugin-integrations#156 * Update Throttling advanced option (#169) * Update Throttling advanced option * messages/ms * Remove throttle number * Update label * Shells/mocks for Kinesis Auto setup API endpoints (#167) * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Update version to 3.1.0-beta.4-SNAPSHOT * Code cleanup - Remove unneeded imports - Replace hyphens with underscores * Add `aws` to root URL path for consistency All other AWS routes are prefixed with this, so do the same here. * Change subscription path Change `add_subscription` route to `create_subscription` * Fix linter warning was introduced during merge conflict resolution. * Add RolePermissions for Kinesis stream (#163) * Add checkKinesisStreamStatus and setRolePermissions method * Add getNewRolePermissions and autoKinesisPermissionRequired method * Rename autoKinesisPermissionsRequired method and add error handling * Code clean up * Refactor autoSetup, split up creating a role and attaching a policy * Add error handling for creating a new role * Update autoKinesisPermissionsRequired logic and add comment block * Update version to 3.1.0-beta.4-SNAPSHOT * Delete checkKinesisStreamStatus method * Update error logging messages * Add temporary main class for testing, update error handling logic * Delete temporary main, update autoKinesisPermissionsRequired * Change exceptions thrown and code clean up * Update version to 3.1.0-rc.2-SNAPSHOT * Add addSubscriptionFilter to CloudwatchService (#160) * Add addSubscriptionFilter method to KinesisService * Move addSubscriptionFilter method to CloudWatchService * Code clean up, change exception errors * Bump version to 3.1.0-rc.3-SNAPSHOT * Kinesis Auto-setup (#186) * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump js-yaml from 3.13.0 to 3.13.1 Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 3.13.0 to 3.13.1. - [Release notes](https://github.com/nodeca/js-yaml/releases) - [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md) - [Commits](nodeca/js-yaml@3.13.0...3.13.1) Signed-off-by: dependabot[bot] <support@github.com> * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * [Security] Bump lodash from 4.17.11 to 4.17.14 (#131) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.11 to 4.17.14. **This update includes security fixes.** - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.11...4.17.14) Signed-off-by: dependabot-preview[bot] <support@dependabot.com> * Bump package.json version to 3.1.0-beta.2 * [graylog-plugin-integrations] prepare release 3.1.0-beta.2 * Bump package.json version to 3.1.0-beta.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Bump package.json version to 3.1.0-beta.3 * [graylog-plugin-integrations] prepare release 3.1.0-beta.3 * Bump package.json version to 3.1.0-beta.4-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Render Setup screen * cleanup * Allowing toggle to get from Streams to Setup * Toggle back to Streams & cleanup Group on Region change * Add Kinesis auto-setup routes * Add initial `create_stream` request * Define fetchs for auto Kinesis setup * Finish create_stream request * Add all auto-setup API requests * Specify actual region and request properties * Improve presentation for transition to automated Kinesis setup * Improve on-screen documentation for the automatic Kinesis setup * Bump package.json version to 3.1.0-rc.1 * [graylog-plugin-integrations] prepare release 3.1.0-rc.1 * Bump package.json version to 3.1.0-rc.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Update version to 3.1.0-rc.2-SNAPSHOT * Add KinesisSetupStep shell * Add KinesisSetupSteps container * Update createKinesisStream to return streamArn * Update createKinesisStream and testCreateNewKinesisStream * Add edge case error handling to createNewKinesisStream * Add preliminary code for KinesisFullSetup method * Use await for synchronous auto setup steps * Migrate to map for auto-setup state management * Render auto-setup progress to UI * Include success field and initial delay for testing * Delete KinesisFullSetup method * Migrate Kinesis setup to map to allow setup step iteration * Refactor autoKinesisPermissions parameters and return response * Rework setup logic to use individual state fields Individual state fields are needed for each component to update independently. * Add progress icons * Improve icon and step styles * Add auto-setup error handling * Refactor addSubscriptionFilter to return CreateLogSubscriptionResponse * Update api calls in KinesisFullSetup * Update createNewKinesisStream and createPolicies api call * Add flow control for setup steps * Clean up api methods * Add mocking for UI testing Also remove full setup, since the individual steps will be used. * Sync field mappings between UI and backend * Add auto-setup integration unit test * Add auto-setup test assertions * Rename `explanation` -> `result` * Display result to the user for each auto-setup step * Use unique role ane role policy names Since these are only needed in the auto-setup flow, we can automatically generate and assign them (as long as we provide them to the user in the response). Customizing these names is possible, and can be considered in the future. * Adjustments to get auto-setup working * Add support subscription retries to resolve IAM eventual consistency IAM sometimes takes 10+ seconds to initialize, so retrying allows the subscription attempt to be attempted multiple times, so that once IAM is available, then the subscription will be created successfully. * Cleanup retry logic * Add new page for auto-setup steps * Adjust wording * Add auto-setup TODO for adding warning * TODOs and cleanup * Formatting cleanup * Support ability to continue adding input after auto-setup * Bump package.json version to 3.1.0-rc.2 * [graylog-plugin-integrations] prepare release 3.1.0-rc.2 * Bump package.json version to 3.1.0-rc.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Disable Continue Setup button while auto-setup is in progress * Include warning that messages might take time to arrive in new stream * Require explicit agreement from user before starting auto-setup * Bump version to 3.1.0-rc.3-SNAPSHOT * Adjustments to auto-setup buttons, labels, and formatting * Skip CloudWatch control messages in Health Check Fixes Graylog2/graylog-plugin-integrations#178 * Add comment * Add support for the "flow_log_" prefix Closes Graylog2/graylog-plugin-integrations#120 * Fixing lint errors * Return mock responses * DEV COMMIT * Remove unnecessary Row and Col now that we're rendering a sidebar * sidebar styles * Add checkbox for adding Flow Log field prefix * Auto-Setup Sidebar * lint * Adjust wording * Correct label for AWS Message Type * Consolidate Kinesis payload decompression logic * Agree TOS in Modal * Return actual stream name in create stream mock * Modal Styled * Don't include the policy name in response, since it's inline * Remove DEV COMMIT changes * Update header description * Fix failing unit test * Mock responses * Add default setting for prefix field * Kinesis auto-setup Health Check redirect * health check clear sidebar * Add support for assuming an ARN role (#188) * Add support for assuming an ARN role Fixes Graylog2/graylog-plugin-integrations#29 * Add Assume Role ARN config field * Remove unneeded imports * Fix formatting * Countdown and Retry button * CheckAgain * Record when users agree to auto-setup (#194) * Add log message recording that a user agreed to auto-setup * Remove unneeded import * Fixing select default value (#199) * Adding pattern to Stream Name input (#200) * Adding pattern to Stream Name input * Don't submit if field has error * Add API request to get permissions (#196) * Add request to retireve Kinesis permissions Includes permissions for both the full setup and the auto-setup. Closes #174 * Alphabetize permissions, add comments, remove unused imports * Add unit test * Make auto-setup unit test pass when mocking enabled * Remove unneeded create log group/stream permissions * Minor code clean up (#201) * Minor code clean up * Fix typo * Make auto-setup unit test pass when mocking enabled * Bump version to 3.2.0-SNAPSHOT * Bump version to 3.2.0-SNAPSHOT * Starting testing * Linting Fix (#212) * Revert "frontend merge conflicts" This reverts commit fa5c2775a377e0585b1d771cec552f8ae0fd6de6, reversing changes made to 207fca0533051521e5cad58b39b43f15e5e44179. * remove test

* Initial commit * Add AWS SDK v2 dependency * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Add Kinesis SDK * General UI Skeleton (#75) * Simple Clickthrough without API * Clickable Skeleton All Steps * Feedback * Lint * AWS Cloudwatch backend API calls Add initial backend API calls for CloudWatch integration: getRegions, getLogGroups, getStreams, retrieveKinesisMessages, healthCheck * AWS Cloud Watch services and resources (#24) Adds beginnings of API endpoint and Kinesis/Cloudwatch services Includes a structure that we will continue to build from. * Rework organization of classes for unified structure The goal is to establish some structure that we can implement AWS API calls within. There's now one resource (for API calls), one service (for business logic), and one AWSClient (for all AWS/API SDK interactions). * Cleanup and add comments * Cleanup code around log message auto-detection * Aws cloudwatch client (#31) * Add CloudWatchService class * Add AWSConfigSettings class * Add UserCredentials class * Add temporary Main class * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Update AWSConfigSettings class * Update CloudWatchService class * Update temporary Main class * Fix inject error in AWSResource * Fix merge conflicts * Restructure Resources and Service classes * Add getRegion into AWSResource (#43) * Add getRegion into AWSResource * Update api paths * Increase max retry limit for stream get to 1000 Runaway looping Stopping at 100 is probably to small. * Use underscores instead of camel case for json names It is a general project standard to use underscores. * Add paging functionality to getLogGroupNames (#53) * Add paging functionality to getLogGroupNames * Add unit test for CloudWatch log groups * Add comments in log group name unit test * Code clean up and remove unneeded code * Revert unintended change from unit test commit * Add retrievelogs for Kinesis Healthcheck (#76) * Refactor getKinesisStreams in KinesisService * Add validCredentials method in AWSService class * Temporary Main class added to test putting records into a Kinesis stream * Add retrieveKinesisLogs in KinesisService * Update temporary Main class * Update retrieveKinesisLogs to loop through shard iterators * Update Main class * Update pom file * Update validateCredentials in AWSService class * Add createKinesisClient method in KinesisService class * Add testGetStreamCredentials and update testGetStreams * Healthcheck merge (#81) * empty commit to push branch * Add Kinesis Healthcheck (#45) * Improve organization for Flow Log message detection * Improve Flow Log test It now tests for a message with too many and too few spaces. * Add TODOs for healthCheck method logic * Add beginnings of Kinesis healthChecker This will pull a establish a Kinesis subscription and pull a single message from a Kinesis stream. * Fix failing unit test * Continue developing KinesisHealthCheck - Remove unneeded metric tracking - Remove extra parsing logic (this object should just hand back payload and not do any parsing) - Improve application name handling - Add comments * Add detection logic for raw vs. CloudWatch logs * Remove KinesisHealthCheck class The KinesisConsumer does not work well for the health check (designed for realtime processing, takes a long time to start, cannot detect empty stream, and is really hard to use in a quick API request). Now, we're planning to directly retrieve the messages using the Kinesis client. This is the most straight-forward thing We might revisit this later. * Fix JSON parsing of Kinesis CloudWatch subscription record Parse the record just as was done in the existing AWS plugin. The logic now includes autodetection of compressed/vs not compressed. Mock Kinesis CloudWatch subscription record included for testing purposes. * Add CloudWatch logs codec and tests from existing AWS plugin * Parse Flow Log message into object * Load appropriate codec during healthCheck process When the message type is detected, load the respective codec for that message type. * Parse message with appropriate codec Once the log message type is detected, then the codec is looked up. The message is then parsed with the codec. * Supply log group name with the response * Improve comments, logging, and error checking The log group name is now also included in the response. * Add Flow Log codec test * Use AutoValue for CloudWatchLogEntry class * Use AutoValue for all remaining CloudWatch value classes * Cleanup merge conflicts after rebasing * Specify constants for all JsonProperty annotations * Delete uneeded KinesisDTO All data will be stored in the input * Establish a base AWSRequest JSON class * Fix Guice injection error for KinesisService * Add sample cURL command for healthCheck method A similar cURL command will be used for other methods, so that it is clear how the UI will use them. * Remove unneeded Kinesis Client 1.x dependency * Add formatted message summary in the Health Check response * Cleanup formatting and TODOs * Minor cleanup after rebasing and merging * Fix failing unit tests * Integrate AWS Health Check with Kinesis record retrieval (#86) * Fix incorrect pass of AWS key instead of secret Also improve comments for fake message retrieval with TODOs. * Update and connect retrieveRecords * Add handleCompressedMessages method * Delete temporary main class * Update retrieveRecords to only return sample size * Update KinesisService for healthCheck to function properly * Add unit test for selecting random record * Add unit test for retrieveRecords * Add Available Services API call (#82) * Resolves #50: Add Available Services API call * Add a test * Add missing spaces, change Amazon > AWS * Backend code cleanup (#90) * Pass region and AWS credentials with AWS API requests (#92) * Require POST object containing region and credentials for all requests Specifically adds a POST body requirement for the getKinesisSteams and getLogGroupNames methods. * Use snake_case for paths * Update region api call (#110) * Migrate Regions request from a list to a full response object with total * Update Region API call to include label and value * Reformat code * Aws permissions check (#113) * Add AWSPermissions class and update AWSResource with permission checks * Rename and remove permissions in AWSPermissions * Register AWSPermissions in IntegrationsModule * Remove space between methods * Save AWS input API endpoint (#93) * First-pass structure for saving AWS input * Add more structure for general AWS input - Add a type enum to differentiate the various types of log messages that are possible. - Add metacodec that will eventually differentiate between the types of log messages. - Fill configuration values when saving the input. * Resolve merge conflicts after rebasing over latest aws branch * Consolidate log type detection and input type identification There's no longer a need to use two enums for this. Also added healthCheck tests covering all message types: flow log, raw Cloud Watch and raw Kinesis. * Clean up saveInput request parameters and handling * Fix invalid type specification that was preventing input save * Add unit test for saving input * Fix incorrectly specified arguments * Minor cleanup * Cleanup for PR review * Remove uneeded log statements * Make save AWS input path and description more specific * Indicate that the save request is specifically for Kinesis In the future, each type of AWS input will likely require it's own request object and endpoint due to the fact that unique fields will probably be required for each. * AWSMessageType cleanup * Remove uneeded isFlowLog, isRaw methods. * Remove invalid AWSMessageType.Source.CLOUD_WATCH enum value. Messages are always read from Kinesis, and therefore the source is always Kinesis. Source is meant to differentiate messages from Kinesis and S3 for example. * Improve comments for AWSMessageType.Source enum class and method. * Remove typo * Generify the create AWS integration endpoint The naming, description, and comment now reflect that a generic AWS input is being created. * Return InputSummary response entity for AWS input creation request Also remove unneeded AWSResourceTest * More cleanup of healthCheck after input creation changes - Remove unneeded log_group field for health_check request. Resolves #108 - Add Kinesis stream name as a field in both raw and CloudWatch messages This change is lumped in with the other changes related to saving the input, since lots of healthCheck changes were already made there. This fixes some problems, so might as well have these improvements included with the review. * Clarify that AWSMetaCodec is a general AWS codec This class no longer erroneously extends AbstractKinesisCodec, which was only intended for Kinesis-specific codecs. * Use DateTime instead of long in KinesisLogEntry It turns out that Kinesis Record objects do have an arrival ime Instant timestamp. This is now being used instead of just using the date/time when the message was read by Graylog. * Add missing Guice annotation that were preventing server startup * Return JSON message field list in response from Health Check (#114) * Return JSON fields in healthCheck response There is still an issue with serializing the DateTime timestamp (serializing the deep object tree). This will be addressed separately. * Adapt unit test timestamp check to object value Health Check now returns a map of actual field values, so the test needs to check the JSON value. * Fix incorrect codec specified for Flow Logs, add tests * Fix incorrectly named field * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn * Advanced Kinesis Options * Cleanup * currently busted toggle * redo state handling * Remove artifact * removing commented code * removing unnecessary functions * pr feedback * Rebase conflicts * Fix several issues from PR review * "Final Review" layout (#83) * General Content * Review view layout complete * Cleanup * Update utils file name and export * Change LogFlow > Flow Log * default_values.js files * Route constants (#111) * Route constants * Fixing conflicts * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn Route constants Fixing conflicts * Correct import path * Changing route object * Adjust Kinesis setup wording * Remove the unneeded * Fixing up rebase artifacts from squash of irrelevant commits * AWS lockfile * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * Setting up necessary hooks * Removing input validation steps to reduce PR * Rebasing * Rename StyledForm * DEFAULT_VALUES * cleanup hook returns * Moving some reducers to state and breaking apart the rest * Remove functions from Input values * Steps function cleanup * Remove commented import * FormDataProvider and Context * Steps Context & provider * log and advancedoptions moved to context and provider * FormWrap functional component * renaming things * lint error * PR feedback * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * error and dirty states * Error on submit * Update validation to new state hooks * remove unneeded defaults * Output error messages * text key instead of password * Remove field validation * Update comment blocks (#137) * Update comment blocks * Remove minor typos * Update version to 3.1.0-beta.2-SNAPSHOT * Implement AWS codec selector with tests (#116) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Remove erroneous comment * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump package.json version to 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * Authorize and Streams API * All API calls except Save * reverting to hooks for api calls * Regions hook * Streams and Logs * streams useFetch * log sample * generic loading states * move regions * adding some documentation * Adding all the files * removing uneeded * cleanup * ApiRoutes * import proper path * Changing up useFetch * need the values... * lint * Removed unneeded inline cURL API documentation (#143) No need for this docs duplication. The System > Nodes > API Browser already documents them. * Use standard field names for AWS key and secret (#144) * Use standard naming for AWS secret and access key ids for Save Input Use these standard names for AWS key and secret from the Save Input AWS endpoint (From the AWSRequest interface). All other AWS API requests should be using these already. `aws_access_key_id` `aws_secret_access_key` * Use central constant for Region also * Fix incorrectly set Region property name * Fix access and secret key method names to match the AWSRequest interface * rename useFetch file * better var naming in kinesisstreams: * Update useFetch props * AWS transport selector with tests (#117) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Remove duplicate AWSTestingUtils class * Add missing permissions for sample AWS policy (#122) * Migrate AWS policy to autovalue object This avoids embedding JSON in a Java constant, which is error prone. * Add missing permissions Added: - kinesis:ListStreams - logs:DescribeLogGroups * Define JSON policy field order that matches AWS examples * Return the policy JSON string to the UI as string This is consistent with how this was done before. The UI can then format and display this to the user on the Available Services page. This is the policy that the user will create the AWS user for the AWS integration with. * Specify the correct AWS policy version * Change AWS Policy to a list * Add missing permissions Based on this [sample KCL code](aws/aws-sdk-java-v2#1214 (comment)) * Update version to 3.1.0-beta.2-SNAPSHOT * Remove uneeded import * Add Available Services API response error documentation * Throw InternalServerErrorException instead of JSON exception This is a bit cleaner from the API consumption side. A nice short error message is now returned instead of an obscure JSON error. * API Save Input (#145) * Save after Review and redirect to Inputs * Save functional * Adding match function to pluginmanifest * custom component * Update version to 3.1.0-beta.3-SNAPSHOT * Update version to 3.1.0-beta.3-SNAPSHOT * Aws kinesis newstream (#129) * Add create new Kinesis stream method * Add API call for createNewKinesisStream and KinesisNewStream classes * Update createNewKinesisStream * Update version to 3.1.0-beta.2-SNAPSHOT * Add error logging to createNewKinesisStream method * Add unit test for createNewKinesisStream method * Update error logging for createNewKinesisStream * Update testCreateNewKinesisStream * Remove shard count variable and set default value * Update testCreateNewKinesisStream * Code clean up * Update api comment blocks * Code clean up * Update error handling in createNewKinesisStream * Update unit test testCreateNewKinesisStream * Unload Confirmation Dialog (#148) * Playing with onunload * use ConfirmLeaveDialog common component * Question * Update version to 3.1.0-beta.3-SNAPSHOT * lint * cleanup * Don't alert on last step. Save call failed if desc wasn't available * Remove emotion and use styled-components (#157) * Remove emotion and use styled-components * import styled-components * Move title and description to FormWrap for consistent layout * Handle validation per-step * fix aws key h4x * styled validatedinput * enable health check and review forms * Starting Error Handling * Output API Errors * remove some more debugging * Validate API Key & Secret and output styled errors * Error message if stream doesn't contain any logs * Return a 4xx response for an unsuccessful health check * Save API Error Handling * Remove `success` flag from HealthCheckResponse Failure is now tracked by throwing a `BadRequestException` resulting in an `ApiError` json response * Throw `BadRequestException` if no Kinesis streams are found * Throw `BadRequestException` if no Kinesis CloudWatch groups are found * Error message styling * wrap up api errors and adding styles * rebasing and cleanup * trim input values * remove defaults * Create input with togglable masking * Include stream name in error * Port of existing KinesisTransport (#140) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Remove unused imports and fix formatting * Update version to 3.1.0-beta.3-SNAPSHOT * Merge branch 'aws' into aws-transport-kinesis and resolve conflicts # Conflicts: # src/main/java/org/graylog/integrations/aws/service/KinesisService.java * Upgrade to latest Kinesis Client version (#151) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Update Kinesis Client version * Add Kinesis client library v2 Consumer sample * Migrate KinesisConsumer to Kinesis Client Library v2 * Remove unused imports and fix formatting * Finish migration of Kinesis client to new version * Complete upgrade to Kinesis Client Library v2 * Bump KCL version * Migrate shard processor to its own class * Bump version to 3.1.0-beta.3-SNAPSHOT * Add batch size limit * Code cleanup * Remove unneeded throttle time limit reached shutdown * Remove kinesis_max_throttled_wait_ms save input field This field is no longer needed, since the new Kinesis Consumer appears to correctly handle longer throttling and pausing in `processRecords` without making the consumer unhealthy. * Temporarily remove Assume Role Arn auth This will be added back later. See #29 * Adjust logging levels * Remove sample class * Ignore unmapped properties due to removed max_throttled_wait field See Graylog2/graylog-plugin-integrations#156 * Update Throttling advanced option (#169) * Update Throttling advanced option * messages/ms * Remove throttle number * Update label * Shells/mocks for Kinesis Auto setup API endpoints (#167) * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Update version to 3.1.0-beta.4-SNAPSHOT * Code cleanup - Remove unneeded imports - Replace hyphens with underscores * Add `aws` to root URL path for consistency All other AWS routes are prefixed with this, so do the same here. * Change subscription path Change `add_subscription` route to `create_subscription` * Fix linter warning was introduced during merge conflict resolution. * Add RolePermissions for Kinesis stream (#163) * Add checkKinesisStreamStatus and setRolePermissions method * Add getNewRolePermissions and autoKinesisPermissionRequired method * Rename autoKinesisPermissionsRequired method and add error handling * Code clean up * Refactor autoSetup, split up creating a role and attaching a policy * Add error handling for creating a new role * Update autoKinesisPermissionsRequired logic and add comment block * Update version to 3.1.0-beta.4-SNAPSHOT * Delete checkKinesisStreamStatus method * Update error logging messages * Add temporary main class for testing, update error handling logic * Delete temporary main, update autoKinesisPermissionsRequired * Change exceptions thrown and code clean up * Update version to 3.1.0-rc.2-SNAPSHOT * Add addSubscriptionFilter to CloudwatchService (#160) * Add addSubscriptionFilter method to KinesisService * Move addSubscriptionFilter method to CloudWatchService * Code clean up, change exception errors * Bump version to 3.1.0-rc.3-SNAPSHOT * Kinesis Auto-setup (#186) * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump js-yaml from 3.13.0 to 3.13.1 Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 3.13.0 to 3.13.1. - [Release notes](https://github.com/nodeca/js-yaml/releases) - [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md) - [Commits](nodeca/js-yaml@3.13.0...3.13.1) Signed-off-by: dependabot[bot] <support@github.com> * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * [Security] Bump lodash from 4.17.11 to 4.17.14 (#131) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.11 to 4.17.14. **This update includes security fixes.** - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.11...4.17.14) Signed-off-by: dependabot-preview[bot] <support@dependabot.com> * Bump package.json version to 3.1.0-beta.2 * [graylog-plugin-integrations] prepare release 3.1.0-beta.2 * Bump package.json version to 3.1.0-beta.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Bump package.json version to 3.1.0-beta.3 * [graylog-plugin-integrations] prepare release 3.1.0-beta.3 * Bump package.json version to 3.1.0-beta.4-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Render Setup screen * cleanup * Allowing toggle to get from Streams to Setup * Toggle back to Streams & cleanup Group on Region change * Add Kinesis auto-setup routes * Add initial `create_stream` request * Define fetchs for auto Kinesis setup * Finish create_stream request * Add all auto-setup API requests * Specify actual region and request properties * Improve presentation for transition to automated Kinesis setup * Improve on-screen documentation for the automatic Kinesis setup * Bump package.json version to 3.1.0-rc.1 * [graylog-plugin-integrations] prepare release 3.1.0-rc.1 * Bump package.json version to 3.1.0-rc.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Update version to 3.1.0-rc.2-SNAPSHOT * Add KinesisSetupStep shell * Add KinesisSetupSteps container * Update createKinesisStream to return streamArn * Update createKinesisStream and testCreateNewKinesisStream * Add edge case error handling to createNewKinesisStream * Add preliminary code for KinesisFullSetup method * Use await for synchronous auto setup steps * Migrate to map for auto-setup state management * Render auto-setup progress to UI * Include success field and initial delay for testing * Delete KinesisFullSetup method * Migrate Kinesis setup to map to allow setup step iteration * Refactor autoKinesisPermissions parameters and return response * Rework setup logic to use individual state fields Individual state fields are needed for each component to update independently. * Add progress icons * Improve icon and step styles * Add auto-setup error handling * Refactor addSubscriptionFilter to return CreateLogSubscriptionResponse * Update api calls in KinesisFullSetup * Update createNewKinesisStream and createPolicies api call * Add flow control for setup steps * Clean up api methods * Add mocking for UI testing Also remove full setup, since the individual steps will be used. * Sync field mappings between UI and backend * Add auto-setup integration unit test * Add auto-setup test assertions * Rename `explanation` -> `result` * Display result to the user for each auto-setup step * Use unique role ane role policy names Since these are only needed in the auto-setup flow, we can automatically generate and assign them (as long as we provide them to the user in the response). Customizing these names is possible, and can be considered in the future. * Adjustments to get auto-setup working * Add support subscription retries to resolve IAM eventual consistency IAM sometimes takes 10+ seconds to initialize, so retrying allows the subscription attempt to be attempted multiple times, so that once IAM is available, then the subscription will be created successfully. * Cleanup retry logic * Add new page for auto-setup steps * Adjust wording * Add auto-setup TODO for adding warning * TODOs and cleanup * Formatting cleanup * Support ability to continue adding input after auto-setup * Bump package.json version to 3.1.0-rc.2 * [graylog-plugin-integrations] prepare release 3.1.0-rc.2 * Bump package.json version to 3.1.0-rc.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Disable Continue Setup button while auto-setup is in progress * Include warning that messages might take time to arrive in new stream * Require explicit agreement from user before starting auto-setup * Bump version to 3.1.0-rc.3-SNAPSHOT * Adjustments to auto-setup buttons, labels, and formatting * Skip CloudWatch control messages in Health Check Fixes Graylog2/graylog-plugin-integrations#178 * Add comment * Add support for the "flow_log_" prefix Closes Graylog2/graylog-plugin-integrations#120 * Fixing lint errors * Return mock responses * DEV COMMIT * Remove unnecessary Row and Col now that we're rendering a sidebar * sidebar styles * Add checkbox for adding Flow Log field prefix * Auto-Setup Sidebar * lint * Adjust wording * Correct label for AWS Message Type * Consolidate Kinesis payload decompression logic * Agree TOS in Modal * Return actual stream name in create stream mock * Modal Styled * Don't include the policy name in response, since it's inline * Remove DEV COMMIT changes * Update header description * Fix failing unit test * Mock responses * Add default setting for prefix field * Add support for assuming an ARN role (#188) * Add support for assuming an ARN role Fixes Graylog2/graylog-plugin-integrations#29 * Add Assume Role ARN config field * Remove unneeded imports * Fix formatting * Record when users agree to auto-setup (#194) * Add log message recording that a user agreed to auto-setup * Remove unneeded import * Fixing select default value (#199) * Adding pattern to Stream Name input (#200) * Adding pattern to Stream Name input * Don't submit if field has error * Add API request to get permissions (#196) * Add request to retireve Kinesis permissions Includes permissions for both the full setup and the auto-setup. Closes #174 * Alphabetize permissions, add comments, remove unused imports * Add unit test * Make auto-setup unit test pass when mocking enabled * Remove unneeded create log group/stream permissions * Minor code clean up (#201) * Minor code clean up * Fix typo * Proposed wording changes We'll pick this up again in Hamburg * Bump version to 3.2.0-SNAPSHOT * Wording adjustments * Change "Setup" -> "Set Up" * Fix lint errors * More wording changes * Removing empty doc link

* Initial commit * Add AWS SDK v2 dependency * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Add Kinesis SDK * General UI Skeleton (#75) * Simple Clickthrough without API * Clickable Skeleton All Steps * Feedback * Lint * AWS Cloudwatch backend API calls Add initial backend API calls for CloudWatch integration: getRegions, getLogGroups, getStreams, retrieveKinesisMessages, healthCheck * AWS Cloud Watch services and resources (#24) Adds beginnings of API endpoint and Kinesis/Cloudwatch services Includes a structure that we will continue to build from. * Rework organization of classes for unified structure The goal is to establish some structure that we can implement AWS API calls within. There's now one resource (for API calls), one service (for business logic), and one AWSClient (for all AWS/API SDK interactions). * Cleanup and add comments * Cleanup code around log message auto-detection * Aws cloudwatch client (#31) * Add CloudWatchService class * Add AWSConfigSettings class * Add UserCredentials class * Add temporary Main class * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Update AWSConfigSettings class * Update CloudWatchService class * Update temporary Main class * Fix inject error in AWSResource * Fix merge conflicts * Restructure Resources and Service classes * Add getRegion into AWSResource (#43) * Add getRegion into AWSResource * Update api paths * Increase max retry limit for stream get to 1000 Runaway looping Stopping at 100 is probably to small. * Use underscores instead of camel case for json names It is a general project standard to use underscores. * Add paging functionality to getLogGroupNames (#53) * Add paging functionality to getLogGroupNames * Add unit test for CloudWatch log groups * Add comments in log group name unit test * Code clean up and remove unneeded code * Revert unintended change from unit test commit * Add retrievelogs for Kinesis Healthcheck (#76) * Refactor getKinesisStreams in KinesisService * Add validCredentials method in AWSService class * Temporary Main class added to test putting records into a Kinesis stream * Add retrieveKinesisLogs in KinesisService * Update temporary Main class * Update retrieveKinesisLogs to loop through shard iterators * Update Main class * Update pom file * Update validateCredentials in AWSService class * Add createKinesisClient method in KinesisService class * Add testGetStreamCredentials and update testGetStreams * Healthcheck merge (#81) * empty commit to push branch * Add Kinesis Healthcheck (#45) * Improve organization for Flow Log message detection * Improve Flow Log test It now tests for a message with too many and too few spaces. * Add TODOs for healthCheck method logic * Add beginnings of Kinesis healthChecker This will pull a establish a Kinesis subscription and pull a single message from a Kinesis stream. * Fix failing unit test * Continue developing KinesisHealthCheck - Remove unneeded metric tracking - Remove extra parsing logic (this object should just hand back payload and not do any parsing) - Improve application name handling - Add comments * Add detection logic for raw vs. CloudWatch logs * Remove KinesisHealthCheck class The KinesisConsumer does not work well for the health check (designed for realtime processing, takes a long time to start, cannot detect empty stream, and is really hard to use in a quick API request). Now, we're planning to directly retrieve the messages using the Kinesis client. This is the most straight-forward thing We might revisit this later. * Fix JSON parsing of Kinesis CloudWatch subscription record Parse the record just as was done in the existing AWS plugin. The logic now includes autodetection of compressed/vs not compressed. Mock Kinesis CloudWatch subscription record included for testing purposes. * Add CloudWatch logs codec and tests from existing AWS plugin * Parse Flow Log message into object * Load appropriate codec during healthCheck process When the message type is detected, load the respective codec for that message type. * Parse message with appropriate codec Once the log message type is detected, then the codec is looked up. The message is then parsed with the codec. * Supply log group name with the response * Improve comments, logging, and error checking The log group name is now also included in the response. * Add Flow Log codec test * Use AutoValue for CloudWatchLogEntry class * Use AutoValue for all remaining CloudWatch value classes * Cleanup merge conflicts after rebasing * Specify constants for all JsonProperty annotations * Delete uneeded KinesisDTO All data will be stored in the input * Establish a base AWSRequest JSON class * Fix Guice injection error for KinesisService * Add sample cURL command for healthCheck method A similar cURL command will be used for other methods, so that it is clear how the UI will use them. * Remove unneeded Kinesis Client 1.x dependency * Add formatted message summary in the Health Check response * Cleanup formatting and TODOs * Minor cleanup after rebasing and merging * Fix failing unit tests * Integrate AWS Health Check with Kinesis record retrieval (#86) * Fix incorrect pass of AWS key instead of secret Also improve comments for fake message retrieval with TODOs. * Update and connect retrieveRecords * Add handleCompressedMessages method * Delete temporary main class * Update retrieveRecords to only return sample size * Update KinesisService for healthCheck to function properly * Add unit test for selecting random record * Add unit test for retrieveRecords * Add Available Services API call (#82) * Resolves #50: Add Available Services API call * Add a test * Add missing spaces, change Amazon > AWS * Backend code cleanup (#90) * Pass region and AWS credentials with AWS API requests (#92) * Require POST object containing region and credentials for all requests Specifically adds a POST body requirement for the getKinesisSteams and getLogGroupNames methods. * Use snake_case for paths * Update region api call (#110) * Migrate Regions request from a list to a full response object with total * Update Region API call to include label and value * Reformat code * Aws permissions check (#113) * Add AWSPermissions class and update AWSResource with permission checks * Rename and remove permissions in AWSPermissions * Register AWSPermissions in IntegrationsModule * Remove space between methods * Save AWS input API endpoint (#93) * First-pass structure for saving AWS input * Add more structure for general AWS input - Add a type enum to differentiate the various types of log messages that are possible. - Add metacodec that will eventually differentiate between the types of log messages. - Fill configuration values when saving the input. * Resolve merge conflicts after rebasing over latest aws branch * Consolidate log type detection and input type identification There's no longer a need to use two enums for this. Also added healthCheck tests covering all message types: flow log, raw Cloud Watch and raw Kinesis. * Clean up saveInput request parameters and handling * Fix invalid type specification that was preventing input save * Add unit test for saving input * Fix incorrectly specified arguments * Minor cleanup * Cleanup for PR review * Remove uneeded log statements * Make save AWS input path and description more specific * Indicate that the save request is specifically for Kinesis In the future, each type of AWS input will likely require it's own request object and endpoint due to the fact that unique fields will probably be required for each. * AWSMessageType cleanup * Remove uneeded isFlowLog, isRaw methods. * Remove invalid AWSMessageType.Source.CLOUD_WATCH enum value. Messages are always read from Kinesis, and therefore the source is always Kinesis. Source is meant to differentiate messages from Kinesis and S3 for example. * Improve comments for AWSMessageType.Source enum class and method. * Remove typo * Generify the create AWS integration endpoint The naming, description, and comment now reflect that a generic AWS input is being created. * Return InputSummary response entity for AWS input creation request Also remove unneeded AWSResourceTest * More cleanup of healthCheck after input creation changes - Remove unneeded log_group field for health_check request. Resolves #108 - Add Kinesis stream name as a field in both raw and CloudWatch messages This change is lumped in with the other changes related to saving the input, since lots of healthCheck changes were already made there. This fixes some problems, so might as well have these improvements included with the review. * Clarify that AWSMetaCodec is a general AWS codec This class no longer erroneously extends AbstractKinesisCodec, which was only intended for Kinesis-specific codecs. * Use DateTime instead of long in KinesisLogEntry It turns out that Kinesis Record objects do have an arrival ime Instant timestamp. This is now being used instead of just using the date/time when the message was read by Graylog. * Add missing Guice annotation that were preventing server startup * Return JSON message field list in response from Health Check (#114) * Return JSON fields in healthCheck response There is still an issue with serializing the DateTime timestamp (serializing the deep object tree). This will be addressed separately. * Adapt unit test timestamp check to object value Health Check now returns a map of actual field values, so the test needs to check the JSON value. * Fix incorrect codec specified for Flow Logs, add tests * Fix incorrectly named field * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn * Advanced Kinesis Options * Cleanup * currently busted toggle * redo state handling * Remove artifact * removing commented code * removing unnecessary functions * pr feedback * Rebase conflicts * Fix several issues from PR review * "Final Review" layout (#83) * General Content * Review view layout complete * Cleanup * Update utils file name and export * Change LogFlow > Flow Log * default_values.js files * Route constants (#111) * Route constants * Fixing conflicts * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn Route constants Fixing conflicts * Correct import path * Changing route object * Adjust Kinesis setup wording * Remove the unneeded * Fixing up rebase artifacts from squash of irrelevant commits * AWS lockfile * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * Setting up necessary hooks * Removing input validation steps to reduce PR * Rebasing * Rename StyledForm * DEFAULT_VALUES * cleanup hook returns * Moving some reducers to state and breaking apart the rest * Remove functions from Input values * Steps function cleanup * Remove commented import * FormDataProvider and Context * Steps Context & provider * log and advancedoptions moved to context and provider * FormWrap functional component * renaming things * lint error * PR feedback * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * error and dirty states * Error on submit * Update validation to new state hooks * remove unneeded defaults * Output error messages * text key instead of password * Remove field validation * Update comment blocks (#137) * Update comment blocks * Remove minor typos * Update version to 3.1.0-beta.2-SNAPSHOT * Implement AWS codec selector with tests (#116) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Remove erroneous comment * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump package.json version to 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * Authorize and Streams API * All API calls except Save * reverting to hooks for api calls * Regions hook * Streams and Logs * streams useFetch * log sample * generic loading states * move regions * adding some documentation * Adding all the files * removing uneeded * cleanup * ApiRoutes * import proper path * Changing up useFetch * need the values... * lint * Removed unneeded inline cURL API documentation (#143) No need for this docs duplication. The System > Nodes > API Browser already documents them. * Use standard field names for AWS key and secret (#144) * Use standard naming for AWS secret and access key ids for Save Input Use these standard names for AWS key and secret from the Save Input AWS endpoint (From the AWSRequest interface). All other AWS API requests should be using these already. `aws_access_key_id` `aws_secret_access_key` * Use central constant for Region also * Fix incorrectly set Region property name * Fix access and secret key method names to match the AWSRequest interface * rename useFetch file * better var naming in kinesisstreams: * Update useFetch props * AWS transport selector with tests (#117) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Remove duplicate AWSTestingUtils class * Add missing permissions for sample AWS policy (#122) * Migrate AWS policy to autovalue object This avoids embedding JSON in a Java constant, which is error prone. * Add missing permissions Added: - kinesis:ListStreams - logs:DescribeLogGroups * Define JSON policy field order that matches AWS examples * Return the policy JSON string to the UI as string This is consistent with how this was done before. The UI can then format and display this to the user on the Available Services page. This is the policy that the user will create the AWS user for the AWS integration with. * Specify the correct AWS policy version * Change AWS Policy to a list * Add missing permissions Based on this [sample KCL code](aws/aws-sdk-java-v2#1214 (comment)) * Update version to 3.1.0-beta.2-SNAPSHOT * Remove uneeded import * Add Available Services API response error documentation * Throw InternalServerErrorException instead of JSON exception This is a bit cleaner from the API consumption side. A nice short error message is now returned instead of an obscure JSON error. * API Save Input (#145) * Save after Review and redirect to Inputs * Save functional * Adding match function to pluginmanifest * custom component * Update version to 3.1.0-beta.3-SNAPSHOT * Update version to 3.1.0-beta.3-SNAPSHOT * Aws kinesis newstream (#129) * Add create new Kinesis stream method * Add API call for createNewKinesisStream and KinesisNewStream classes * Update createNewKinesisStream * Update version to 3.1.0-beta.2-SNAPSHOT * Add error logging to createNewKinesisStream method * Add unit test for createNewKinesisStream method * Update error logging for createNewKinesisStream * Update testCreateNewKinesisStream * Remove shard count variable and set default value * Update testCreateNewKinesisStream * Code clean up * Update api comment blocks * Code clean up * Update error handling in createNewKinesisStream * Update unit test testCreateNewKinesisStream * Unload Confirmation Dialog (#148) * Playing with onunload * use ConfirmLeaveDialog common component * Question * Update version to 3.1.0-beta.3-SNAPSHOT * lint * cleanup * Don't alert on last step. Save call failed if desc wasn't available * Remove emotion and use styled-components (#157) * Remove emotion and use styled-components * import styled-components * Move title and description to FormWrap for consistent layout * Handle validation per-step * fix aws key h4x * styled validatedinput * enable health check and review forms * Starting Error Handling * Output API Errors * remove some more debugging * Validate API Key & Secret and output styled errors * Error message if stream doesn't contain any logs * Return a 4xx response for an unsuccessful health check * Save API Error Handling * Remove `success` flag from HealthCheckResponse Failure is now tracked by throwing a `BadRequestException` resulting in an `ApiError` json response * Throw `BadRequestException` if no Kinesis streams are found * Throw `BadRequestException` if no Kinesis CloudWatch groups are found * Error message styling * wrap up api errors and adding styles * rebasing and cleanup * trim input values * remove defaults * Create input with togglable masking * Include stream name in error * Port of existing KinesisTransport (#140) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Remove unused imports and fix formatting * Update version to 3.1.0-beta.3-SNAPSHOT * Merge branch 'aws' into aws-transport-kinesis and resolve conflicts # Conflicts: # src/main/java/org/graylog/integrations/aws/service/KinesisService.java * Upgrade to latest Kinesis Client version (#151) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Update Kinesis Client version * Add Kinesis client library v2 Consumer sample * Migrate KinesisConsumer to Kinesis Client Library v2 * Remove unused imports and fix formatting * Finish migration of Kinesis client to new version * Complete upgrade to Kinesis Client Library v2 * Bump KCL version * Migrate shard processor to its own class * Bump version to 3.1.0-beta.3-SNAPSHOT * Add batch size limit * Code cleanup * Remove unneeded throttle time limit reached shutdown * Remove kinesis_max_throttled_wait_ms save input field This field is no longer needed, since the new Kinesis Consumer appears to correctly handle longer throttling and pausing in `processRecords` without making the consumer unhealthy. * Temporarily remove Assume Role Arn auth This will be added back later. See #29 * Adjust logging levels * Remove sample class * Ignore unmapped properties due to removed max_throttled_wait field See Graylog2/graylog-plugin-integrations#156 * Update Throttling advanced option (#169) * Update Throttling advanced option * messages/ms * Remove throttle number * Update label * Shells/mocks for Kinesis Auto setup API endpoints (#167) * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Update version to 3.1.0-beta.4-SNAPSHOT * Code cleanup - Remove unneeded imports - Replace hyphens with underscores * Add `aws` to root URL path for consistency All other AWS routes are prefixed with this, so do the same here. * Change subscription path Change `add_subscription` route to `create_subscription` * Fix linter warning was introduced during merge conflict resolution. * Add RolePermissions for Kinesis stream (#163) * Add checkKinesisStreamStatus and setRolePermissions method * Add getNewRolePermissions and autoKinesisPermissionRequired method * Rename autoKinesisPermissionsRequired method and add error handling * Code clean up * Refactor autoSetup, split up creating a role and attaching a policy * Add error handling for creating a new role * Update autoKinesisPermissionsRequired logic and add comment block * Update version to 3.1.0-beta.4-SNAPSHOT * Delete checkKinesisStreamStatus method * Update error logging messages * Add temporary main class for testing, update error handling logic * Delete temporary main, update autoKinesisPermissionsRequired * Change exceptions thrown and code clean up * Update version to 3.1.0-rc.2-SNAPSHOT * Add addSubscriptionFilter to CloudwatchService (#160) * Add addSubscriptionFilter method to KinesisService * Move addSubscriptionFilter method to CloudWatchService * Code clean up, change exception errors * Bump version to 3.1.0-rc.3-SNAPSHOT * Kinesis Auto-setup (#186) * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump js-yaml from 3.13.0 to 3.13.1 Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 3.13.0 to 3.13.1. - [Release notes](https://github.com/nodeca/js-yaml/releases) - [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md) - [Commits](nodeca/js-yaml@3.13.0...3.13.1) Signed-off-by: dependabot[bot] <support@github.com> * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * [Security] Bump lodash from 4.17.11 to 4.17.14 (#131) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.11 to 4.17.14. **This update includes security fixes.** - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.11...4.17.14) Signed-off-by: dependabot-preview[bot] <support@dependabot.com> * Bump package.json version to 3.1.0-beta.2 * [graylog-plugin-integrations] prepare release 3.1.0-beta.2 * Bump package.json version to 3.1.0-beta.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Bump package.json version to 3.1.0-beta.3 * [graylog-plugin-integrations] prepare release 3.1.0-beta.3 * Bump package.json version to 3.1.0-beta.4-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Render Setup screen * cleanup * Allowing toggle to get from Streams to Setup * Toggle back to Streams & cleanup Group on Region change * Add Kinesis auto-setup routes * Add initial `create_stream` request * Define fetchs for auto Kinesis setup * Finish create_stream request * Add all auto-setup API requests * Specify actual region and request properties * Improve presentation for transition to automated Kinesis setup * Improve on-screen documentation for the automatic Kinesis setup * Bump package.json version to 3.1.0-rc.1 * [graylog-plugin-integrations] prepare release 3.1.0-rc.1 * Bump package.json version to 3.1.0-rc.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Update version to 3.1.0-rc.2-SNAPSHOT * Add KinesisSetupStep shell * Add KinesisSetupSteps container * Update createKinesisStream to return streamArn * Update createKinesisStream and testCreateNewKinesisStream * Add edge case error handling to createNewKinesisStream * Add preliminary code for KinesisFullSetup method * Use await for synchronous auto setup steps * Migrate to map for auto-setup state management * Render auto-setup progress to UI * Include success field and initial delay for testing * Delete KinesisFullSetup method * Migrate Kinesis setup to map to allow setup step iteration * Refactor autoKinesisPermissions parameters and return response * Rework setup logic to use individual state fields Individual state fields are needed for each component to update independently. * Add progress icons * Improve icon and step styles * Add auto-setup error handling * Refactor addSubscriptionFilter to return CreateLogSubscriptionResponse * Update api calls in KinesisFullSetup * Update createNewKinesisStream and createPolicies api call * Add flow control for setup steps * Clean up api methods * Add mocking for UI testing Also remove full setup, since the individual steps will be used. * Sync field mappings between UI and backend * Add auto-setup integration unit test * Add auto-setup test assertions * Rename `explanation` -> `result` * Display result to the user for each auto-setup step * Use unique role ane role policy names Since these are only needed in the auto-setup flow, we can automatically generate and assign them (as long as we provide them to the user in the response). Customizing these names is possible, and can be considered in the future. * Adjustments to get auto-setup working * Add support subscription retries to resolve IAM eventual consistency IAM sometimes takes 10+ seconds to initialize, so retrying allows the subscription attempt to be attempted multiple times, so that once IAM is available, then the subscription will be created successfully. * Cleanup retry logic * Add new page for auto-setup steps * Adjust wording * Add auto-setup TODO for adding warning * TODOs and cleanup * Formatting cleanup * Support ability to continue adding input after auto-setup * Bump package.json version to 3.1.0-rc.2 * [graylog-plugin-integrations] prepare release 3.1.0-rc.2 * Bump package.json version to 3.1.0-rc.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Disable Continue Setup button while auto-setup is in progress * Include warning that messages might take time to arrive in new stream * Require explicit agreement from user before starting auto-setup * Bump version to 3.1.0-rc.3-SNAPSHOT * Adjustments to auto-setup buttons, labels, and formatting * Skip CloudWatch control messages in Health Check Fixes Graylog2/graylog-plugin-integrations#178 * Add comment * Add support for the "flow_log_" prefix Closes Graylog2/graylog-plugin-integrations#120 * Fixing lint errors * Return mock responses * DEV COMMIT * Remove unnecessary Row and Col now that we're rendering a sidebar * sidebar styles * Add checkbox for adding Flow Log field prefix * Auto-Setup Sidebar * lint * Adjust wording * Correct label for AWS Message Type * Consolidate Kinesis payload decompression logic * Agree TOS in Modal * Return actual stream name in create stream mock * Modal Styled * Don't include the policy name in response, since it's inline * Remove DEV COMMIT changes * Update header description * Fix failing unit test * Mock responses * Add default setting for prefix field * Kinesis auto-setup Health Check redirect * health check clear sidebar * Add support for assuming an ARN role (#188) * Add support for assuming an ARN role Fixes Graylog2/graylog-plugin-integrations#29 * Add Assume Role ARN config field * Remove unneeded imports * Fix formatting * Countdown and Retry button * CheckAgain * Record when users agree to auto-setup (#194) * Add log message recording that a user agreed to auto-setup * Remove unneeded import * Fixing select default value (#199) * Adding pattern to Stream Name input (#200) * Adding pattern to Stream Name input * Don't submit if field has error * Add API request to get permissions (#196) * Add request to retireve Kinesis permissions Includes permissions for both the full setup and the auto-setup. Closes #174 * Alphabetize permissions, add comments, remove unused imports * Add unit test * Make auto-setup unit test pass when mocking enabled * Remove unneeded create log group/stream permissions * Minor code clean up (#201) * Minor code clean up * Fix typo * Make auto-setup unit test pass when mocking enabled * Bump version to 3.2.0-SNAPSHOT * Bump version to 3.2.0-SNAPSHOT * Starting testing * Linting Fix (#212) * Revert "frontend merge conflicts" This reverts commit fa5c2775a377e0585b1d771cec552f8ae0fd6de6, reversing changes made to 207fca0533051521e5cad58b39b43f15e5e44179. * remove test * Use Moment * cleanup * Helps if countdown counts down * 120 seconds

* Initial commit * Add AWS SDK v2 dependency * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Add Kinesis SDK * General UI Skeleton (#75) * Simple Clickthrough without API * Clickable Skeleton All Steps * Feedback * Lint * AWS Cloudwatch backend API calls Add initial backend API calls for CloudWatch integration: getRegions, getLogGroups, getStreams, retrieveKinesisMessages, healthCheck * AWS Cloud Watch services and resources (#24) Adds beginnings of API endpoint and Kinesis/Cloudwatch services Includes a structure that we will continue to build from. * Rework organization of classes for unified structure The goal is to establish some structure that we can implement AWS API calls within. There's now one resource (for API calls), one service (for business logic), and one AWSClient (for all AWS/API SDK interactions). * Cleanup and add comments * Cleanup code around log message auto-detection * Aws cloudwatch client (#31) * Add CloudWatchService class * Add AWSConfigSettings class * Add UserCredentials class * Add temporary Main class * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Update AWSConfigSettings class * Update CloudWatchService class * Update temporary Main class * Fix inject error in AWSResource * Fix merge conflicts * Restructure Resources and Service classes * Add getRegion into AWSResource (#43) * Add getRegion into AWSResource * Update api paths * Increase max retry limit for stream get to 1000 Runaway looping Stopping at 100 is probably to small. * Use underscores instead of camel case for json names It is a general project standard to use underscores. * Add paging functionality to getLogGroupNames (#53) * Add paging functionality to getLogGroupNames * Add unit test for CloudWatch log groups * Add comments in log group name unit test * Code clean up and remove unneeded code * Revert unintended change from unit test commit * Add retrievelogs for Kinesis Healthcheck (#76) * Refactor getKinesisStreams in KinesisService * Add validCredentials method in AWSService class * Temporary Main class added to test putting records into a Kinesis stream * Add retrieveKinesisLogs in KinesisService * Update temporary Main class * Update retrieveKinesisLogs to loop through shard iterators * Update Main class * Update pom file * Update validateCredentials in AWSService class * Add createKinesisClient method in KinesisService class * Add testGetStreamCredentials and update testGetStreams * Healthcheck merge (#81) * empty commit to push branch * Add Kinesis Healthcheck (#45) * Improve organization for Flow Log message detection * Improve Flow Log test It now tests for a message with too many and too few spaces. * Add TODOs for healthCheck method logic * Add beginnings of Kinesis healthChecker This will pull a establish a Kinesis subscription and pull a single message from a Kinesis stream. * Fix failing unit test * Continue developing KinesisHealthCheck - Remove unneeded metric tracking - Remove extra parsing logic (this object should just hand back payload and not do any parsing) - Improve application name handling - Add comments * Add detection logic for raw vs. CloudWatch logs * Remove KinesisHealthCheck class The KinesisConsumer does not work well for the health check (designed for realtime processing, takes a long time to start, cannot detect empty stream, and is really hard to use in a quick API request). Now, we're planning to directly retrieve the messages using the Kinesis client. This is the most straight-forward thing We might revisit this later. * Fix JSON parsing of Kinesis CloudWatch subscription record Parse the record just as was done in the existing AWS plugin. The logic now includes autodetection of compressed/vs not compressed. Mock Kinesis CloudWatch subscription record included for testing purposes. * Add CloudWatch logs codec and tests from existing AWS plugin * Parse Flow Log message into object * Load appropriate codec during healthCheck process When the message type is detected, load the respective codec for that message type. * Parse message with appropriate codec Once the log message type is detected, then the codec is looked up. The message is then parsed with the codec. * Supply log group name with the response * Improve comments, logging, and error checking The log group name is now also included in the response. * Add Flow Log codec test * Use AutoValue for CloudWatchLogEntry class * Use AutoValue for all remaining CloudWatch value classes * Cleanup merge conflicts after rebasing * Specify constants for all JsonProperty annotations * Delete uneeded KinesisDTO All data will be stored in the input * Establish a base AWSRequest JSON class * Fix Guice injection error for KinesisService * Add sample cURL command for healthCheck method A similar cURL command will be used for other methods, so that it is clear how the UI will use them. * Remove unneeded Kinesis Client 1.x dependency * Add formatted message summary in the Health Check response * Cleanup formatting and TODOs * Minor cleanup after rebasing and merging * Fix failing unit tests * Integrate AWS Health Check with Kinesis record retrieval (#86) * Fix incorrect pass of AWS key instead of secret Also improve comments for fake message retrieval with TODOs. * Update and connect retrieveRecords * Add handleCompressedMessages method * Delete temporary main class * Update retrieveRecords to only return sample size * Update KinesisService for healthCheck to function properly * Add unit test for selecting random record * Add unit test for retrieveRecords * Add Available Services API call (#82) * Resolves #50: Add Available Services API call * Add a test * Add missing spaces, change Amazon > AWS * Backend code cleanup (#90) * Pass region and AWS credentials with AWS API requests (#92) * Require POST object containing region and credentials for all requests Specifically adds a POST body requirement for the getKinesisSteams and getLogGroupNames methods. * Use snake_case for paths * Update region api call (#110) * Migrate Regions request from a list to a full response object with total * Update Region API call to include label and value * Reformat code * Aws permissions check (#113) * Add AWSPermissions class and update AWSResource with permission checks * Rename and remove permissions in AWSPermissions * Register AWSPermissions in IntegrationsModule * Remove space between methods * Save AWS input API endpoint (#93) * First-pass structure for saving AWS input * Add more structure for general AWS input - Add a type enum to differentiate the various types of log messages that are possible. - Add metacodec that will eventually differentiate between the types of log messages. - Fill configuration values when saving the input. * Resolve merge conflicts after rebasing over latest aws branch * Consolidate log type detection and input type identification There's no longer a need to use two enums for this. Also added healthCheck tests covering all message types: flow log, raw Cloud Watch and raw Kinesis. * Clean up saveInput request parameters and handling * Fix invalid type specification that was preventing input save * Add unit test for saving input * Fix incorrectly specified arguments * Minor cleanup * Cleanup for PR review * Remove uneeded log statements * Make save AWS input path and description more specific * Indicate that the save request is specifically for Kinesis In the future, each type of AWS input will likely require it's own request object and endpoint due to the fact that unique fields will probably be required for each. * AWSMessageType cleanup * Remove uneeded isFlowLog, isRaw methods. * Remove invalid AWSMessageType.Source.CLOUD_WATCH enum value. Messages are always read from Kinesis, and therefore the source is always Kinesis. Source is meant to differentiate messages from Kinesis and S3 for example. * Improve comments for AWSMessageType.Source enum class and method. * Remove typo * Generify the create AWS integration endpoint The naming, description, and comment now reflect that a generic AWS input is being created. * Return InputSummary response entity for AWS input creation request Also remove unneeded AWSResourceTest * More cleanup of healthCheck after input creation changes - Remove unneeded log_group field for health_check request. Resolves #108 - Add Kinesis stream name as a field in both raw and CloudWatch messages This change is lumped in with the other changes related to saving the input, since lots of healthCheck changes were already made there. This fixes some problems, so might as well have these improvements included with the review. * Clarify that AWSMetaCodec is a general AWS codec This class no longer erroneously extends AbstractKinesisCodec, which was only intended for Kinesis-specific codecs. * Use DateTime instead of long in KinesisLogEntry It turns out that Kinesis Record objects do have an arrival ime Instant timestamp. This is now being used instead of just using the date/time when the message was read by Graylog. * Add missing Guice annotation that were preventing server startup * Return JSON message field list in response from Health Check (#114) * Return JSON fields in healthCheck response There is still an issue with serializing the DateTime timestamp (serializing the deep object tree). This will be addressed separately. * Adapt unit test timestamp check to object value Health Check now returns a map of actual field values, so the test needs to check the JSON value. * Fix incorrect codec specified for Flow Logs, add tests * Fix incorrectly named field * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn * Advanced Kinesis Options * Cleanup * currently busted toggle * redo state handling * Remove artifact * removing commented code * removing unnecessary functions * pr feedback * Rebase conflicts * Fix several issues from PR review * "Final Review" layout (#83) * General Content * Review view layout complete * Cleanup * Update utils file name and export * Change LogFlow > Flow Log * default_values.js files * Route constants (#111) * Route constants * Fixing conflicts * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn Route constants Fixing conflicts * Correct import path * Changing route object * Adjust Kinesis setup wording * Remove the unneeded * Fixing up rebase artifacts from squash of irrelevant commits * AWS lockfile * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * Setting up necessary hooks * Removing input validation steps to reduce PR * Rebasing * Rename StyledForm * DEFAULT_VALUES * cleanup hook returns * Moving some reducers to state and breaking apart the rest * Remove functions from Input values * Steps function cleanup * Remove commented import * FormDataProvider and Context * Steps Context & provider * log and advancedoptions moved to context and provider * FormWrap functional component * renaming things * lint error * PR feedback * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * error and dirty states * Error on submit * Update validation to new state hooks * remove unneeded defaults * Output error messages * text key instead of password * Remove field validation * Update comment blocks (#137) * Update comment blocks * Remove minor typos * Update version to 3.1.0-beta.2-SNAPSHOT * Implement AWS codec selector with tests (#116) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Remove erroneous comment * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump package.json version to 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * Authorize and Streams API * All API calls except Save * reverting to hooks for api calls * Regions hook * Streams and Logs * streams useFetch * log sample * generic loading states * move regions * adding some documentation * Adding all the files * removing uneeded * cleanup * ApiRoutes * import proper path * Changing up useFetch * need the values... * lint * Removed unneeded inline cURL API documentation (#143) No need for this docs duplication. The System > Nodes > API Browser already documents them. * Use standard field names for AWS key and secret (#144) * Use standard naming for AWS secret and access key ids for Save Input Use these standard names for AWS key and secret from the Save Input AWS endpoint (From the AWSRequest interface). All other AWS API requests should be using these already. `aws_access_key_id` `aws_secret_access_key` * Use central constant for Region also * Fix incorrectly set Region property name * Fix access and secret key method names to match the AWSRequest interface * rename useFetch file * better var naming in kinesisstreams: * Update useFetch props * AWS transport selector with tests (#117) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Remove duplicate AWSTestingUtils class * Add missing permissions for sample AWS policy (#122) * Migrate AWS policy to autovalue object This avoids embedding JSON in a Java constant, which is error prone. * Add missing permissions Added: - kinesis:ListStreams - logs:DescribeLogGroups * Define JSON policy field order that matches AWS examples * Return the policy JSON string to the UI as string This is consistent with how this was done before. The UI can then format and display this to the user on the Available Services page. This is the policy that the user will create the AWS user for the AWS integration with. * Specify the correct AWS policy version * Change AWS Policy to a list * Add missing permissions Based on this [sample KCL code](aws/aws-sdk-java-v2#1214 (comment)) * Update version to 3.1.0-beta.2-SNAPSHOT * Remove uneeded import * Add Available Services API response error documentation * Throw InternalServerErrorException instead of JSON exception This is a bit cleaner from the API consumption side. A nice short error message is now returned instead of an obscure JSON error. * API Save Input (#145) * Save after Review and redirect to Inputs * Save functional * Adding match function to pluginmanifest * custom component * Update version to 3.1.0-beta.3-SNAPSHOT * Update version to 3.1.0-beta.3-SNAPSHOT * Aws kinesis newstream (#129) * Add create new Kinesis stream method * Add API call for createNewKinesisStream and KinesisNewStream classes * Update createNewKinesisStream * Update version to 3.1.0-beta.2-SNAPSHOT * Add error logging to createNewKinesisStream method * Add unit test for createNewKinesisStream method * Update error logging for createNewKinesisStream * Update testCreateNewKinesisStream * Remove shard count variable and set default value * Update testCreateNewKinesisStream * Code clean up * Update api comment blocks * Code clean up * Update error handling in createNewKinesisStream * Update unit test testCreateNewKinesisStream * Unload Confirmation Dialog (#148) * Playing with onunload * use ConfirmLeaveDialog common component * Question * Update version to 3.1.0-beta.3-SNAPSHOT * lint * cleanup * Don't alert on last step. Save call failed if desc wasn't available * Remove emotion and use styled-components (#157) * Remove emotion and use styled-components * import styled-components * Move title and description to FormWrap for consistent layout * Handle validation per-step * fix aws key h4x * styled validatedinput * enable health check and review forms * Starting Error Handling * Output API Errors * remove some more debugging * Validate API Key & Secret and output styled errors * Error message if stream doesn't contain any logs * Return a 4xx response for an unsuccessful health check * Save API Error Handling * Remove `success` flag from HealthCheckResponse Failure is now tracked by throwing a `BadRequestException` resulting in an `ApiError` json response * Throw `BadRequestException` if no Kinesis streams are found * Throw `BadRequestException` if no Kinesis CloudWatch groups are found * Error message styling * wrap up api errors and adding styles * rebasing and cleanup * trim input values * remove defaults * Create input with togglable masking * Include stream name in error * Port of existing KinesisTransport (#140) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Remove unused imports and fix formatting * Update version to 3.1.0-beta.3-SNAPSHOT * Merge branch 'aws' into aws-transport-kinesis and resolve conflicts # Conflicts: # src/main/java/org/graylog/integrations/aws/service/KinesisService.java * Upgrade to latest Kinesis Client version (#151) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Update Kinesis Client version * Add Kinesis client library v2 Consumer sample * Migrate KinesisConsumer to Kinesis Client Library v2 * Remove unused imports and fix formatting * Finish migration of Kinesis client to new version * Complete upgrade to Kinesis Client Library v2 * Bump KCL version * Migrate shard processor to its own class * Bump version to 3.1.0-beta.3-SNAPSHOT * Add batch size limit * Code cleanup * Remove unneeded throttle time limit reached shutdown * Remove kinesis_max_throttled_wait_ms save input field This field is no longer needed, since the new Kinesis Consumer appears to correctly handle longer throttling and pausing in `processRecords` without making the consumer unhealthy. * Temporarily remove Assume Role Arn auth This will be added back later. See #29 * Adjust logging levels * Remove sample class * Ignore unmapped properties due to removed max_throttled_wait field See Graylog2/graylog-plugin-integrations#156 * Update Throttling advanced option (#169) * Update Throttling advanced option * messages/ms * Remove throttle number * Update label * Shells/mocks for Kinesis Auto setup API endpoints (#167) * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Update version to 3.1.0-beta.4-SNAPSHOT * Code cleanup - Remove unneeded imports - Replace hyphens with underscores * Add `aws` to root URL path for consistency All other AWS routes are prefixed with this, so do the same here. * Change subscription path Change `add_subscription` route to `create_subscription` * Fix linter warning was introduced during merge conflict resolution. * Add RolePermissions for Kinesis stream (#163) * Add checkKinesisStreamStatus and setRolePermissions method * Add getNewRolePermissions and autoKinesisPermissionRequired method * Rename autoKinesisPermissionsRequired method and add error handling * Code clean up * Refactor autoSetup, split up creating a role and attaching a policy * Add error handling for creating a new role * Update autoKinesisPermissionsRequired logic and add comment block * Update version to 3.1.0-beta.4-SNAPSHOT * Delete checkKinesisStreamStatus method * Update error logging messages * Add temporary main class for testing, update error handling logic * Delete temporary main, update autoKinesisPermissionsRequired * Change exceptions thrown and code clean up * Update version to 3.1.0-rc.2-SNAPSHOT * Add addSubscriptionFilter to CloudwatchService (#160) * Add addSubscriptionFilter method to KinesisService * Move addSubscriptionFilter method to CloudWatchService * Code clean up, change exception errors * Bump version to 3.1.0-rc.3-SNAPSHOT * Kinesis Auto-setup (#186) * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump js-yaml from 3.13.0 to 3.13.1 Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 3.13.0 to 3.13.1. - [Release notes](https://github.com/nodeca/js-yaml/releases) - [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md) - [Commits](nodeca/js-yaml@3.13.0...3.13.1) Signed-off-by: dependabot[bot] <support@github.com> * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * [Security] Bump lodash from 4.17.11 to 4.17.14 (#131) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.11 to 4.17.14. **This update includes security fixes.** - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.11...4.17.14) Signed-off-by: dependabot-preview[bot] <support@dependabot.com> * Bump package.json version to 3.1.0-beta.2 * [graylog-plugin-integrations] prepare release 3.1.0-beta.2 * Bump package.json version to 3.1.0-beta.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Bump package.json version to 3.1.0-beta.3 * [graylog-plugin-integrations] prepare release 3.1.0-beta.3 * Bump package.json version to 3.1.0-beta.4-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Render Setup screen * cleanup * Allowing toggle to get from Streams to Setup * Toggle back to Streams & cleanup Group on Region change * Add Kinesis auto-setup routes * Add initial `create_stream` request * Define fetchs for auto Kinesis setup * Finish create_stream request * Add all auto-setup API requests * Specify actual region and request properties * Improve presentation for transition to automated Kinesis setup * Improve on-screen documentation for the automatic Kinesis setup * Bump package.json version to 3.1.0-rc.1 * [graylog-plugin-integrations] prepare release 3.1.0-rc.1 * Bump package.json version to 3.1.0-rc.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Update version to 3.1.0-rc.2-SNAPSHOT * Add KinesisSetupStep shell * Add KinesisSetupSteps container * Update createKinesisStream to return streamArn * Update createKinesisStream and testCreateNewKinesisStream * Add edge case error handling to createNewKinesisStream * Add preliminary code for KinesisFullSetup method * Use await for synchronous auto setup steps * Migrate to map for auto-setup state management * Render auto-setup progress to UI * Include success field and initial delay for testing * Delete KinesisFullSetup method * Migrate Kinesis setup to map to allow setup step iteration * Refactor autoKinesisPermissions parameters and return response * Rework setup logic to use individual state fields Individual state fields are needed for each component to update independently. * Add progress icons * Improve icon and step styles * Add auto-setup error handling * Refactor addSubscriptionFilter to return CreateLogSubscriptionResponse * Update api calls in KinesisFullSetup * Update createNewKinesisStream and createPolicies api call * Add flow control for setup steps * Clean up api methods * Add mocking for UI testing Also remove full setup, since the individual steps will be used. * Sync field mappings between UI and backend * Add auto-setup integration unit test * Add auto-setup test assertions * Rename `explanation` -> `result` * Display result to the user for each auto-setup step * Use unique role ane role policy names Since these are only needed in the auto-setup flow, we can automatically generate and assign them (as long as we provide them to the user in the response). Customizing these names is possible, and can be considered in the future. * Adjustments to get auto-setup working * Add support subscription retries to resolve IAM eventual consistency IAM sometimes takes 10+ seconds to initialize, so retrying allows the subscription attempt to be attempted multiple times, so that once IAM is available, then the subscription will be created successfully. * Cleanup retry logic * Add new page for auto-setup steps * Adjust wording * Add auto-setup TODO for adding warning * TODOs and cleanup * Formatting cleanup * Support ability to continue adding input after auto-setup * Bump package.json version to 3.1.0-rc.2 * [graylog-plugin-integrations] prepare release 3.1.0-rc.2 * Bump package.json version to 3.1.0-rc.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Disable Continue Setup button while auto-setup is in progress * Include warning that messages might take time to arrive in new stream * Require explicit agreement from user before starting auto-setup * Bump version to 3.1.0-rc.3-SNAPSHOT * Adjustments to auto-setup buttons, labels, and formatting * Skip CloudWatch control messages in Health Check Fixes Graylog2/graylog-plugin-integrations#178 * Add comment * Add support for the "flow_log_" prefix Closes Graylog2/graylog-plugin-integrations#120 * Fixing lint errors * Return mock responses * DEV COMMIT * Remove unnecessary Row and Col now that we're rendering a sidebar * sidebar styles * Add checkbox for adding Flow Log field prefix * Auto-Setup Sidebar * lint * Adjust wording * Correct label for AWS Message Type * Consolidate Kinesis payload decompression logic * Agree TOS in Modal * Return actual stream name in create stream mock * Modal Styled * Don't include the policy name in response, since it's inline * Remove DEV COMMIT changes * Update header description * Fix failing unit test * Mock responses * Add default setting for prefix field * Add support for assuming an ARN role (#188) * Add support for assuming an ARN role Fixes Graylog2/graylog-plugin-integrations#29 * Add Assume Role ARN config field * Remove unneeded imports * Fix formatting * Record when users agree to auto-setup (#194) * Add log message recording that a user agreed to auto-setup * Remove unneeded import * Fixing select default value (#199) * Adding pattern to Stream Name input (#200) * Adding pattern to Stream Name input * Don't submit if field has error * Add API request to get permissions (#196) * Add request to retireve Kinesis permissions Includes permissions for both the full setup and the auto-setup. Closes #174 * Alphabetize permissions, add comments, remove unused imports * Add unit test * Make auto-setup unit test pass when mocking enabled * Remove unneeded create log group/stream permissions * Minor code clean up (#201) * Minor code clean up * Fix typo * Proposed wording changes We'll pick this up again in Hamburg * Bump version to 3.2.0-SNAPSHOT * Wording adjustments * Change "Setup" -> "Set Up" * Fix lint errors * More wording changes * Removing empty doc link * Updating Review log type output

lennartkoopmann closed this as completed in 152e901 Sep 18, 2013

dennisoelkers mentioned this issue Mar 24, 2016

"match regular expression" stream rule type fails to properly validate message starting with a blank space #1936

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Trim all String field values #186

Trim all String field values #186

lennartkoopmann commented Aug 20, 2013

Trim all String field values #186

Trim all String field values #186

Comments

lennartkoopmann commented Aug 20, 2013