Upgrade to latest RabbitMQ client library version #131

lennartkoopmann · 2013-04-14T19:20:55Z

No description provided.

joschi · 2013-05-01T16:24:54Z

Updated in ac768c7.

lennartkoopmann · 2013-05-04T20:26:46Z

And updated for develop branch in 91203bb. (to 3.1.0)

Bumps [lodash](https://github.com/lodash/lodash) from 4.17.11 to 4.17.14. **This update includes security fixes.** - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.11...4.17.14) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Initial commit * Add AWS SDK v2 dependency * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Add Kinesis SDK * General UI Skeleton (#75) * Simple Clickthrough without API * Clickable Skeleton All Steps * Feedback * Lint * AWS Cloudwatch backend API calls Add initial backend API calls for CloudWatch integration: getRegions, getLogGroups, getStreams, retrieveKinesisMessages, healthCheck * AWS Cloud Watch services and resources (#24) Adds beginnings of API endpoint and Kinesis/Cloudwatch services Includes a structure that we will continue to build from. * Rework organization of classes for unified structure The goal is to establish some structure that we can implement AWS API calls within. There's now one resource (for API calls), one service (for business logic), and one AWSClient (for all AWS/API SDK interactions). * Cleanup and add comments * Cleanup code around log message auto-detection * Aws cloudwatch client (#31) * Add CloudWatchService class * Add AWSConfigSettings class * Add UserCredentials class * Add temporary Main class * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Update AWSConfigSettings class * Update CloudWatchService class * Update temporary Main class * Fix inject error in AWSResource * Fix merge conflicts * Restructure Resources and Service classes * Add getRegion into AWSResource (#43) * Add getRegion into AWSResource * Update api paths * Increase max retry limit for stream get to 1000 Runaway looping Stopping at 100 is probably to small. * Use underscores instead of camel case for json names It is a general project standard to use underscores. * Add paging functionality to getLogGroupNames (#53) * Add paging functionality to getLogGroupNames * Add unit test for CloudWatch log groups * Add comments in log group name unit test * Code clean up and remove unneeded code * Revert unintended change from unit test commit * Add retrievelogs for Kinesis Healthcheck (#76) * Refactor getKinesisStreams in KinesisService * Add validCredentials method in AWSService class * Temporary Main class added to test putting records into a Kinesis stream * Add retrieveKinesisLogs in KinesisService * Update temporary Main class * Update retrieveKinesisLogs to loop through shard iterators * Update Main class * Update pom file * Update validateCredentials in AWSService class * Add createKinesisClient method in KinesisService class * Add testGetStreamCredentials and update testGetStreams * Healthcheck merge (#81) * empty commit to push branch * Add Kinesis Healthcheck (#45) * Improve organization for Flow Log message detection * Improve Flow Log test It now tests for a message with too many and too few spaces. * Add TODOs for healthCheck method logic * Add beginnings of Kinesis healthChecker This will pull a establish a Kinesis subscription and pull a single message from a Kinesis stream. * Fix failing unit test * Continue developing KinesisHealthCheck - Remove unneeded metric tracking - Remove extra parsing logic (this object should just hand back payload and not do any parsing) - Improve application name handling - Add comments * Add detection logic for raw vs. CloudWatch logs * Remove KinesisHealthCheck class The KinesisConsumer does not work well for the health check (designed for realtime processing, takes a long time to start, cannot detect empty stream, and is really hard to use in a quick API request). Now, we're planning to directly retrieve the messages using the Kinesis client. This is the most straight-forward thing We might revisit this later. * Fix JSON parsing of Kinesis CloudWatch subscription record Parse the record just as was done in the existing AWS plugin. The logic now includes autodetection of compressed/vs not compressed. Mock Kinesis CloudWatch subscription record included for testing purposes. * Add CloudWatch logs codec and tests from existing AWS plugin * Parse Flow Log message into object * Load appropriate codec during healthCheck process When the message type is detected, load the respective codec for that message type. * Parse message with appropriate codec Once the log message type is detected, then the codec is looked up. The message is then parsed with the codec. * Supply log group name with the response * Improve comments, logging, and error checking The log group name is now also included in the response. * Add Flow Log codec test * Use AutoValue for CloudWatchLogEntry class * Use AutoValue for all remaining CloudWatch value classes * Cleanup merge conflicts after rebasing * Specify constants for all JsonProperty annotations * Delete uneeded KinesisDTO All data will be stored in the input * Establish a base AWSRequest JSON class * Fix Guice injection error for KinesisService * Add sample cURL command for healthCheck method A similar cURL command will be used for other methods, so that it is clear how the UI will use them. * Remove unneeded Kinesis Client 1.x dependency * Add formatted message summary in the Health Check response * Cleanup formatting and TODOs * Minor cleanup after rebasing and merging * Fix failing unit tests * Integrate AWS Health Check with Kinesis record retrieval (#86) * Fix incorrect pass of AWS key instead of secret Also improve comments for fake message retrieval with TODOs. * Update and connect retrieveRecords * Add handleCompressedMessages method * Delete temporary main class * Update retrieveRecords to only return sample size * Update KinesisService for healthCheck to function properly * Add unit test for selecting random record * Add unit test for retrieveRecords * Add Available Services API call (#82) * Resolves #50: Add Available Services API call * Add a test * Add missing spaces, change Amazon > AWS * Backend code cleanup (#90) * Pass region and AWS credentials with AWS API requests (#92) * Require POST object containing region and credentials for all requests Specifically adds a POST body requirement for the getKinesisSteams and getLogGroupNames methods. * Use snake_case for paths * Update region api call (#110) * Migrate Regions request from a list to a full response object with total * Update Region API call to include label and value * Reformat code * Aws permissions check (#113) * Add AWSPermissions class and update AWSResource with permission checks * Rename and remove permissions in AWSPermissions * Register AWSPermissions in IntegrationsModule * Remove space between methods * Save AWS input API endpoint (#93) * First-pass structure for saving AWS input * Add more structure for general AWS input - Add a type enum to differentiate the various types of log messages that are possible. - Add metacodec that will eventually differentiate between the types of log messages. - Fill configuration values when saving the input. * Resolve merge conflicts after rebasing over latest aws branch * Consolidate log type detection and input type identification There's no longer a need to use two enums for this. Also added healthCheck tests covering all message types: flow log, raw Cloud Watch and raw Kinesis. * Clean up saveInput request parameters and handling * Fix invalid type specification that was preventing input save * Add unit test for saving input * Fix incorrectly specified arguments * Minor cleanup * Cleanup for PR review * Remove uneeded log statements * Make save AWS input path and description more specific * Indicate that the save request is specifically for Kinesis In the future, each type of AWS input will likely require it's own request object and endpoint due to the fact that unique fields will probably be required for each. * AWSMessageType cleanup * Remove uneeded isFlowLog, isRaw methods. * Remove invalid AWSMessageType.Source.CLOUD_WATCH enum value. Messages are always read from Kinesis, and therefore the source is always Kinesis. Source is meant to differentiate messages from Kinesis and S3 for example. * Improve comments for AWSMessageType.Source enum class and method. * Remove typo * Generify the create AWS integration endpoint The naming, description, and comment now reflect that a generic AWS input is being created. * Return InputSummary response entity for AWS input creation request Also remove unneeded AWSResourceTest * More cleanup of healthCheck after input creation changes - Remove unneeded log_group field for health_check request. Resolves #108 - Add Kinesis stream name as a field in both raw and CloudWatch messages This change is lumped in with the other changes related to saving the input, since lots of healthCheck changes were already made there. This fixes some problems, so might as well have these improvements included with the review. * Clarify that AWSMetaCodec is a general AWS codec This class no longer erroneously extends AbstractKinesisCodec, which was only intended for Kinesis-specific codecs. * Use DateTime instead of long in KinesisLogEntry It turns out that Kinesis Record objects do have an arrival ime Instant timestamp. This is now being used instead of just using the date/time when the message was read by Graylog. * Add missing Guice annotation that were preventing server startup * Return JSON message field list in response from Health Check (#114) * Return JSON fields in healthCheck response There is still an issue with serializing the DateTime timestamp (serializing the deep object tree). This will be addressed separately. * Adapt unit test timestamp check to object value Health Check now returns a map of actual field values, so the test needs to check the JSON value. * Fix incorrect codec specified for Flow Logs, add tests * Fix incorrectly named field * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn * Advanced Kinesis Options * Cleanup * currently busted toggle * redo state handling * Remove artifact * removing commented code * removing unnecessary functions * pr feedback * Rebase conflicts * Fix several issues from PR review * "Final Review" layout (#83) * General Content * Review view layout complete * Cleanup * Update utils file name and export * Change LogFlow > Flow Log * default_values.js files * Route constants (#111) * Route constants * Fixing conflicts * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn Route constants Fixing conflicts * Correct import path * Changing route object * Adjust Kinesis setup wording * Remove the unneeded * Fixing up rebase artifacts from squash of irrelevant commits * AWS lockfile * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * Setting up necessary hooks * Removing input validation steps to reduce PR * Rebasing * Rename StyledForm * DEFAULT_VALUES * cleanup hook returns * Moving some reducers to state and breaking apart the rest * Remove functions from Input values * Steps function cleanup * Remove commented import * FormDataProvider and Context * Steps Context & provider * log and advancedoptions moved to context and provider * FormWrap functional component * renaming things * lint error * PR feedback * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * error and dirty states * Error on submit * Update validation to new state hooks * remove unneeded defaults * Output error messages * text key instead of password * Remove field validation * Update comment blocks (#137) * Update comment blocks * Remove minor typos * Update version to 3.1.0-beta.2-SNAPSHOT * Implement AWS codec selector with tests (#116) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Remove erroneous comment * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump package.json version to 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * Authorize and Streams API * All API calls except Save * reverting to hooks for api calls * Regions hook * Streams and Logs * streams useFetch * log sample * generic loading states * move regions * adding some documentation * Adding all the files * removing uneeded * cleanup * ApiRoutes * import proper path * Changing up useFetch * need the values... * lint * Removed unneeded inline cURL API documentation (#143) No need for this docs duplication. The System > Nodes > API Browser already documents them. * Use standard field names for AWS key and secret (#144) * Use standard naming for AWS secret and access key ids for Save Input Use these standard names for AWS key and secret from the Save Input AWS endpoint (From the AWSRequest interface). All other AWS API requests should be using these already. `aws_access_key_id` `aws_secret_access_key` * Use central constant for Region also * Fix incorrectly set Region property name * Fix access and secret key method names to match the AWSRequest interface * rename useFetch file * better var naming in kinesisstreams: * Update useFetch props * AWS transport selector with tests (#117) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Remove duplicate AWSTestingUtils class * Add missing permissions for sample AWS policy (#122) * Migrate AWS policy to autovalue object This avoids embedding JSON in a Java constant, which is error prone. * Add missing permissions Added: - kinesis:ListStreams - logs:DescribeLogGroups * Define JSON policy field order that matches AWS examples * Return the policy JSON string to the UI as string This is consistent with how this was done before. The UI can then format and display this to the user on the Available Services page. This is the policy that the user will create the AWS user for the AWS integration with. * Specify the correct AWS policy version * Change AWS Policy to a list * Add missing permissions Based on this [sample KCL code](aws/aws-sdk-java-v2#1214 (comment)) * Update version to 3.1.0-beta.2-SNAPSHOT * Remove uneeded import * Add Available Services API response error documentation * Throw InternalServerErrorException instead of JSON exception This is a bit cleaner from the API consumption side. A nice short error message is now returned instead of an obscure JSON error. * API Save Input (#145) * Save after Review and redirect to Inputs * Save functional * Adding match function to pluginmanifest * custom component * Update version to 3.1.0-beta.3-SNAPSHOT * Update version to 3.1.0-beta.3-SNAPSHOT * Aws kinesis newstream (#129) * Add create new Kinesis stream method * Add API call for createNewKinesisStream and KinesisNewStream classes * Update createNewKinesisStream * Update version to 3.1.0-beta.2-SNAPSHOT * Add error logging to createNewKinesisStream method * Add unit test for createNewKinesisStream method * Update error logging for createNewKinesisStream * Update testCreateNewKinesisStream * Remove shard count variable and set default value * Update testCreateNewKinesisStream * Code clean up * Update api comment blocks * Code clean up * Update error handling in createNewKinesisStream * Update unit test testCreateNewKinesisStream * Unload Confirmation Dialog (#148) * Playing with onunload * use ConfirmLeaveDialog common component * Question * Update version to 3.1.0-beta.3-SNAPSHOT * lint * cleanup * Don't alert on last step. Save call failed if desc wasn't available * Remove emotion and use styled-components (#157) * Remove emotion and use styled-components * import styled-components * Move title and description to FormWrap for consistent layout * Handle validation per-step * fix aws key h4x * styled validatedinput * enable health check and review forms * Starting Error Handling * Output API Errors * remove some more debugging * Validate API Key & Secret and output styled errors * Error message if stream doesn't contain any logs * Return a 4xx response for an unsuccessful health check * Save API Error Handling * Remove `success` flag from HealthCheckResponse Failure is now tracked by throwing a `BadRequestException` resulting in an `ApiError` json response * Throw `BadRequestException` if no Kinesis streams are found * Throw `BadRequestException` if no Kinesis CloudWatch groups are found * Error message styling * wrap up api errors and adding styles * rebasing and cleanup * trim input values * remove defaults * Create input with togglable masking * Include stream name in error * Port of existing KinesisTransport (#140) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Remove unused imports and fix formatting * Update version to 3.1.0-beta.3-SNAPSHOT * Merge branch 'aws' into aws-transport-kinesis and resolve conflicts # Conflicts: # src/main/java/org/graylog/integrations/aws/service/KinesisService.java * Upgrade to latest Kinesis Client version (#151) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Update Kinesis Client version * Add Kinesis client library v2 Consumer sample * Migrate KinesisConsumer to Kinesis Client Library v2 * Remove unused imports and fix formatting * Finish migration of Kinesis client to new version * Complete upgrade to Kinesis Client Library v2 * Bump KCL version * Migrate shard processor to its own class * Bump version to 3.1.0-beta.3-SNAPSHOT * Add batch size limit * Code cleanup * Remove unneeded throttle time limit reached shutdown * Remove kinesis_max_throttled_wait_ms save input field This field is no longer needed, since the new Kinesis Consumer appears to correctly handle longer throttling and pausing in `processRecords` without making the consumer unhealthy. * Temporarily remove Assume Role Arn auth This will be added back later. See #29 * Adjust logging levels * Remove sample class * Ignore unmapped properties due to removed max_throttled_wait field See Graylog2/graylog-plugin-integrations#156 * Update Throttling advanced option (#169) * Update Throttling advanced option * messages/ms * Remove throttle number * Update label * Shells/mocks for Kinesis Auto setup API endpoints (#167) * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Update version to 3.1.0-beta.4-SNAPSHOT * Code cleanup - Remove unneeded imports - Replace hyphens with underscores * Add `aws` to root URL path for consistency All other AWS routes are prefixed with this, so do the same here. * Change subscription path Change `add_subscription` route to `create_subscription` * Fix linter warning was introduced during merge conflict resolution. * Add RolePermissions for Kinesis stream (#163) * Add checkKinesisStreamStatus and setRolePermissions method * Add getNewRolePermissions and autoKinesisPermissionRequired method * Rename autoKinesisPermissionsRequired method and add error handling * Code clean up * Refactor autoSetup, split up creating a role and attaching a policy * Add error handling for creating a new role * Update autoKinesisPermissionsRequired logic and add comment block * Update version to 3.1.0-beta.4-SNAPSHOT * Delete checkKinesisStreamStatus method * Update error logging messages * Add temporary main class for testing, update error handling logic * Delete temporary main, update autoKinesisPermissionsRequired * Change exceptions thrown and code clean up * Update version to 3.1.0-rc.2-SNAPSHOT * Add addSubscriptionFilter to CloudwatchService (#160) * Add addSubscriptionFilter method to KinesisService * Move addSubscriptionFilter method to CloudWatchService * Code clean up, change exception errors * Bump version to 3.1.0-rc.3-SNAPSHOT * Kinesis Auto-setup (#186) * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump js-yaml from 3.13.0 to 3.13.1 Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 3.13.0 to 3.13.1. - [Release notes](https://github.com/nodeca/js-yaml/releases) - [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md) - [Commits](nodeca/js-yaml@3.13.0...3.13.1) Signed-off-by: dependabot[bot] <support@github.com> * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * [Security] Bump lodash from 4.17.11 to 4.17.14 (#131) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.11 to 4.17.14. **This update includes security fixes.** - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.11...4.17.14) Signed-off-by: dependabot-preview[bot] <support@dependabot.com> * Bump package.json version to 3.1.0-beta.2 * [graylog-plugin-integrations] prepare release 3.1.0-beta.2 * Bump package.json version to 3.1.0-beta.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Bump package.json version to 3.1.0-beta.3 * [graylog-plugin-integrations] prepare release 3.1.0-beta.3 * Bump package.json version to 3.1.0-beta.4-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Render Setup screen * cleanup * Allowing toggle to get from Streams to Setup * Toggle back to Streams & cleanup Group on Region change * Add Kinesis auto-setup routes * Add initial `create_stream` request * Define fetchs for auto Kinesis setup * Finish create_stream request * Add all auto-setup API requests * Specify actual region and request properties * Improve presentation for transition to automated Kinesis setup * Improve on-screen documentation for the automatic Kinesis setup * Bump package.json version to 3.1.0-rc.1 * [graylog-plugin-integrations] prepare release 3.1.0-rc.1 * Bump package.json version to 3.1.0-rc.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Update version to 3.1.0-rc.2-SNAPSHOT * Add KinesisSetupStep shell * Add KinesisSetupSteps container * Update createKinesisStream to return streamArn * Update createKinesisStream and testCreateNewKinesisStream * Add edge case error handling to createNewKinesisStream * Add preliminary code for KinesisFullSetup method * Use await for synchronous auto setup steps * Migrate to map for auto-setup state management * Render auto-setup progress to UI * Include success field and initial delay for testing * Delete KinesisFullSetup method * Migrate Kinesis setup to map to allow setup step iteration * Refactor autoKinesisPermissions parameters and return response * Rework setup logic to use individual state fields Individual state fields are needed for each component to update independently. * Add progress icons * Improve icon and step styles * Add auto-setup error handling * Refactor addSubscriptionFilter to return CreateLogSubscriptionResponse * Update api calls in KinesisFullSetup * Update createNewKinesisStream and createPolicies api call * Add flow control for setup steps * Clean up api methods * Add mocking for UI testing Also remove full setup, since the individual steps will be used. * Sync field mappings between UI and backend * Add auto-setup integration unit test * Add auto-setup test assertions * Rename `explanation` -> `result` * Display result to the user for each auto-setup step * Use unique role ane role policy names Since these are only needed in the auto-setup flow, we can automatically generate and assign them (as long as we provide them to the user in the response). Customizing these names is possible, and can be considered in the future. * Adjustments to get auto-setup working * Add support subscription retries to resolve IAM eventual consistency IAM sometimes takes 10+ seconds to initialize, so retrying allows the subscription attempt to be attempted multiple times, so that once IAM is available, then the subscription will be created successfully. * Cleanup retry logic * Add new page for auto-setup steps * Adjust wording * Add auto-setup TODO for adding warning * TODOs and cleanup * Formatting cleanup * Support ability to continue adding input after auto-setup * Bump package.json version to 3.1.0-rc.2 * [graylog-plugin-integrations] prepare release 3.1.0-rc.2 * Bump package.json version to 3.1.0-rc.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Disable Continue Setup button while auto-setup is in progress * Include warning that messages might take time to arrive in new stream * Require explicit agreement from user before starting auto-setup * Bump version to 3.1.0-rc.3-SNAPSHOT * Adjustments to auto-setup buttons, labels, and formatting * Skip CloudWatch control messages in Health Check Fixes Graylog2/graylog-plugin-integrations#178 * Add comment * Add support for the "flow_log_" prefix Closes Graylog2/graylog-plugin-integrations#120 * Fixing lint errors * Return mock responses * DEV COMMIT * Remove unnecessary Row and Col now that we're rendering a sidebar * sidebar styles * Add checkbox for adding Flow Log field prefix * Auto-Setup Sidebar * lint * Adjust wording * Correct label for AWS Message Type * Consolidate Kinesis payload decompression logic * Agree TOS in Modal * Return actual stream name in create stream mock * Modal Styled * Don't include the policy name in response, since it's inline * Remove DEV COMMIT changes * Update header description * Fix failing unit test * Mock responses * Add default setting for prefix field * Add support for assuming an ARN role (#188) * Add support for assuming an ARN role Fixes Graylog2/graylog-plugin-integrations#29 * Add Assume Role ARN config field * Remove unneeded imports * Fix formatting * Record when users agree to auto-setup (#194) * Add log message recording that a user agreed to auto-setup * Remove unneeded import * Fixing select default value (#199) * Adding pattern to Stream Name input (#200) * Adding pattern to Stream Name input * Don't submit if field has error * Add API request to get permissions (#196) * Add request to retireve Kinesis permissions Includes permissions for both the full setup and the auto-setup. Closes #174 * Alphabetize permissions, add comments, remove unused imports * Add unit test * Make auto-setup unit test pass when mocking enabled * Remove unneeded create log group/stream permissions * Minor code clean up (#201) * Minor code clean up * Fix typo * Bump version to 3.2.0-SNAPSHOT * Linting Fix (#212)

* Initial commit * Add AWS SDK v2 dependency * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Add Kinesis SDK * General UI Skeleton (#75) * Simple Clickthrough without API * Clickable Skeleton All Steps * Feedback * Lint * AWS Cloudwatch backend API calls Add initial backend API calls for CloudWatch integration: getRegions, getLogGroups, getStreams, retrieveKinesisMessages, healthCheck * AWS Cloud Watch services and resources (#24) Adds beginnings of API endpoint and Kinesis/Cloudwatch services Includes a structure that we will continue to build from. * Rework organization of classes for unified structure The goal is to establish some structure that we can implement AWS API calls within. There's now one resource (for API calls), one service (for business logic), and one AWSClient (for all AWS/API SDK interactions). * Cleanup and add comments * Cleanup code around log message auto-detection * Aws cloudwatch client (#31) * Add CloudWatchService class * Add AWSConfigSettings class * Add UserCredentials class * Add temporary Main class * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Update AWSConfigSettings class * Update CloudWatchService class * Update temporary Main class * Fix inject error in AWSResource * Fix merge conflicts * Restructure Resources and Service classes * Add getRegion into AWSResource (#43) * Add getRegion into AWSResource * Update api paths * Increase max retry limit for stream get to 1000 Runaway looping Stopping at 100 is probably to small. * Use underscores instead of camel case for json names It is a general project standard to use underscores. * Add paging functionality to getLogGroupNames (#53) * Add paging functionality to getLogGroupNames * Add unit test for CloudWatch log groups * Add comments in log group name unit test * Code clean up and remove unneeded code * Revert unintended change from unit test commit * Add retrievelogs for Kinesis Healthcheck (#76) * Refactor getKinesisStreams in KinesisService * Add validCredentials method in AWSService class * Temporary Main class added to test putting records into a Kinesis stream * Add retrieveKinesisLogs in KinesisService * Update temporary Main class * Update retrieveKinesisLogs to loop through shard iterators * Update Main class * Update pom file * Update validateCredentials in AWSService class * Add createKinesisClient method in KinesisService class * Add testGetStreamCredentials and update testGetStreams * Healthcheck merge (#81) * empty commit to push branch * Add Kinesis Healthcheck (#45) * Improve organization for Flow Log message detection * Improve Flow Log test It now tests for a message with too many and too few spaces. * Add TODOs for healthCheck method logic * Add beginnings of Kinesis healthChecker This will pull a establish a Kinesis subscription and pull a single message from a Kinesis stream. * Fix failing unit test * Continue developing KinesisHealthCheck - Remove unneeded metric tracking - Remove extra parsing logic (this object should just hand back payload and not do any parsing) - Improve application name handling - Add comments * Add detection logic for raw vs. CloudWatch logs * Remove KinesisHealthCheck class The KinesisConsumer does not work well for the health check (designed for realtime processing, takes a long time to start, cannot detect empty stream, and is really hard to use in a quick API request). Now, we're planning to directly retrieve the messages using the Kinesis client. This is the most straight-forward thing We might revisit this later. * Fix JSON parsing of Kinesis CloudWatch subscription record Parse the record just as was done in the existing AWS plugin. The logic now includes autodetection of compressed/vs not compressed. Mock Kinesis CloudWatch subscription record included for testing purposes. * Add CloudWatch logs codec and tests from existing AWS plugin * Parse Flow Log message into object * Load appropriate codec during healthCheck process When the message type is detected, load the respective codec for that message type. * Parse message with appropriate codec Once the log message type is detected, then the codec is looked up. The message is then parsed with the codec. * Supply log group name with the response * Improve comments, logging, and error checking The log group name is now also included in the response. * Add Flow Log codec test * Use AutoValue for CloudWatchLogEntry class * Use AutoValue for all remaining CloudWatch value classes * Cleanup merge conflicts after rebasing * Specify constants for all JsonProperty annotations * Delete uneeded KinesisDTO All data will be stored in the input * Establish a base AWSRequest JSON class * Fix Guice injection error for KinesisService * Add sample cURL command for healthCheck method A similar cURL command will be used for other methods, so that it is clear how the UI will use them. * Remove unneeded Kinesis Client 1.x dependency * Add formatted message summary in the Health Check response * Cleanup formatting and TODOs * Minor cleanup after rebasing and merging * Fix failing unit tests * Integrate AWS Health Check with Kinesis record retrieval (#86) * Fix incorrect pass of AWS key instead of secret Also improve comments for fake message retrieval with TODOs. * Update and connect retrieveRecords * Add handleCompressedMessages method * Delete temporary main class * Update retrieveRecords to only return sample size * Update KinesisService for healthCheck to function properly * Add unit test for selecting random record * Add unit test for retrieveRecords * Add Available Services API call (#82) * Resolves #50: Add Available Services API call * Add a test * Add missing spaces, change Amazon > AWS * Backend code cleanup (#90) * Pass region and AWS credentials with AWS API requests (#92) * Require POST object containing region and credentials for all requests Specifically adds a POST body requirement for the getKinesisSteams and getLogGroupNames methods. * Use snake_case for paths * Update region api call (#110) * Migrate Regions request from a list to a full response object with total * Update Region API call to include label and value * Reformat code * Aws permissions check (#113) * Add AWSPermissions class and update AWSResource with permission checks * Rename and remove permissions in AWSPermissions * Register AWSPermissions in IntegrationsModule * Remove space between methods * Save AWS input API endpoint (#93) * First-pass structure for saving AWS input * Add more structure for general AWS input - Add a type enum to differentiate the various types of log messages that are possible. - Add metacodec that will eventually differentiate between the types of log messages. - Fill configuration values when saving the input. * Resolve merge conflicts after rebasing over latest aws branch * Consolidate log type detection and input type identification There's no longer a need to use two enums for this. Also added healthCheck tests covering all message types: flow log, raw Cloud Watch and raw Kinesis. * Clean up saveInput request parameters and handling * Fix invalid type specification that was preventing input save * Add unit test for saving input * Fix incorrectly specified arguments * Minor cleanup * Cleanup for PR review * Remove uneeded log statements * Make save AWS input path and description more specific * Indicate that the save request is specifically for Kinesis In the future, each type of AWS input will likely require it's own request object and endpoint due to the fact that unique fields will probably be required for each. * AWSMessageType cleanup * Remove uneeded isFlowLog, isRaw methods. * Remove invalid AWSMessageType.Source.CLOUD_WATCH enum value. Messages are always read from Kinesis, and therefore the source is always Kinesis. Source is meant to differentiate messages from Kinesis and S3 for example. * Improve comments for AWSMessageType.Source enum class and method. * Remove typo * Generify the create AWS integration endpoint The naming, description, and comment now reflect that a generic AWS input is being created. * Return InputSummary response entity for AWS input creation request Also remove unneeded AWSResourceTest * More cleanup of healthCheck after input creation changes - Remove unneeded log_group field for health_check request. Resolves #108 - Add Kinesis stream name as a field in both raw and CloudWatch messages This change is lumped in with the other changes related to saving the input, since lots of healthCheck changes were already made there. This fixes some problems, so might as well have these improvements included with the review. * Clarify that AWSMetaCodec is a general AWS codec This class no longer erroneously extends AbstractKinesisCodec, which was only intended for Kinesis-specific codecs. * Use DateTime instead of long in KinesisLogEntry It turns out that Kinesis Record objects do have an arrival ime Instant timestamp. This is now being used instead of just using the date/time when the message was read by Graylog. * Add missing Guice annotation that were preventing server startup * Return JSON message field list in response from Health Check (#114) * Return JSON fields in healthCheck response There is still an issue with serializing the DateTime timestamp (serializing the deep object tree). This will be addressed separately. * Adapt unit test timestamp check to object value Health Check now returns a map of actual field values, so the test needs to check the JSON value. * Fix incorrect codec specified for Flow Logs, add tests * Fix incorrectly named field * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn * Advanced Kinesis Options * Cleanup * currently busted toggle * redo state handling * Remove artifact * removing commented code * removing unnecessary functions * pr feedback * Rebase conflicts * Fix several issues from PR review * "Final Review" layout (#83) * General Content * Review view layout complete * Cleanup * Update utils file name and export * Change LogFlow > Flow Log * default_values.js files * Route constants (#111) * Route constants * Fixing conflicts * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn Route constants Fixing conflicts * Correct import path * Changing route object * Adjust Kinesis setup wording * Remove the unneeded * Fixing up rebase artifacts from squash of irrelevant commits * AWS lockfile * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * Setting up necessary hooks * Removing input validation steps to reduce PR * Rebasing * Rename StyledForm * DEFAULT_VALUES * cleanup hook returns * Moving some reducers to state and breaking apart the rest * Remove functions from Input values * Steps function cleanup * Remove commented import * FormDataProvider and Context * Steps Context & provider * log and advancedoptions moved to context and provider * FormWrap functional component * renaming things * lint error * PR feedback * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * error and dirty states * Error on submit * Update validation to new state hooks * remove unneeded defaults * Output error messages * text key instead of password * Remove field validation * Update comment blocks (#137) * Update comment blocks * Remove minor typos * Update version to 3.1.0-beta.2-SNAPSHOT * Implement AWS codec selector with tests (#116) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Remove erroneous comment * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump package.json version to 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * Authorize and Streams API * All API calls except Save * reverting to hooks for api calls * Regions hook * Streams and Logs * streams useFetch * log sample * generic loading states * move regions * adding some documentation * Adding all the files * removing uneeded * cleanup * ApiRoutes * import proper path * Changing up useFetch * need the values... * lint * Removed unneeded inline cURL API documentation (#143) No need for this docs duplication. The System > Nodes > API Browser already documents them. * Use standard field names for AWS key and secret (#144) * Use standard naming for AWS secret and access key ids for Save Input Use these standard names for AWS key and secret from the Save Input AWS endpoint (From the AWSRequest interface). All other AWS API requests should be using these already. `aws_access_key_id` `aws_secret_access_key` * Use central constant for Region also * Fix incorrectly set Region property name * Fix access and secret key method names to match the AWSRequest interface * rename useFetch file * better var naming in kinesisstreams: * Update useFetch props * AWS transport selector with tests (#117) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Remove duplicate AWSTestingUtils class * Add missing permissions for sample AWS policy (#122) * Migrate AWS policy to autovalue object This avoids embedding JSON in a Java constant, which is error prone. * Add missing permissions Added: - kinesis:ListStreams - logs:DescribeLogGroups * Define JSON policy field order that matches AWS examples * Return the policy JSON string to the UI as string This is consistent with how this was done before. The UI can then format and display this to the user on the Available Services page. This is the policy that the user will create the AWS user for the AWS integration with. * Specify the correct AWS policy version * Change AWS Policy to a list * Add missing permissions Based on this [sample KCL code](aws/aws-sdk-java-v2#1214 (comment)) * Update version to 3.1.0-beta.2-SNAPSHOT * Remove uneeded import * Add Available Services API response error documentation * Throw InternalServerErrorException instead of JSON exception This is a bit cleaner from the API consumption side. A nice short error message is now returned instead of an obscure JSON error. * API Save Input (#145) * Save after Review and redirect to Inputs * Save functional * Adding match function to pluginmanifest * custom component * Update version to 3.1.0-beta.3-SNAPSHOT * Update version to 3.1.0-beta.3-SNAPSHOT * Aws kinesis newstream (#129) * Add create new Kinesis stream method * Add API call for createNewKinesisStream and KinesisNewStream classes * Update createNewKinesisStream * Update version to 3.1.0-beta.2-SNAPSHOT * Add error logging to createNewKinesisStream method * Add unit test for createNewKinesisStream method * Update error logging for createNewKinesisStream * Update testCreateNewKinesisStream * Remove shard count variable and set default value * Update testCreateNewKinesisStream * Code clean up * Update api comment blocks * Code clean up * Update error handling in createNewKinesisStream * Update unit test testCreateNewKinesisStream * Unload Confirmation Dialog (#148) * Playing with onunload * use ConfirmLeaveDialog common component * Question * Update version to 3.1.0-beta.3-SNAPSHOT * lint * cleanup * Don't alert on last step. Save call failed if desc wasn't available * Remove emotion and use styled-components (#157) * Remove emotion and use styled-components * import styled-components * Move title and description to FormWrap for consistent layout * Handle validation per-step * fix aws key h4x * styled validatedinput * enable health check and review forms * Starting Error Handling * Output API Errors * remove some more debugging * Validate API Key & Secret and output styled errors * Error message if stream doesn't contain any logs * Return a 4xx response for an unsuccessful health check * Save API Error Handling * Remove `success` flag from HealthCheckResponse Failure is now tracked by throwing a `BadRequestException` resulting in an `ApiError` json response * Throw `BadRequestException` if no Kinesis streams are found * Throw `BadRequestException` if no Kinesis CloudWatch groups are found * Error message styling * wrap up api errors and adding styles * rebasing and cleanup * trim input values * remove defaults * Create input with togglable masking * Include stream name in error * Port of existing KinesisTransport (#140) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Remove unused imports and fix formatting * Update version to 3.1.0-beta.3-SNAPSHOT * Merge branch 'aws' into aws-transport-kinesis and resolve conflicts # Conflicts: # src/main/java/org/graylog/integrations/aws/service/KinesisService.java * Upgrade to latest Kinesis Client version (#151) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Update Kinesis Client version * Add Kinesis client library v2 Consumer sample * Migrate KinesisConsumer to Kinesis Client Library v2 * Remove unused imports and fix formatting * Finish migration of Kinesis client to new version * Complete upgrade to Kinesis Client Library v2 * Bump KCL version * Migrate shard processor to its own class * Bump version to 3.1.0-beta.3-SNAPSHOT * Add batch size limit * Code cleanup * Remove unneeded throttle time limit reached shutdown * Remove kinesis_max_throttled_wait_ms save input field This field is no longer needed, since the new Kinesis Consumer appears to correctly handle longer throttling and pausing in `processRecords` without making the consumer unhealthy. * Temporarily remove Assume Role Arn auth This will be added back later. See #29 * Adjust logging levels * Remove sample class * Ignore unmapped properties due to removed max_throttled_wait field See Graylog2/graylog-plugin-integrations#156 * Update Throttling advanced option (#169) * Update Throttling advanced option * messages/ms * Remove throttle number * Update label * Shells/mocks for Kinesis Auto setup API endpoints (#167) * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Update version to 3.1.0-beta.4-SNAPSHOT * Code cleanup - Remove unneeded imports - Replace hyphens with underscores * Add `aws` to root URL path for consistency All other AWS routes are prefixed with this, so do the same here. * Change subscription path Change `add_subscription` route to `create_subscription` * Fix linter warning was introduced during merge conflict resolution. * Add RolePermissions for Kinesis stream (#163) * Add checkKinesisStreamStatus and setRolePermissions method * Add getNewRolePermissions and autoKinesisPermissionRequired method * Rename autoKinesisPermissionsRequired method and add error handling * Code clean up * Refactor autoSetup, split up creating a role and attaching a policy * Add error handling for creating a new role * Update autoKinesisPermissionsRequired logic and add comment block * Update version to 3.1.0-beta.4-SNAPSHOT * Delete checkKinesisStreamStatus method * Update error logging messages * Add temporary main class for testing, update error handling logic * Delete temporary main, update autoKinesisPermissionsRequired * Change exceptions thrown and code clean up * Update version to 3.1.0-rc.2-SNAPSHOT * Add addSubscriptionFilter to CloudwatchService (#160) * Add addSubscriptionFilter method to KinesisService * Move addSubscriptionFilter method to CloudWatchService * Code clean up, change exception errors * Bump version to 3.1.0-rc.3-SNAPSHOT * Kinesis Auto-setup (#186) * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump js-yaml from 3.13.0 to 3.13.1 Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 3.13.0 to 3.13.1. - [Release notes](https://github.com/nodeca/js-yaml/releases) - [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md) - [Commits](nodeca/js-yaml@3.13.0...3.13.1) Signed-off-by: dependabot[bot] <support@github.com> * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * [Security] Bump lodash from 4.17.11 to 4.17.14 (#131) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.11 to 4.17.14. **This update includes security fixes.** - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.11...4.17.14) Signed-off-by: dependabot-preview[bot] <support@dependabot.com> * Bump package.json version to 3.1.0-beta.2 * [graylog-plugin-integrations] prepare release 3.1.0-beta.2 * Bump package.json version to 3.1.0-beta.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Bump package.json version to 3.1.0-beta.3 * [graylog-plugin-integrations] prepare release 3.1.0-beta.3 * Bump package.json version to 3.1.0-beta.4-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Render Setup screen * cleanup * Allowing toggle to get from Streams to Setup * Toggle back to Streams & cleanup Group on Region change * Add Kinesis auto-setup routes * Add initial `create_stream` request * Define fetchs for auto Kinesis setup * Finish create_stream request * Add all auto-setup API requests * Specify actual region and request properties * Improve presentation for transition to automated Kinesis setup * Improve on-screen documentation for the automatic Kinesis setup * Bump package.json version to 3.1.0-rc.1 * [graylog-plugin-integrations] prepare release 3.1.0-rc.1 * Bump package.json version to 3.1.0-rc.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Update version to 3.1.0-rc.2-SNAPSHOT * Add KinesisSetupStep shell * Add KinesisSetupSteps container * Update createKinesisStream to return streamArn * Update createKinesisStream and testCreateNewKinesisStream * Add edge case error handling to createNewKinesisStream * Add preliminary code for KinesisFullSetup method * Use await for synchronous auto setup steps * Migrate to map for auto-setup state management * Render auto-setup progress to UI * Include success field and initial delay for testing * Delete KinesisFullSetup method * Migrate Kinesis setup to map to allow setup step iteration * Refactor autoKinesisPermissions parameters and return response * Rework setup logic to use individual state fields Individual state fields are needed for each component to update independently. * Add progress icons * Improve icon and step styles * Add auto-setup error handling * Refactor addSubscriptionFilter to return CreateLogSubscriptionResponse * Update api calls in KinesisFullSetup * Update createNewKinesisStream and createPolicies api call * Add flow control for setup steps * Clean up api methods * Add mocking for UI testing Also remove full setup, since the individual steps will be used. * Sync field mappings between UI and backend * Add auto-setup integration unit test * Add auto-setup test assertions * Rename `explanation` -> `result` * Display result to the user for each auto-setup step * Use unique role ane role policy names Since these are only needed in the auto-setup flow, we can automatically generate and assign them (as long as we provide them to the user in the response). Customizing these names is possible, and can be considered in the future. * Adjustments to get auto-setup working * Add support subscription retries to resolve IAM eventual consistency IAM sometimes takes 10+ seconds to initialize, so retrying allows the subscription attempt to be attempted multiple times, so that once IAM is available, then the subscription will be created successfully. * Cleanup retry logic * Add new page for auto-setup steps * Adjust wording * Add auto-setup TODO for adding warning * TODOs and cleanup * Formatting cleanup * Support ability to continue adding input after auto-setup * Bump package.json version to 3.1.0-rc.2 * [graylog-plugin-integrations] prepare release 3.1.0-rc.2 * Bump package.json version to 3.1.0-rc.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Disable Continue Setup button while auto-setup is in progress * Include warning that messages might take time to arrive in new stream * Require explicit agreement from user before starting auto-setup * Bump version to 3.1.0-rc.3-SNAPSHOT * Adjustments to auto-setup buttons, labels, and formatting * Skip CloudWatch control messages in Health Check Fixes Graylog2/graylog-plugin-integrations#178 * Add comment * Add support for the "flow_log_" prefix Closes Graylog2/graylog-plugin-integrations#120 * Fixing lint errors * Return mock responses * DEV COMMIT * Remove unnecessary Row and Col now that we're rendering a sidebar * sidebar styles * Add checkbox for adding Flow Log field prefix * Auto-Setup Sidebar * lint * Adjust wording * Correct label for AWS Message Type * Consolidate Kinesis payload decompression logic * Agree TOS in Modal * Return actual stream name in create stream mock * Modal Styled * Don't include the policy name in response, since it's inline * Remove DEV COMMIT changes * Update header description * Fix failing unit test * Mock responses * Add default setting for prefix field * Add support for assuming an ARN role (#188) * Add support for assuming an ARN role Fixes Graylog2/graylog-plugin-integrations#29 * Add Assume Role ARN config field * Remove unneeded imports * Fix formatting * Record when users agree to auto-setup (#194) * Add log message recording that a user agreed to auto-setup * Remove unneeded import * Adding Permissions component to StepAuthorize * clear sidebar on unmount * Fixing select default value (#199) * Adding pattern to Stream Name input (#200) * Adding pattern to Stream Name input * Don't submit if field has error * Add API request to get permissions (#196) * Add request to retireve Kinesis permissions Includes permissions for both the full setup and the auto-setup. Closes #174 * Alphabetize permissions, add comments, remove unused imports * Add unit test * Make auto-setup unit test pass when mocking enabled * Remove unneeded create log group/stream permissions * Minor code clean up (#201) * Minor code clean up * Fix typo * Make auto-setup unit test pass when mocking enabled * Bump version to 3.2.0-SNAPSHOT * Bump version to 3.2.0-SNAPSHOT * Adding Policies API * bad comment * Adjust permissions wording

* Initial commit * Add AWS SDK v2 dependency * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Add Kinesis SDK * General UI Skeleton (#75) * Simple Clickthrough without API * Clickable Skeleton All Steps * Feedback * Lint * AWS Cloudwatch backend API calls Add initial backend API calls for CloudWatch integration: getRegions, getLogGroups, getStreams, retrieveKinesisMessages, healthCheck * AWS Cloud Watch services and resources (#24) Adds beginnings of API endpoint and Kinesis/Cloudwatch services Includes a structure that we will continue to build from. * Rework organization of classes for unified structure The goal is to establish some structure that we can implement AWS API calls within. There's now one resource (for API calls), one service (for business logic), and one AWSClient (for all AWS/API SDK interactions). * Cleanup and add comments * Cleanup code around log message auto-detection * Aws cloudwatch client (#31) * Add CloudWatchService class * Add AWSConfigSettings class * Add UserCredentials class * Add temporary Main class * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Update AWSConfigSettings class * Update CloudWatchService class * Update temporary Main class * Fix inject error in AWSResource * Fix merge conflicts * Restructure Resources and Service classes * Add getRegion into AWSResource (#43) * Add getRegion into AWSResource * Update api paths * Increase max retry limit for stream get to 1000 Runaway looping Stopping at 100 is probably to small. * Use underscores instead of camel case for json names It is a general project standard to use underscores. * Add paging functionality to getLogGroupNames (#53) * Add paging functionality to getLogGroupNames * Add unit test for CloudWatch log groups * Add comments in log group name unit test * Code clean up and remove unneeded code * Revert unintended change from unit test commit * Add retrievelogs for Kinesis Healthcheck (#76) * Refactor getKinesisStreams in KinesisService * Add validCredentials method in AWSService class * Temporary Main class added to test putting records into a Kinesis stream * Add retrieveKinesisLogs in KinesisService * Update temporary Main class * Update retrieveKinesisLogs to loop through shard iterators * Update Main class * Update pom file * Update validateCredentials in AWSService class * Add createKinesisClient method in KinesisService class * Add testGetStreamCredentials and update testGetStreams * Healthcheck merge (#81) * empty commit to push branch * Add Kinesis Healthcheck (#45) * Improve organization for Flow Log message detection * Improve Flow Log test It now tests for a message with too many and too few spaces. * Add TODOs for healthCheck method logic * Add beginnings of Kinesis healthChecker This will pull a establish a Kinesis subscription and pull a single message from a Kinesis stream. * Fix failing unit test * Continue developing KinesisHealthCheck - Remove unneeded metric tracking - Remove extra parsing logic (this object should just hand back payload and not do any parsing) - Improve application name handling - Add comments * Add detection logic for raw vs. CloudWatch logs * Remove KinesisHealthCheck class The KinesisConsumer does not work well for the health check (designed for realtime processing, takes a long time to start, cannot detect empty stream, and is really hard to use in a quick API request). Now, we're planning to directly retrieve the messages using the Kinesis client. This is the most straight-forward thing We might revisit this later. * Fix JSON parsing of Kinesis CloudWatch subscription record Parse the record just as was done in the existing AWS plugin. The logic now includes autodetection of compressed/vs not compressed. Mock Kinesis CloudWatch subscription record included for testing purposes. * Add CloudWatch logs codec and tests from existing AWS plugin * Parse Flow Log message into object * Load appropriate codec during healthCheck process When the message type is detected, load the respective codec for that message type. * Parse message with appropriate codec Once the log message type is detected, then the codec is looked up. The message is then parsed with the codec. * Supply log group name with the response * Improve comments, logging, and error checking The log group name is now also included in the response. * Add Flow Log codec test * Use AutoValue for CloudWatchLogEntry class * Use AutoValue for all remaining CloudWatch value classes * Cleanup merge conflicts after rebasing * Specify constants for all JsonProperty annotations * Delete uneeded KinesisDTO All data will be stored in the input * Establish a base AWSRequest JSON class * Fix Guice injection error for KinesisService * Add sample cURL command for healthCheck method A similar cURL command will be used for other methods, so that it is clear how the UI will use them. * Remove unneeded Kinesis Client 1.x dependency * Add formatted message summary in the Health Check response * Cleanup formatting and TODOs * Minor cleanup after rebasing and merging * Fix failing unit tests * Integrate AWS Health Check with Kinesis record retrieval (#86) * Fix incorrect pass of AWS key instead of secret Also improve comments for fake message retrieval with TODOs. * Update and connect retrieveRecords * Add handleCompressedMessages method * Delete temporary main class * Update retrieveRecords to only return sample size * Update KinesisService for healthCheck to function properly * Add unit test for selecting random record * Add unit test for retrieveRecords * Add Available Services API call (#82) * Resolves #50: Add Available Services API call * Add a test * Add missing spaces, change Amazon > AWS * Backend code cleanup (#90) * Pass region and AWS credentials with AWS API requests (#92) * Require POST object containing region and credentials for all requests Specifically adds a POST body requirement for the getKinesisSteams and getLogGroupNames methods. * Use snake_case for paths * Update region api call (#110) * Migrate Regions request from a list to a full response object with total * Update Region API call to include label and value * Reformat code * Aws permissions check (#113) * Add AWSPermissions class and update AWSResource with permission checks * Rename and remove permissions in AWSPermissions * Register AWSPermissions in IntegrationsModule * Remove space between methods * Save AWS input API endpoint (#93) * First-pass structure for saving AWS input * Add more structure for general AWS input - Add a type enum to differentiate the various types of log messages that are possible. - Add metacodec that will eventually differentiate between the types of log messages. - Fill configuration values when saving the input. * Resolve merge conflicts after rebasing over latest aws branch * Consolidate log type detection and input type identification There's no longer a need to use two enums for this. Also added healthCheck tests covering all message types: flow log, raw Cloud Watch and raw Kinesis. * Clean up saveInput request parameters and handling * Fix invalid type specification that was preventing input save * Add unit test for saving input * Fix incorrectly specified arguments * Minor cleanup * Cleanup for PR review * Remove uneeded log statements * Make save AWS input path and description more specific * Indicate that the save request is specifically for Kinesis In the future, each type of AWS input will likely require it's own request object and endpoint due to the fact that unique fields will probably be required for each. * AWSMessageType cleanup * Remove uneeded isFlowLog, isRaw methods. * Remove invalid AWSMessageType.Source.CLOUD_WATCH enum value. Messages are always read from Kinesis, and therefore the source is always Kinesis. Source is meant to differentiate messages from Kinesis and S3 for example. * Improve comments for AWSMessageType.Source enum class and method. * Remove typo * Generify the create AWS integration endpoint The naming, description, and comment now reflect that a generic AWS input is being created. * Return InputSummary response entity for AWS input creation request Also remove unneeded AWSResourceTest * More cleanup of healthCheck after input creation changes - Remove unneeded log_group field for health_check request. Resolves #108 - Add Kinesis stream name as a field in both raw and CloudWatch messages This change is lumped in with the other changes related to saving the input, since lots of healthCheck changes were already made there. This fixes some problems, so might as well have these improvements included with the review. * Clarify that AWSMetaCodec is a general AWS codec This class no longer erroneously extends AbstractKinesisCodec, which was only intended for Kinesis-specific codecs. * Use DateTime instead of long in KinesisLogEntry It turns out that Kinesis Record objects do have an arrival ime Instant timestamp. This is now being used instead of just using the date/time when the message was read by Graylog. * Add missing Guice annotation that were preventing server startup * Return JSON message field list in response from Health Check (#114) * Return JSON fields in healthCheck response There is still an issue with serializing the DateTime timestamp (serializing the deep object tree). This will be addressed separately. * Adapt unit test timestamp check to object value Health Check now returns a map of actual field values, so the test needs to check the JSON value. * Fix incorrect codec specified for Flow Logs, add tests * Fix incorrectly named field * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn * Advanced Kinesis Options * Cleanup * currently busted toggle * redo state handling * Remove artifact * removing commented code * removing unnecessary functions * pr feedback * Rebase conflicts * Fix several issues from PR review * "Final Review" layout (#83) * General Content * Review view layout complete * Cleanup * Update utils file name and export * Change LogFlow > Flow Log * default_values.js files * Route constants (#111) * Route constants * Fixing conflicts * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn Route constants Fixing conflicts * Correct import path * Changing route object * Adjust Kinesis setup wording * Remove the unneeded * Fixing up rebase artifacts from squash of irrelevant commits * AWS lockfile * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * Setting up necessary hooks * Removing input validation steps to reduce PR * Rebasing * Rename StyledForm * DEFAULT_VALUES * cleanup hook returns * Moving some reducers to state and breaking apart the rest * Remove functions from Input values * Steps function cleanup * Remove commented import * FormDataProvider and Context * Steps Context & provider * log and advancedoptions moved to context and provider * FormWrap functional component * renaming things * lint error * PR feedback * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * error and dirty states * Error on submit * Update validation to new state hooks * remove unneeded defaults * Output error messages * text key instead of password * Remove field validation * Update comment blocks (#137) * Update comment blocks * Remove minor typos * Update version to 3.1.0-beta.2-SNAPSHOT * Implement AWS codec selector with tests (#116) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Remove erroneous comment * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump package.json version to 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * Authorize and Streams API * All API calls except Save * reverting to hooks for api calls * Regions hook * Streams and Logs * streams useFetch * log sample * generic loading states * move regions * adding some documentation * Adding all the files * removing uneeded * cleanup * ApiRoutes * import proper path * Changing up useFetch * need the values... * lint * Removed unneeded inline cURL API documentation (#143) No need for this docs duplication. The System > Nodes > API Browser already documents them. * Use standard field names for AWS key and secret (#144) * Use standard naming for AWS secret and access key ids for Save Input Use these standard names for AWS key and secret from the Save Input AWS endpoint (From the AWSRequest interface). All other AWS API requests should be using these already. `aws_access_key_id` `aws_secret_access_key` * Use central constant for Region also * Fix incorrectly set Region property name * Fix access and secret key method names to match the AWSRequest interface * rename useFetch file * better var naming in kinesisstreams: * Update useFetch props * AWS transport selector with tests (#117) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Remove duplicate AWSTestingUtils class * Add missing permissions for sample AWS policy (#122) * Migrate AWS policy to autovalue object This avoids embedding JSON in a Java constant, which is error prone. * Add missing permissions Added: - kinesis:ListStreams - logs:DescribeLogGroups * Define JSON policy field order that matches AWS examples * Return the policy JSON string to the UI as string This is consistent with how this was done before. The UI can then format and display this to the user on the Available Services page. This is the policy that the user will create the AWS user for the AWS integration with. * Specify the correct AWS policy version * Change AWS Policy to a list * Add missing permissions Based on this [sample KCL code](aws/aws-sdk-java-v2#1214 (comment)) * Update version to 3.1.0-beta.2-SNAPSHOT * Remove uneeded import * Add Available Services API response error documentation * Throw InternalServerErrorException instead of JSON exception This is a bit cleaner from the API consumption side. A nice short error message is now returned instead of an obscure JSON error. * API Save Input (#145) * Save after Review and redirect to Inputs * Save functional * Adding match function to pluginmanifest * custom component * Update version to 3.1.0-beta.3-SNAPSHOT * Update version to 3.1.0-beta.3-SNAPSHOT * Aws kinesis newstream (#129) * Add create new Kinesis stream method * Add API call for createNewKinesisStream and KinesisNewStream classes * Update createNewKinesisStream * Update version to 3.1.0-beta.2-SNAPSHOT * Add error logging to createNewKinesisStream method * Add unit test for createNewKinesisStream method * Update error logging for createNewKinesisStream * Update testCreateNewKinesisStream * Remove shard count variable and set default value * Update testCreateNewKinesisStream * Code clean up * Update api comment blocks * Code clean up * Update error handling in createNewKinesisStream * Update unit test testCreateNewKinesisStream * Unload Confirmation Dialog (#148) * Playing with onunload * use ConfirmLeaveDialog common component * Question * Update version to 3.1.0-beta.3-SNAPSHOT * lint * cleanup * Don't alert on last step. Save call failed if desc wasn't available * Remove emotion and use styled-components (#157) * Remove emotion and use styled-components * import styled-components * Move title and description to FormWrap for consistent layout * Handle validation per-step * fix aws key h4x * styled validatedinput * enable health check and review forms * Starting Error Handling * Output API Errors * remove some more debugging * Validate API Key & Secret and output styled errors * Error message if stream doesn't contain any logs * Return a 4xx response for an unsuccessful health check * Save API Error Handling * Remove `success` flag from HealthCheckResponse Failure is now tracked by throwing a `BadRequestException` resulting in an `ApiError` json response * Throw `BadRequestException` if no Kinesis streams are found * Throw `BadRequestException` if no Kinesis CloudWatch groups are found * Error message styling * wrap up api errors and adding styles * rebasing and cleanup * trim input values * remove defaults * Create input with togglable masking * Include stream name in error * Port of existing KinesisTransport (#140) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Remove unused imports and fix formatting * Update version to 3.1.0-beta.3-SNAPSHOT * Merge branch 'aws' into aws-transport-kinesis and resolve conflicts # Conflicts: # src/main/java/org/graylog/integrations/aws/service/KinesisService.java * Upgrade to latest Kinesis Client version (#151) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Update Kinesis Client version * Add Kinesis client library v2 Consumer sample * Migrate KinesisConsumer to Kinesis Client Library v2 * Remove unused imports and fix formatting * Finish migration of Kinesis client to new version * Complete upgrade to Kinesis Client Library v2 * Bump KCL version * Migrate shard processor to its own class * Bump version to 3.1.0-beta.3-SNAPSHOT * Add batch size limit * Code cleanup * Remove unneeded throttle time limit reached shutdown * Remove kinesis_max_throttled_wait_ms save input field This field is no longer needed, since the new Kinesis Consumer appears to correctly handle longer throttling and pausing in `processRecords` without making the consumer unhealthy. * Temporarily remove Assume Role Arn auth This will be added back later. See #29 * Adjust logging levels * Remove sample class * Ignore unmapped properties due to removed max_throttled_wait field See Graylog2/graylog-plugin-integrations#156 * Update Throttling advanced option (#169) * Update Throttling advanced option * messages/ms * Remove throttle number * Update label * Shells/mocks for Kinesis Auto setup API endpoints (#167) * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Update version to 3.1.0-beta.4-SNAPSHOT * Code cleanup - Remove unneeded imports - Replace hyphens with underscores * Add `aws` to root URL path for consistency All other AWS routes are prefixed with this, so do the same here. * Change subscription path Change `add_subscription` route to `create_subscription` * Fix linter warning was introduced during merge conflict resolution. * Add RolePermissions for Kinesis stream (#163) * Add checkKinesisStreamStatus and setRolePermissions method * Add getNewRolePermissions and autoKinesisPermissionRequired method * Rename autoKinesisPermissionsRequired method and add error handling * Code clean up * Refactor autoSetup, split up creating a role and attaching a policy * Add error handling for creating a new role * Update autoKinesisPermissionsRequired logic and add comment block * Update version to 3.1.0-beta.4-SNAPSHOT * Delete checkKinesisStreamStatus method * Update error logging messages * Add temporary main class for testing, update error handling logic * Delete temporary main, update autoKinesisPermissionsRequired * Change exceptions thrown and code clean up * Update version to 3.1.0-rc.2-SNAPSHOT * Add addSubscriptionFilter to CloudwatchService (#160) * Add addSubscriptionFilter method to KinesisService * Move addSubscriptionFilter method to CloudWatchService * Code clean up, change exception errors * Bump version to 3.1.0-rc.3-SNAPSHOT * Kinesis Auto-setup (#186) * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump js-yaml from 3.13.0 to 3.13.1 Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 3.13.0 to 3.13.1. - [Release notes](https://github.com/nodeca/js-yaml/releases) - [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md) - [Commits](nodeca/js-yaml@3.13.0...3.13.1) Signed-off-by: dependabot[bot] <support@github.com> * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * [Security] Bump lodash from 4.17.11 to 4.17.14 (#131) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.11 to 4.17.14. **This update includes security fixes.** - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.11...4.17.14) Signed-off-by: dependabot-preview[bot] <support@dependabot.com> * Bump package.json version to 3.1.0-beta.2 * [graylog-plugin-integrations] prepare release 3.1.0-beta.2 * Bump package.json version to 3.1.0-beta.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Bump package.json version to 3.1.0-beta.3 * [graylog-plugin-integrations] prepare release 3.1.0-beta.3 * Bump package.json version to 3.1.0-beta.4-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Render Setup screen * cleanup * Allowing toggle to get from Streams to Setup * Toggle back to Streams & cleanup Group on Region change * Add Kinesis auto-setup routes * Add initial `create_stream` request * Define fetchs for auto Kinesis setup * Finish create_stream request * Add all auto-setup API requests * Specify actual region and request properties * Improve presentation for transition to automated Kinesis setup * Improve on-screen documentation for the automatic Kinesis setup * Bump package.json version to 3.1.0-rc.1 * [graylog-plugin-integrations] prepare release 3.1.0-rc.1 * Bump package.json version to 3.1.0-rc.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Update version to 3.1.0-rc.2-SNAPSHOT * Add KinesisSetupStep shell * Add KinesisSetupSteps container * Update createKinesisStream to return streamArn * Update createKinesisStream and testCreateNewKinesisStream * Add edge case error handling to createNewKinesisStream * Add preliminary code for KinesisFullSetup method * Use await for synchronous auto setup steps * Migrate to map for auto-setup state management * Render auto-setup progress to UI * Include success field and initial delay for testing * Delete KinesisFullSetup method * Migrate Kinesis setup to map to allow setup step iteration * Refactor autoKinesisPermissions parameters and return response * Rework setup logic to use individual state fields Individual state fields are needed for each component to update independently. * Add progress icons * Improve icon and step styles * Add auto-setup error handling * Refactor addSubscriptionFilter to return CreateLogSubscriptionResponse * Update api calls in KinesisFullSetup * Update createNewKinesisStream and createPolicies api call * Add flow control for setup steps * Clean up api methods * Add mocking for UI testing Also remove full setup, since the individual steps will be used. * Sync field mappings between UI and backend * Add auto-setup integration unit test * Add auto-setup test assertions * Rename `explanation` -> `result` * Display result to the user for each auto-setup step * Use unique role ane role policy names Since these are only needed in the auto-setup flow, we can automatically generate and assign them (as long as we provide them to the user in the response). Customizing these names is possible, and can be considered in the future. * Adjustments to get auto-setup working * Add support subscription retries to resolve IAM eventual consistency IAM sometimes takes 10+ seconds to initialize, so retrying allows the subscription attempt to be attempted multiple times, so that once IAM is available, then the subscription will be created successfully. * Cleanup retry logic * Add new page for auto-setup steps * Adjust wording * Add auto-setup TODO for adding warning * TODOs and cleanup * Formatting cleanup * Support ability to continue adding input after auto-setup * Bump package.json version to 3.1.0-rc.2 * [graylog-plugin-integrations] prepare release 3.1.0-rc.2 * Bump package.json version to 3.1.0-rc.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Disable Continue Setup button while auto-setup is in progress * Include warning that messages might take time to arrive in new stream * Require explicit agreement from user before starting auto-setup * Bump version to 3.1.0-rc.3-SNAPSHOT * Adjustments to auto-setup buttons, labels, and formatting * Skip CloudWatch control messages in Health Check Fixes Graylog2/graylog-plugin-integrations#178 * Add comment * Add support for the "flow_log_" prefix Closes Graylog2/graylog-plugin-integrations#120 * Fixing lint errors * Return mock responses * DEV COMMIT * Remove unnecessary Row and Col now that we're rendering a sidebar * sidebar styles * Add checkbox for adding Flow Log field prefix * Auto-Setup Sidebar * lint * Adjust wording * Correct label for AWS Message Type * Consolidate Kinesis payload decompression logic * Agree TOS in Modal * Return actual stream name in create stream mock * Modal Styled * Don't include the policy name in response, since it's inline * Remove DEV COMMIT changes * Update header description * Fix failing unit test * Mock responses * Add default setting for prefix field * Kinesis auto-setup Health Check redirect * health check clear sidebar * Add support for assuming an ARN role (#188) * Add support for assuming an ARN role Fixes Graylog2/graylog-plugin-integrations#29 * Add Assume Role ARN config field * Remove unneeded imports * Fix formatting * Countdown and Retry button * CheckAgain * Record when users agree to auto-setup (#194) * Add log message recording that a user agreed to auto-setup * Remove unneeded import * Fixing select default value (#199) * Adding pattern to Stream Name input (#200) * Adding pattern to Stream Name input * Don't submit if field has error * Add API request to get permissions (#196) * Add request to retireve Kinesis permissions Includes permissions for both the full setup and the auto-setup. Closes #174 * Alphabetize permissions, add comments, remove unused imports * Add unit test * Make auto-setup unit test pass when mocking enabled * Remove unneeded create log group/stream permissions * Minor code clean up (#201) * Minor code clean up * Fix typo * Make auto-setup unit test pass when mocking enabled * Bump version to 3.2.0-SNAPSHOT * Bump version to 3.2.0-SNAPSHOT * Starting testing * Linting Fix (#212) * Revert "frontend merge conflicts" This reverts commit fa5c2775a377e0585b1d771cec552f8ae0fd6de6, reversing changes made to 207fca0533051521e5cad58b39b43f15e5e44179. * remove test

* Initial commit * Add AWS SDK v2 dependency * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Add Kinesis SDK * General UI Skeleton (#75) * Simple Clickthrough without API * Clickable Skeleton All Steps * Feedback * Lint * AWS Cloudwatch backend API calls Add initial backend API calls for CloudWatch integration: getRegions, getLogGroups, getStreams, retrieveKinesisMessages, healthCheck * AWS Cloud Watch services and resources (#24) Adds beginnings of API endpoint and Kinesis/Cloudwatch services Includes a structure that we will continue to build from. * Rework organization of classes for unified structure The goal is to establish some structure that we can implement AWS API calls within. There's now one resource (for API calls), one service (for business logic), and one AWSClient (for all AWS/API SDK interactions). * Cleanup and add comments * Cleanup code around log message auto-detection * Aws cloudwatch client (#31) * Add CloudWatchService class * Add AWSConfigSettings class * Add UserCredentials class * Add temporary Main class * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Update AWSConfigSettings class * Update CloudWatchService class * Update temporary Main class * Fix inject error in AWSResource * Fix merge conflicts * Restructure Resources and Service classes * Add getRegion into AWSResource (#43) * Add getRegion into AWSResource * Update api paths * Increase max retry limit for stream get to 1000 Runaway looping Stopping at 100 is probably to small. * Use underscores instead of camel case for json names It is a general project standard to use underscores. * Add paging functionality to getLogGroupNames (#53) * Add paging functionality to getLogGroupNames * Add unit test for CloudWatch log groups * Add comments in log group name unit test * Code clean up and remove unneeded code * Revert unintended change from unit test commit * Add retrievelogs for Kinesis Healthcheck (#76) * Refactor getKinesisStreams in KinesisService * Add validCredentials method in AWSService class * Temporary Main class added to test putting records into a Kinesis stream * Add retrieveKinesisLogs in KinesisService * Update temporary Main class * Update retrieveKinesisLogs to loop through shard iterators * Update Main class * Update pom file * Update validateCredentials in AWSService class * Add createKinesisClient method in KinesisService class * Add testGetStreamCredentials and update testGetStreams * Healthcheck merge (#81) * empty commit to push branch * Add Kinesis Healthcheck (#45) * Improve organization for Flow Log message detection * Improve Flow Log test It now tests for a message with too many and too few spaces. * Add TODOs for healthCheck method logic * Add beginnings of Kinesis healthChecker This will pull a establish a Kinesis subscription and pull a single message from a Kinesis stream. * Fix failing unit test * Continue developing KinesisHealthCheck - Remove unneeded metric tracking - Remove extra parsing logic (this object should just hand back payload and not do any parsing) - Improve application name handling - Add comments * Add detection logic for raw vs. CloudWatch logs * Remove KinesisHealthCheck class The KinesisConsumer does not work well for the health check (designed for realtime processing, takes a long time to start, cannot detect empty stream, and is really hard to use in a quick API request). Now, we're planning to directly retrieve the messages using the Kinesis client. This is the most straight-forward thing We might revisit this later. * Fix JSON parsing of Kinesis CloudWatch subscription record Parse the record just as was done in the existing AWS plugin. The logic now includes autodetection of compressed/vs not compressed. Mock Kinesis CloudWatch subscription record included for testing purposes. * Add CloudWatch logs codec and tests from existing AWS plugin * Parse Flow Log message into object * Load appropriate codec during healthCheck process When the message type is detected, load the respective codec for that message type. * Parse message with appropriate codec Once the log message type is detected, then the codec is looked up. The message is then parsed with the codec. * Supply log group name with the response * Improve comments, logging, and error checking The log group name is now also included in the response. * Add Flow Log codec test * Use AutoValue for CloudWatchLogEntry class * Use AutoValue for all remaining CloudWatch value classes * Cleanup merge conflicts after rebasing * Specify constants for all JsonProperty annotations * Delete uneeded KinesisDTO All data will be stored in the input * Establish a base AWSRequest JSON class * Fix Guice injection error for KinesisService * Add sample cURL command for healthCheck method A similar cURL command will be used for other methods, so that it is clear how the UI will use them. * Remove unneeded Kinesis Client 1.x dependency * Add formatted message summary in the Health Check response * Cleanup formatting and TODOs * Minor cleanup after rebasing and merging * Fix failing unit tests * Integrate AWS Health Check with Kinesis record retrieval (#86) * Fix incorrect pass of AWS key instead of secret Also improve comments for fake message retrieval with TODOs. * Update and connect retrieveRecords * Add handleCompressedMessages method * Delete temporary main class * Update retrieveRecords to only return sample size * Update KinesisService for healthCheck to function properly * Add unit test for selecting random record * Add unit test for retrieveRecords * Add Available Services API call (#82) * Resolves #50: Add Available Services API call * Add a test * Add missing spaces, change Amazon > AWS * Backend code cleanup (#90) * Pass region and AWS credentials with AWS API requests (#92) * Require POST object containing region and credentials for all requests Specifically adds a POST body requirement for the getKinesisSteams and getLogGroupNames methods. * Use snake_case for paths * Update region api call (#110) * Migrate Regions request from a list to a full response object with total * Update Region API call to include label and value * Reformat code * Aws permissions check (#113) * Add AWSPermissions class and update AWSResource with permission checks * Rename and remove permissions in AWSPermissions * Register AWSPermissions in IntegrationsModule * Remove space between methods * Save AWS input API endpoint (#93) * First-pass structure for saving AWS input * Add more structure for general AWS input - Add a type enum to differentiate the various types of log messages that are possible. - Add metacodec that will eventually differentiate between the types of log messages. - Fill configuration values when saving the input. * Resolve merge conflicts after rebasing over latest aws branch * Consolidate log type detection and input type identification There's no longer a need to use two enums for this. Also added healthCheck tests covering all message types: flow log, raw Cloud Watch and raw Kinesis. * Clean up saveInput request parameters and handling * Fix invalid type specification that was preventing input save * Add unit test for saving input * Fix incorrectly specified arguments * Minor cleanup * Cleanup for PR review * Remove uneeded log statements * Make save AWS input path and description more specific * Indicate that the save request is specifically for Kinesis In the future, each type of AWS input will likely require it's own request object and endpoint due to the fact that unique fields will probably be required for each. * AWSMessageType cleanup * Remove uneeded isFlowLog, isRaw methods. * Remove invalid AWSMessageType.Source.CLOUD_WATCH enum value. Messages are always read from Kinesis, and therefore the source is always Kinesis. Source is meant to differentiate messages from Kinesis and S3 for example. * Improve comments for AWSMessageType.Source enum class and method. * Remove typo * Generify the create AWS integration endpoint The naming, description, and comment now reflect that a generic AWS input is being created. * Return InputSummary response entity for AWS input creation request Also remove unneeded AWSResourceTest * More cleanup of healthCheck after input creation changes - Remove unneeded log_group field for health_check request. Resolves #108 - Add Kinesis stream name as a field in both raw and CloudWatch messages This change is lumped in with the other changes related to saving the input, since lots of healthCheck changes were already made there. This fixes some problems, so might as well have these improvements included with the review. * Clarify that AWSMetaCodec is a general AWS codec This class no longer erroneously extends AbstractKinesisCodec, which was only intended for Kinesis-specific codecs. * Use DateTime instead of long in KinesisLogEntry It turns out that Kinesis Record objects do have an arrival ime Instant timestamp. This is now being used instead of just using the date/time when the message was read by Graylog. * Add missing Guice annotation that were preventing server startup * Return JSON message field list in response from Health Check (#114) * Return JSON fields in healthCheck response There is still an issue with serializing the DateTime timestamp (serializing the deep object tree). This will be addressed separately. * Adapt unit test timestamp check to object value Health Check now returns a map of actual field values, so the test needs to check the JSON value. * Fix incorrect codec specified for Flow Logs, add tests * Fix incorrectly named field * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn * Advanced Kinesis Options * Cleanup * currently busted toggle * redo state handling * Remove artifact * removing commented code * removing unnecessary functions * pr feedback * Rebase conflicts * Fix several issues from PR review * "Final Review" layout (#83) * General Content * Review view layout complete * Cleanup * Update utils file name and export * Change LogFlow > Flow Log * default_values.js files * Route constants (#111) * Route constants * Fixing conflicts * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn Route constants Fixing conflicts * Correct import path * Changing route object * Adjust Kinesis setup wording * Remove the unneeded * Fixing up rebase artifacts from squash of irrelevant commits * AWS lockfile * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * Setting up necessary hooks * Removing input validation steps to reduce PR * Rebasing * Rename StyledForm * DEFAULT_VALUES * cleanup hook returns * Moving some reducers to state and breaking apart the rest * Remove functions from Input values * Steps function cleanup * Remove commented import * FormDataProvider and Context * Steps Context & provider * log and advancedoptions moved to context and provider * FormWrap functional component * renaming things * lint error * PR feedback * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * error and dirty states * Error on submit * Update validation to new state hooks * remove unneeded defaults * Output error messages * text key instead of password * Remove field validation * Update comment blocks (#137) * Update comment blocks * Remove minor typos * Update version to 3.1.0-beta.2-SNAPSHOT * Implement AWS codec selector with tests (#116) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Remove erroneous comment * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump package.json version to 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * Authorize and Streams API * All API calls except Save * reverting to hooks for api calls * Regions hook * Streams and Logs * streams useFetch * log sample * generic loading states * move regions * adding some documentation * Adding all the files * removing uneeded * cleanup * ApiRoutes * import proper path * Changing up useFetch * need the values... * lint * Removed unneeded inline cURL API documentation (#143) No need for this docs duplication. The System > Nodes > API Browser already documents them. * Use standard field names for AWS key and secret (#144) * Use standard naming for AWS secret and access key ids for Save Input Use these standard names for AWS key and secret from the Save Input AWS endpoint (From the AWSRequest interface). All other AWS API requests should be using these already. `aws_access_key_id` `aws_secret_access_key` * Use central constant for Region also * Fix incorrectly set Region property name * Fix access and secret key method names to match the AWSRequest interface * rename useFetch file * better var naming in kinesisstreams: * Update useFetch props * AWS transport selector with tests (#117) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Remove duplicate AWSTestingUtils class * Add missing permissions for sample AWS policy (#122) * Migrate AWS policy to autovalue object This avoids embedding JSON in a Java constant, which is error prone. * Add missing permissions Added: - kinesis:ListStreams - logs:DescribeLogGroups * Define JSON policy field order that matches AWS examples * Return the policy JSON string to the UI as string This is consistent with how this was done before. The UI can then format and display this to the user on the Available Services page. This is the policy that the user will create the AWS user for the AWS integration with. * Specify the correct AWS policy version * Change AWS Policy to a list * Add missing permissions Based on this [sample KCL code](aws/aws-sdk-java-v2#1214 (comment)) * Update version to 3.1.0-beta.2-SNAPSHOT * Remove uneeded import * Add Available Services API response error documentation * Throw InternalServerErrorException instead of JSON exception This is a bit cleaner from the API consumption side. A nice short error message is now returned instead of an obscure JSON error. * API Save Input (#145) * Save after Review and redirect to Inputs * Save functional * Adding match function to pluginmanifest * custom component * Update version to 3.1.0-beta.3-SNAPSHOT * Update version to 3.1.0-beta.3-SNAPSHOT * Aws kinesis newstream (#129) * Add create new Kinesis stream method * Add API call for createNewKinesisStream and KinesisNewStream classes * Update createNewKinesisStream * Update version to 3.1.0-beta.2-SNAPSHOT * Add error logging to createNewKinesisStream method * Add unit test for createNewKinesisStream method * Update error logging for createNewKinesisStream * Update testCreateNewKinesisStream * Remove shard count variable and set default value * Update testCreateNewKinesisStream * Code clean up * Update api comment blocks * Code clean up * Update error handling in createNewKinesisStream * Update unit test testCreateNewKinesisStream * Unload Confirmation Dialog (#148) * Playing with onunload * use ConfirmLeaveDialog common component * Question * Update version to 3.1.0-beta.3-SNAPSHOT * lint * cleanup * Don't alert on last step. Save call failed if desc wasn't available * Remove emotion and use styled-components (#157) * Remove emotion and use styled-components * import styled-components * Move title and description to FormWrap for consistent layout * Handle validation per-step * fix aws key h4x * styled validatedinput * enable health check and review forms * Starting Error Handling * Output API Errors * remove some more debugging * Validate API Key & Secret and output styled errors * Error message if stream doesn't contain any logs * Return a 4xx response for an unsuccessful health check * Save API Error Handling * Remove `success` flag from HealthCheckResponse Failure is now tracked by throwing a `BadRequestException` resulting in an `ApiError` json response * Throw `BadRequestException` if no Kinesis streams are found * Throw `BadRequestException` if no Kinesis CloudWatch groups are found * Error message styling * wrap up api errors and adding styles * rebasing and cleanup * trim input values * remove defaults * Create input with togglable masking * Include stream name in error * Port of existing KinesisTransport (#140) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Remove unused imports and fix formatting * Update version to 3.1.0-beta.3-SNAPSHOT * Merge branch 'aws' into aws-transport-kinesis and resolve conflicts # Conflicts: # src/main/java/org/graylog/integrations/aws/service/KinesisService.java * Upgrade to latest Kinesis Client version (#151) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Update Kinesis Client version * Add Kinesis client library v2 Consumer sample * Migrate KinesisConsumer to Kinesis Client Library v2 * Remove unused imports and fix formatting * Finish migration of Kinesis client to new version * Complete upgrade to Kinesis Client Library v2 * Bump KCL version * Migrate shard processor to its own class * Bump version to 3.1.0-beta.3-SNAPSHOT * Add batch size limit * Code cleanup * Remove unneeded throttle time limit reached shutdown * Remove kinesis_max_throttled_wait_ms save input field This field is no longer needed, since the new Kinesis Consumer appears to correctly handle longer throttling and pausing in `processRecords` without making the consumer unhealthy. * Temporarily remove Assume Role Arn auth This will be added back later. See #29 * Adjust logging levels * Remove sample class * Ignore unmapped properties due to removed max_throttled_wait field See Graylog2/graylog-plugin-integrations#156 * Update Throttling advanced option (#169) * Update Throttling advanced option * messages/ms * Remove throttle number * Update label * Shells/mocks for Kinesis Auto setup API endpoints (#167) * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Update version to 3.1.0-beta.4-SNAPSHOT * Code cleanup - Remove unneeded imports - Replace hyphens with underscores * Add `aws` to root URL path for consistency All other AWS routes are prefixed with this, so do the same here. * Change subscription path Change `add_subscription` route to `create_subscription` * Fix linter warning was introduced during merge conflict resolution. * Add RolePermissions for Kinesis stream (#163) * Add checkKinesisStreamStatus and setRolePermissions method * Add getNewRolePermissions and autoKinesisPermissionRequired method * Rename autoKinesisPermissionsRequired method and add error handling * Code clean up * Refactor autoSetup, split up creating a role and attaching a policy * Add error handling for creating a new role * Update autoKinesisPermissionsRequired logic and add comment block * Update version to 3.1.0-beta.4-SNAPSHOT * Delete checkKinesisStreamStatus method * Update error logging messages * Add temporary main class for testing, update error handling logic * Delete temporary main, update autoKinesisPermissionsRequired * Change exceptions thrown and code clean up * Update version to 3.1.0-rc.2-SNAPSHOT * Add addSubscriptionFilter to CloudwatchService (#160) * Add addSubscriptionFilter method to KinesisService * Move addSubscriptionFilter method to CloudWatchService * Code clean up, change exception errors * Bump version to 3.1.0-rc.3-SNAPSHOT * Kinesis Auto-setup (#186) * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump js-yaml from 3.13.0 to 3.13.1 Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 3.13.0 to 3.13.1. - [Release notes](https://github.com/nodeca/js-yaml/releases) - [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md) - [Commits](nodeca/js-yaml@3.13.0...3.13.1) Signed-off-by: dependabot[bot] <support@github.com> * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * [Security] Bump lodash from 4.17.11 to 4.17.14 (#131) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.11 to 4.17.14. **This update includes security fixes.** - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.11...4.17.14) Signed-off-by: dependabot-preview[bot] <support@dependabot.com> * Bump package.json version to 3.1.0-beta.2 * [graylog-plugin-integrations] prepare release 3.1.0-beta.2 * Bump package.json version to 3.1.0-beta.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Bump package.json version to 3.1.0-beta.3 * [graylog-plugin-integrations] prepare release 3.1.0-beta.3 * Bump package.json version to 3.1.0-beta.4-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Render Setup screen * cleanup * Allowing toggle to get from Streams to Setup * Toggle back to Streams & cleanup Group on Region change * Add Kinesis auto-setup routes * Add initial `create_stream` request * Define fetchs for auto Kinesis setup * Finish create_stream request * Add all auto-setup API requests * Specify actual region and request properties * Improve presentation for transition to automated Kinesis setup * Improve on-screen documentation for the automatic Kinesis setup * Bump package.json version to 3.1.0-rc.1 * [graylog-plugin-integrations] prepare release 3.1.0-rc.1 * Bump package.json version to 3.1.0-rc.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Update version to 3.1.0-rc.2-SNAPSHOT * Add KinesisSetupStep shell * Add KinesisSetupSteps container * Update createKinesisStream to return streamArn * Update createKinesisStream and testCreateNewKinesisStream * Add edge case error handling to createNewKinesisStream * Add preliminary code for KinesisFullSetup method * Use await for synchronous auto setup steps * Migrate to map for auto-setup state management * Render auto-setup progress to UI * Include success field and initial delay for testing * Delete KinesisFullSetup method * Migrate Kinesis setup to map to allow setup step iteration * Refactor autoKinesisPermissions parameters and return response * Rework setup logic to use individual state fields Individual state fields are needed for each component to update independently. * Add progress icons * Improve icon and step styles * Add auto-setup error handling * Refactor addSubscriptionFilter to return CreateLogSubscriptionResponse * Update api calls in KinesisFullSetup * Update createNewKinesisStream and createPolicies api call * Add flow control for setup steps * Clean up api methods * Add mocking for UI testing Also remove full setup, since the individual steps will be used. * Sync field mappings between UI and backend * Add auto-setup integration unit test * Add auto-setup test assertions * Rename `explanation` -> `result` * Display result to the user for each auto-setup step * Use unique role ane role policy names Since these are only needed in the auto-setup flow, we can automatically generate and assign them (as long as we provide them to the user in the response). Customizing these names is possible, and can be considered in the future. * Adjustments to get auto-setup working * Add support subscription retries to resolve IAM eventual consistency IAM sometimes takes 10+ seconds to initialize, so retrying allows the subscription attempt to be attempted multiple times, so that once IAM is available, then the subscription will be created successfully. * Cleanup retry logic * Add new page for auto-setup steps * Adjust wording * Add auto-setup TODO for adding warning * TODOs and cleanup * Formatting cleanup * Support ability to continue adding input after auto-setup * Bump package.json version to 3.1.0-rc.2 * [graylog-plugin-integrations] prepare release 3.1.0-rc.2 * Bump package.json version to 3.1.0-rc.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Disable Continue Setup button while auto-setup is in progress * Include warning that messages might take time to arrive in new stream * Require explicit agreement from user before starting auto-setup * Bump version to 3.1.0-rc.3-SNAPSHOT * Adjustments to auto-setup buttons, labels, and formatting * Skip CloudWatch control messages in Health Check Fixes Graylog2/graylog-plugin-integrations#178 * Add comment * Add support for the "flow_log_" prefix Closes Graylog2/graylog-plugin-integrations#120 * Fixing lint errors * Return mock responses * DEV COMMIT * Remove unnecessary Row and Col now that we're rendering a sidebar * sidebar styles * Add checkbox for adding Flow Log field prefix * Auto-Setup Sidebar * lint * Adjust wording * Correct label for AWS Message Type * Consolidate Kinesis payload decompression logic * Agree TOS in Modal * Return actual stream name in create stream mock * Modal Styled * Don't include the policy name in response, since it's inline * Remove DEV COMMIT changes * Update header description * Fix failing unit test * Mock responses * Add default setting for prefix field * Add support for assuming an ARN role (#188) * Add support for assuming an ARN role Fixes Graylog2/graylog-plugin-integrations#29 * Add Assume Role ARN config field * Remove unneeded imports * Fix formatting * Record when users agree to auto-setup (#194) * Add log message recording that a user agreed to auto-setup * Remove unneeded import * Fixing select default value (#199) * Adding pattern to Stream Name input (#200) * Adding pattern to Stream Name input * Don't submit if field has error * Add API request to get permissions (#196) * Add request to retireve Kinesis permissions Includes permissions for both the full setup and the auto-setup. Closes #174 * Alphabetize permissions, add comments, remove unused imports * Add unit test * Make auto-setup unit test pass when mocking enabled * Remove unneeded create log group/stream permissions * Minor code clean up (#201) * Minor code clean up * Fix typo * Proposed wording changes We'll pick this up again in Hamburg * Bump version to 3.2.0-SNAPSHOT * Wording adjustments * Change "Setup" -> "Set Up" * Fix lint errors * More wording changes * Removing empty doc link

* Initial commit * Add AWS SDK v2 dependency * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Add Kinesis SDK * General UI Skeleton (#75) * Simple Clickthrough without API * Clickable Skeleton All Steps * Feedback * Lint * AWS Cloudwatch backend API calls Add initial backend API calls for CloudWatch integration: getRegions, getLogGroups, getStreams, retrieveKinesisMessages, healthCheck * AWS Cloud Watch services and resources (#24) Adds beginnings of API endpoint and Kinesis/Cloudwatch services Includes a structure that we will continue to build from. * Rework organization of classes for unified structure The goal is to establish some structure that we can implement AWS API calls within. There's now one resource (for API calls), one service (for business logic), and one AWSClient (for all AWS/API SDK interactions). * Cleanup and add comments * Cleanup code around log message auto-detection * Aws cloudwatch client (#31) * Add CloudWatchService class * Add AWSConfigSettings class * Add UserCredentials class * Add temporary Main class * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Update AWSConfigSettings class * Update CloudWatchService class * Update temporary Main class * Fix inject error in AWSResource * Fix merge conflicts * Restructure Resources and Service classes * Add getRegion into AWSResource (#43) * Add getRegion into AWSResource * Update api paths * Increase max retry limit for stream get to 1000 Runaway looping Stopping at 100 is probably to small. * Use underscores instead of camel case for json names It is a general project standard to use underscores. * Add paging functionality to getLogGroupNames (#53) * Add paging functionality to getLogGroupNames * Add unit test for CloudWatch log groups * Add comments in log group name unit test * Code clean up and remove unneeded code * Revert unintended change from unit test commit * Add retrievelogs for Kinesis Healthcheck (#76) * Refactor getKinesisStreams in KinesisService * Add validCredentials method in AWSService class * Temporary Main class added to test putting records into a Kinesis stream * Add retrieveKinesisLogs in KinesisService * Update temporary Main class * Update retrieveKinesisLogs to loop through shard iterators * Update Main class * Update pom file * Update validateCredentials in AWSService class * Add createKinesisClient method in KinesisService class * Add testGetStreamCredentials and update testGetStreams * Healthcheck merge (#81) * empty commit to push branch * Add Kinesis Healthcheck (#45) * Improve organization for Flow Log message detection * Improve Flow Log test It now tests for a message with too many and too few spaces. * Add TODOs for healthCheck method logic * Add beginnings of Kinesis healthChecker This will pull a establish a Kinesis subscription and pull a single message from a Kinesis stream. * Fix failing unit test * Continue developing KinesisHealthCheck - Remove unneeded metric tracking - Remove extra parsing logic (this object should just hand back payload and not do any parsing) - Improve application name handling - Add comments * Add detection logic for raw vs. CloudWatch logs * Remove KinesisHealthCheck class The KinesisConsumer does not work well for the health check (designed for realtime processing, takes a long time to start, cannot detect empty stream, and is really hard to use in a quick API request). Now, we're planning to directly retrieve the messages using the Kinesis client. This is the most straight-forward thing We might revisit this later. * Fix JSON parsing of Kinesis CloudWatch subscription record Parse the record just as was done in the existing AWS plugin. The logic now includes autodetection of compressed/vs not compressed. Mock Kinesis CloudWatch subscription record included for testing purposes. * Add CloudWatch logs codec and tests from existing AWS plugin * Parse Flow Log message into object * Load appropriate codec during healthCheck process When the message type is detected, load the respective codec for that message type. * Parse message with appropriate codec Once the log message type is detected, then the codec is looked up. The message is then parsed with the codec. * Supply log group name with the response * Improve comments, logging, and error checking The log group name is now also included in the response. * Add Flow Log codec test * Use AutoValue for CloudWatchLogEntry class * Use AutoValue for all remaining CloudWatch value classes * Cleanup merge conflicts after rebasing * Specify constants for all JsonProperty annotations * Delete uneeded KinesisDTO All data will be stored in the input * Establish a base AWSRequest JSON class * Fix Guice injection error for KinesisService * Add sample cURL command for healthCheck method A similar cURL command will be used for other methods, so that it is clear how the UI will use them. * Remove unneeded Kinesis Client 1.x dependency * Add formatted message summary in the Health Check response * Cleanup formatting and TODOs * Minor cleanup after rebasing and merging * Fix failing unit tests * Integrate AWS Health Check with Kinesis record retrieval (#86) * Fix incorrect pass of AWS key instead of secret Also improve comments for fake message retrieval with TODOs. * Update and connect retrieveRecords * Add handleCompressedMessages method * Delete temporary main class * Update retrieveRecords to only return sample size * Update KinesisService for healthCheck to function properly * Add unit test for selecting random record * Add unit test for retrieveRecords * Add Available Services API call (#82) * Resolves #50: Add Available Services API call * Add a test * Add missing spaces, change Amazon > AWS * Backend code cleanup (#90) * Pass region and AWS credentials with AWS API requests (#92) * Require POST object containing region and credentials for all requests Specifically adds a POST body requirement for the getKinesisSteams and getLogGroupNames methods. * Use snake_case for paths * Update region api call (#110) * Migrate Regions request from a list to a full response object with total * Update Region API call to include label and value * Reformat code * Aws permissions check (#113) * Add AWSPermissions class and update AWSResource with permission checks * Rename and remove permissions in AWSPermissions * Register AWSPermissions in IntegrationsModule * Remove space between methods * Save AWS input API endpoint (#93) * First-pass structure for saving AWS input * Add more structure for general AWS input - Add a type enum to differentiate the various types of log messages that are possible. - Add metacodec that will eventually differentiate between the types of log messages. - Fill configuration values when saving the input. * Resolve merge conflicts after rebasing over latest aws branch * Consolidate log type detection and input type identification There's no longer a need to use two enums for this. Also added healthCheck tests covering all message types: flow log, raw Cloud Watch and raw Kinesis. * Clean up saveInput request parameters and handling * Fix invalid type specification that was preventing input save * Add unit test for saving input * Fix incorrectly specified arguments * Minor cleanup * Cleanup for PR review * Remove uneeded log statements * Make save AWS input path and description more specific * Indicate that the save request is specifically for Kinesis In the future, each type of AWS input will likely require it's own request object and endpoint due to the fact that unique fields will probably be required for each. * AWSMessageType cleanup * Remove uneeded isFlowLog, isRaw methods. * Remove invalid AWSMessageType.Source.CLOUD_WATCH enum value. Messages are always read from Kinesis, and therefore the source is always Kinesis. Source is meant to differentiate messages from Kinesis and S3 for example. * Improve comments for AWSMessageType.Source enum class and method. * Remove typo * Generify the create AWS integration endpoint The naming, description, and comment now reflect that a generic AWS input is being created. * Return InputSummary response entity for AWS input creation request Also remove unneeded AWSResourceTest * More cleanup of healthCheck after input creation changes - Remove unneeded log_group field for health_check request. Resolves #108 - Add Kinesis stream name as a field in both raw and CloudWatch messages This change is lumped in with the other changes related to saving the input, since lots of healthCheck changes were already made there. This fixes some problems, so might as well have these improvements included with the review. * Clarify that AWSMetaCodec is a general AWS codec This class no longer erroneously extends AbstractKinesisCodec, which was only intended for Kinesis-specific codecs. * Use DateTime instead of long in KinesisLogEntry It turns out that Kinesis Record objects do have an arrival ime Instant timestamp. This is now being used instead of just using the date/time when the message was read by Graylog. * Add missing Guice annotation that were preventing server startup * Return JSON message field list in response from Health Check (#114) * Return JSON fields in healthCheck response There is still an issue with serializing the DateTime timestamp (serializing the deep object tree). This will be addressed separately. * Adapt unit test timestamp check to object value Health Check now returns a map of actual field values, so the test needs to check the JSON value. * Fix incorrect codec specified for Flow Logs, add tests * Fix incorrectly named field * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn * Advanced Kinesis Options * Cleanup * currently busted toggle * redo state handling * Remove artifact * removing commented code * removing unnecessary functions * pr feedback * Rebase conflicts * Fix several issues from PR review * "Final Review" layout (#83) * General Content * Review view layout complete * Cleanup * Update utils file name and export * Change LogFlow > Flow Log * default_values.js files * Route constants (#111) * Route constants * Fixing conflicts * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn Route constants Fixing conflicts * Correct import path * Changing route object * Adjust Kinesis setup wording * Remove the unneeded * Fixing up rebase artifacts from squash of irrelevant commits * AWS lockfile * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * Setting up necessary hooks * Removing input validation steps to reduce PR * Rebasing * Rename StyledForm * DEFAULT_VALUES * cleanup hook returns * Moving some reducers to state and breaking apart the rest * Remove functions from Input values * Steps function cleanup * Remove commented import * FormDataProvider and Context * Steps Context & provider * log and advancedoptions moved to context and provider * FormWrap functional component * renaming things * lint error * PR feedback * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * error and dirty states * Error on submit * Update validation to new state hooks * remove unneeded defaults * Output error messages * text key instead of password * Remove field validation * Update comment blocks (#137) * Update comment blocks * Remove minor typos * Update version to 3.1.0-beta.2-SNAPSHOT * Implement AWS codec selector with tests (#116) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Remove erroneous comment * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump package.json version to 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * Authorize and Streams API * All API calls except Save * reverting to hooks for api calls * Regions hook * Streams and Logs * streams useFetch * log sample * generic loading states * move regions * adding some documentation * Adding all the files * removing uneeded * cleanup * ApiRoutes * import proper path * Changing up useFetch * need the values... * lint * Removed unneeded inline cURL API documentation (#143) No need for this docs duplication. The System > Nodes > API Browser already documents them. * Use standard field names for AWS key and secret (#144) * Use standard naming for AWS secret and access key ids for Save Input Use these standard names for AWS key and secret from the Save Input AWS endpoint (From the AWSRequest interface). All other AWS API requests should be using these already. `aws_access_key_id` `aws_secret_access_key` * Use central constant for Region also * Fix incorrectly set Region property name * Fix access and secret key method names to match the AWSRequest interface * rename useFetch file * better var naming in kinesisstreams: * Update useFetch props * AWS transport selector with tests (#117) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Remove duplicate AWSTestingUtils class * Add missing permissions for sample AWS policy (#122) * Migrate AWS policy to autovalue object This avoids embedding JSON in a Java constant, which is error prone. * Add missing permissions Added: - kinesis:ListStreams - logs:DescribeLogGroups * Define JSON policy field order that matches AWS examples * Return the policy JSON string to the UI as string This is consistent with how this was done before. The UI can then format and display this to the user on the Available Services page. This is the policy that the user will create the AWS user for the AWS integration with. * Specify the correct AWS policy version * Change AWS Policy to a list * Add missing permissions Based on this [sample KCL code](aws/aws-sdk-java-v2#1214 (comment)) * Update version to 3.1.0-beta.2-SNAPSHOT * Remove uneeded import * Add Available Services API response error documentation * Throw InternalServerErrorException instead of JSON exception This is a bit cleaner from the API consumption side. A nice short error message is now returned instead of an obscure JSON error. * API Save Input (#145) * Save after Review and redirect to Inputs * Save functional * Adding match function to pluginmanifest * custom component * Update version to 3.1.0-beta.3-SNAPSHOT * Update version to 3.1.0-beta.3-SNAPSHOT * Aws kinesis newstream (#129) * Add create new Kinesis stream method * Add API call for createNewKinesisStream and KinesisNewStream classes * Update createNewKinesisStream * Update version to 3.1.0-beta.2-SNAPSHOT * Add error logging to createNewKinesisStream method * Add unit test for createNewKinesisStream method * Update error logging for createNewKinesisStream * Update testCreateNewKinesisStream * Remove shard count variable and set default value * Update testCreateNewKinesisStream * Code clean up * Update api comment blocks * Code clean up * Update error handling in createNewKinesisStream * Update unit test testCreateNewKinesisStream * Unload Confirmation Dialog (#148) * Playing with onunload * use ConfirmLeaveDialog common component * Question * Update version to 3.1.0-beta.3-SNAPSHOT * lint * cleanup * Don't alert on last step. Save call failed if desc wasn't available * Remove emotion and use styled-components (#157) * Remove emotion and use styled-components * import styled-components * Move title and description to FormWrap for consistent layout * Handle validation per-step * fix aws key h4x * styled validatedinput * enable health check and review forms * Starting Error Handling * Output API Errors * remove some more debugging * Validate API Key & Secret and output styled errors * Error message if stream doesn't contain any logs * Return a 4xx response for an unsuccessful health check * Save API Error Handling * Remove `success` flag from HealthCheckResponse Failure is now tracked by throwing a `BadRequestException` resulting in an `ApiError` json response * Throw `BadRequestException` if no Kinesis streams are found * Throw `BadRequestException` if no Kinesis CloudWatch groups are found * Error message styling * wrap up api errors and adding styles * rebasing and cleanup * trim input values * remove defaults * Create input with togglable masking * Include stream name in error * Port of existing KinesisTransport (#140) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Remove unused imports and fix formatting * Update version to 3.1.0-beta.3-SNAPSHOT * Merge branch 'aws' into aws-transport-kinesis and resolve conflicts # Conflicts: # src/main/java/org/graylog/integrations/aws/service/KinesisService.java * Upgrade to latest Kinesis Client version (#151) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Update Kinesis Client version * Add Kinesis client library v2 Consumer sample * Migrate KinesisConsumer to Kinesis Client Library v2 * Remove unused imports and fix formatting * Finish migration of Kinesis client to new version * Complete upgrade to Kinesis Client Library v2 * Bump KCL version * Migrate shard processor to its own class * Bump version to 3.1.0-beta.3-SNAPSHOT * Add batch size limit * Code cleanup * Remove unneeded throttle time limit reached shutdown * Remove kinesis_max_throttled_wait_ms save input field This field is no longer needed, since the new Kinesis Consumer appears to correctly handle longer throttling and pausing in `processRecords` without making the consumer unhealthy. * Temporarily remove Assume Role Arn auth This will be added back later. See #29 * Adjust logging levels * Remove sample class * Ignore unmapped properties due to removed max_throttled_wait field See Graylog2/graylog-plugin-integrations#156 * Update Throttling advanced option (#169) * Update Throttling advanced option * messages/ms * Remove throttle number * Update label * Shells/mocks for Kinesis Auto setup API endpoints (#167) * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Update version to 3.1.0-beta.4-SNAPSHOT * Code cleanup - Remove unneeded imports - Replace hyphens with underscores * Add `aws` to root URL path for consistency All other AWS routes are prefixed with this, so do the same here. * Change subscription path Change `add_subscription` route to `create_subscription` * Fix linter warning was introduced during merge conflict resolution. * Add RolePermissions for Kinesis stream (#163) * Add checkKinesisStreamStatus and setRolePermissions method * Add getNewRolePermissions and autoKinesisPermissionRequired method * Rename autoKinesisPermissionsRequired method and add error handling * Code clean up * Refactor autoSetup, split up creating a role and attaching a policy * Add error handling for creating a new role * Update autoKinesisPermissionsRequired logic and add comment block * Update version to 3.1.0-beta.4-SNAPSHOT * Delete checkKinesisStreamStatus method * Update error logging messages * Add temporary main class for testing, update error handling logic * Delete temporary main, update autoKinesisPermissionsRequired * Change exceptions thrown and code clean up * Update version to 3.1.0-rc.2-SNAPSHOT * Add addSubscriptionFilter to CloudwatchService (#160) * Add addSubscriptionFilter method to KinesisService * Move addSubscriptionFilter method to CloudWatchService * Code clean up, change exception errors * Bump version to 3.1.0-rc.3-SNAPSHOT * Kinesis Auto-setup (#186) * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump js-yaml from 3.13.0 to 3.13.1 Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 3.13.0 to 3.13.1. - [Release notes](https://github.com/nodeca/js-yaml/releases) - [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md) - [Commits](nodeca/js-yaml@3.13.0...3.13.1) Signed-off-by: dependabot[bot] <support@github.com> * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * [Security] Bump lodash from 4.17.11 to 4.17.14 (#131) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.11 to 4.17.14. **This update includes security fixes.** - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.11...4.17.14) Signed-off-by: dependabot-preview[bot] <support@dependabot.com> * Bump package.json version to 3.1.0-beta.2 * [graylog-plugin-integrations] prepare release 3.1.0-beta.2 * Bump package.json version to 3.1.0-beta.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Bump package.json version to 3.1.0-beta.3 * [graylog-plugin-integrations] prepare release 3.1.0-beta.3 * Bump package.json version to 3.1.0-beta.4-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Render Setup screen * cleanup * Allowing toggle to get from Streams to Setup * Toggle back to Streams & cleanup Group on Region change * Add Kinesis auto-setup routes * Add initial `create_stream` request * Define fetchs for auto Kinesis setup * Finish create_stream request * Add all auto-setup API requests * Specify actual region and request properties * Improve presentation for transition to automated Kinesis setup * Improve on-screen documentation for the automatic Kinesis setup * Bump package.json version to 3.1.0-rc.1 * [graylog-plugin-integrations] prepare release 3.1.0-rc.1 * Bump package.json version to 3.1.0-rc.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Update version to 3.1.0-rc.2-SNAPSHOT * Add KinesisSetupStep shell * Add KinesisSetupSteps container * Update createKinesisStream to return streamArn * Update createKinesisStream and testCreateNewKinesisStream * Add edge case error handling to createNewKinesisStream * Add preliminary code for KinesisFullSetup method * Use await for synchronous auto setup steps * Migrate to map for auto-setup state management * Render auto-setup progress to UI * Include success field and initial delay for testing * Delete KinesisFullSetup method * Migrate Kinesis setup to map to allow setup step iteration * Refactor autoKinesisPermissions parameters and return response * Rework setup logic to use individual state fields Individual state fields are needed for each component to update independently. * Add progress icons * Improve icon and step styles * Add auto-setup error handling * Refactor addSubscriptionFilter to return CreateLogSubscriptionResponse * Update api calls in KinesisFullSetup * Update createNewKinesisStream and createPolicies api call * Add flow control for setup steps * Clean up api methods * Add mocking for UI testing Also remove full setup, since the individual steps will be used. * Sync field mappings between UI and backend * Add auto-setup integration unit test * Add auto-setup test assertions * Rename `explanation` -> `result` * Display result to the user for each auto-setup step * Use unique role ane role policy names Since these are only needed in the auto-setup flow, we can automatically generate and assign them (as long as we provide them to the user in the response). Customizing these names is possible, and can be considered in the future. * Adjustments to get auto-setup working * Add support subscription retries to resolve IAM eventual consistency IAM sometimes takes 10+ seconds to initialize, so retrying allows the subscription attempt to be attempted multiple times, so that once IAM is available, then the subscription will be created successfully. * Cleanup retry logic * Add new page for auto-setup steps * Adjust wording * Add auto-setup TODO for adding warning * TODOs and cleanup * Formatting cleanup * Support ability to continue adding input after auto-setup * Bump package.json version to 3.1.0-rc.2 * [graylog-plugin-integrations] prepare release 3.1.0-rc.2 * Bump package.json version to 3.1.0-rc.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Disable Continue Setup button while auto-setup is in progress * Include warning that messages might take time to arrive in new stream * Require explicit agreement from user before starting auto-setup * Bump version to 3.1.0-rc.3-SNAPSHOT * Adjustments to auto-setup buttons, labels, and formatting * Skip CloudWatch control messages in Health Check Fixes Graylog2/graylog-plugin-integrations#178 * Add comment * Add support for the "flow_log_" prefix Closes Graylog2/graylog-plugin-integrations#120 * Fixing lint errors * Return mock responses * DEV COMMIT * Remove unnecessary Row and Col now that we're rendering a sidebar * sidebar styles * Add checkbox for adding Flow Log field prefix * Auto-Setup Sidebar * lint * Adjust wording * Correct label for AWS Message Type * Consolidate Kinesis payload decompression logic * Agree TOS in Modal * Return actual stream name in create stream mock * Modal Styled * Don't include the policy name in response, since it's inline * Remove DEV COMMIT changes * Update header description * Fix failing unit test * Mock responses * Add default setting for prefix field * Kinesis auto-setup Health Check redirect * health check clear sidebar * Add support for assuming an ARN role (#188) * Add support for assuming an ARN role Fixes Graylog2/graylog-plugin-integrations#29 * Add Assume Role ARN config field * Remove unneeded imports * Fix formatting * Countdown and Retry button * CheckAgain * Record when users agree to auto-setup (#194) * Add log message recording that a user agreed to auto-setup * Remove unneeded import * Fixing select default value (#199) * Adding pattern to Stream Name input (#200) * Adding pattern to Stream Name input * Don't submit if field has error * Add API request to get permissions (#196) * Add request to retireve Kinesis permissions Includes permissions for both the full setup and the auto-setup. Closes #174 * Alphabetize permissions, add comments, remove unused imports * Add unit test * Make auto-setup unit test pass when mocking enabled * Remove unneeded create log group/stream permissions * Minor code clean up (#201) * Minor code clean up * Fix typo * Make auto-setup unit test pass when mocking enabled * Bump version to 3.2.0-SNAPSHOT * Bump version to 3.2.0-SNAPSHOT * Starting testing * Linting Fix (#212) * Revert "frontend merge conflicts" This reverts commit fa5c2775a377e0585b1d771cec552f8ae0fd6de6, reversing changes made to 207fca0533051521e5cad58b39b43f15e5e44179. * remove test * Use Moment * cleanup * Helps if countdown counts down * 120 seconds

* Initial commit * Add AWS SDK v2 dependency * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Add aws package and service class, also fix broken commons-codec import Explicitly include commons-codec in this POM, since the AWS SDK v2 internally remaps commons-codec to another internal package. This makes the commons-codec from graylog-project-parent unavailable. See https://aws.amazon.com/blogs/developer/java-sdk-bundle * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Add Kinesis SDK * General UI Skeleton (#75) * Simple Clickthrough without API * Clickable Skeleton All Steps * Feedback * Lint * AWS Cloudwatch backend API calls Add initial backend API calls for CloudWatch integration: getRegions, getLogGroups, getStreams, retrieveKinesisMessages, healthCheck * AWS Cloud Watch services and resources (#24) Adds beginnings of API endpoint and Kinesis/Cloudwatch services Includes a structure that we will continue to build from. * Rework organization of classes for unified structure The goal is to establish some structure that we can implement AWS API calls within. There's now one resource (for API calls), one service (for business logic), and one AWSClient (for all AWS/API SDK interactions). * Cleanup and add comments * Cleanup code around log message auto-detection * Aws cloudwatch client (#31) * Add CloudWatchService class * Add AWSConfigSettings class * Add UserCredentials class * Add temporary Main class * Include the latest version of the Apache Http Client * Improve HTTP Client dependency comment * Update AWSConfigSettings class * Update CloudWatchService class * Update temporary Main class * Fix inject error in AWSResource * Fix merge conflicts * Restructure Resources and Service classes * Add getRegion into AWSResource (#43) * Add getRegion into AWSResource * Update api paths * Increase max retry limit for stream get to 1000 Runaway looping Stopping at 100 is probably to small. * Use underscores instead of camel case for json names It is a general project standard to use underscores. * Add paging functionality to getLogGroupNames (#53) * Add paging functionality to getLogGroupNames * Add unit test for CloudWatch log groups * Add comments in log group name unit test * Code clean up and remove unneeded code * Revert unintended change from unit test commit * Add retrievelogs for Kinesis Healthcheck (#76) * Refactor getKinesisStreams in KinesisService * Add validCredentials method in AWSService class * Temporary Main class added to test putting records into a Kinesis stream * Add retrieveKinesisLogs in KinesisService * Update temporary Main class * Update retrieveKinesisLogs to loop through shard iterators * Update Main class * Update pom file * Update validateCredentials in AWSService class * Add createKinesisClient method in KinesisService class * Add testGetStreamCredentials and update testGetStreams * Healthcheck merge (#81) * empty commit to push branch * Add Kinesis Healthcheck (#45) * Improve organization for Flow Log message detection * Improve Flow Log test It now tests for a message with too many and too few spaces. * Add TODOs for healthCheck method logic * Add beginnings of Kinesis healthChecker This will pull a establish a Kinesis subscription and pull a single message from a Kinesis stream. * Fix failing unit test * Continue developing KinesisHealthCheck - Remove unneeded metric tracking - Remove extra parsing logic (this object should just hand back payload and not do any parsing) - Improve application name handling - Add comments * Add detection logic for raw vs. CloudWatch logs * Remove KinesisHealthCheck class The KinesisConsumer does not work well for the health check (designed for realtime processing, takes a long time to start, cannot detect empty stream, and is really hard to use in a quick API request). Now, we're planning to directly retrieve the messages using the Kinesis client. This is the most straight-forward thing We might revisit this later. * Fix JSON parsing of Kinesis CloudWatch subscription record Parse the record just as was done in the existing AWS plugin. The logic now includes autodetection of compressed/vs not compressed. Mock Kinesis CloudWatch subscription record included for testing purposes. * Add CloudWatch logs codec and tests from existing AWS plugin * Parse Flow Log message into object * Load appropriate codec during healthCheck process When the message type is detected, load the respective codec for that message type. * Parse message with appropriate codec Once the log message type is detected, then the codec is looked up. The message is then parsed with the codec. * Supply log group name with the response * Improve comments, logging, and error checking The log group name is now also included in the response. * Add Flow Log codec test * Use AutoValue for CloudWatchLogEntry class * Use AutoValue for all remaining CloudWatch value classes * Cleanup merge conflicts after rebasing * Specify constants for all JsonProperty annotations * Delete uneeded KinesisDTO All data will be stored in the input * Establish a base AWSRequest JSON class * Fix Guice injection error for KinesisService * Add sample cURL command for healthCheck method A similar cURL command will be used for other methods, so that it is clear how the UI will use them. * Remove unneeded Kinesis Client 1.x dependency * Add formatted message summary in the Health Check response * Cleanup formatting and TODOs * Minor cleanup after rebasing and merging * Fix failing unit tests * Integrate AWS Health Check with Kinesis record retrieval (#86) * Fix incorrect pass of AWS key instead of secret Also improve comments for fake message retrieval with TODOs. * Update and connect retrieveRecords * Add handleCompressedMessages method * Delete temporary main class * Update retrieveRecords to only return sample size * Update KinesisService for healthCheck to function properly * Add unit test for selecting random record * Add unit test for retrieveRecords * Add Available Services API call (#82) * Resolves #50: Add Available Services API call * Add a test * Add missing spaces, change Amazon > AWS * Backend code cleanup (#90) * Pass region and AWS credentials with AWS API requests (#92) * Require POST object containing region and credentials for all requests Specifically adds a POST body requirement for the getKinesisSteams and getLogGroupNames methods. * Use snake_case for paths * Update region api call (#110) * Migrate Regions request from a list to a full response object with total * Update Region API call to include label and value * Reformat code * Aws permissions check (#113) * Add AWSPermissions class and update AWSResource with permission checks * Rename and remove permissions in AWSPermissions * Register AWSPermissions in IntegrationsModule * Remove space between methods * Save AWS input API endpoint (#93) * First-pass structure for saving AWS input * Add more structure for general AWS input - Add a type enum to differentiate the various types of log messages that are possible. - Add metacodec that will eventually differentiate between the types of log messages. - Fill configuration values when saving the input. * Resolve merge conflicts after rebasing over latest aws branch * Consolidate log type detection and input type identification There's no longer a need to use two enums for this. Also added healthCheck tests covering all message types: flow log, raw Cloud Watch and raw Kinesis. * Clean up saveInput request parameters and handling * Fix invalid type specification that was preventing input save * Add unit test for saving input * Fix incorrectly specified arguments * Minor cleanup * Cleanup for PR review * Remove uneeded log statements * Make save AWS input path and description more specific * Indicate that the save request is specifically for Kinesis In the future, each type of AWS input will likely require it's own request object and endpoint due to the fact that unique fields will probably be required for each. * AWSMessageType cleanup * Remove uneeded isFlowLog, isRaw methods. * Remove invalid AWSMessageType.Source.CLOUD_WATCH enum value. Messages are always read from Kinesis, and therefore the source is always Kinesis. Source is meant to differentiate messages from Kinesis and S3 for example. * Improve comments for AWSMessageType.Source enum class and method. * Remove typo * Generify the create AWS integration endpoint The naming, description, and comment now reflect that a generic AWS input is being created. * Return InputSummary response entity for AWS input creation request Also remove unneeded AWSResourceTest * More cleanup of healthCheck after input creation changes - Remove unneeded log_group field for health_check request. Resolves #108 - Add Kinesis stream name as a field in both raw and CloudWatch messages This change is lumped in with the other changes related to saving the input, since lots of healthCheck changes were already made there. This fixes some problems, so might as well have these improvements included with the review. * Clarify that AWSMetaCodec is a general AWS codec This class no longer erroneously extends AbstractKinesisCodec, which was only intended for Kinesis-specific codecs. * Use DateTime instead of long in KinesisLogEntry It turns out that Kinesis Record objects do have an arrival ime Instant timestamp. This is now being used instead of just using the date/time when the message was read by Graylog. * Add missing Guice annotation that were preventing server startup * Return JSON message field list in response from Health Check (#114) * Return JSON fields in healthCheck response There is still an issue with serializing the DateTime timestamp (serializing the deep object tree). This will be addressed separately. * Adapt unit test timestamp check to object value Health Check now returns a map of actual field values, so the test needs to check the JSON value. * Fix incorrect codec specified for Flow Logs, add tests * Fix incorrectly named field * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn * Advanced Kinesis Options * Cleanup * currently busted toggle * redo state handling * Remove artifact * removing commented code * removing unnecessary functions * pr feedback * Rebase conflicts * Fix several issues from PR review * "Final Review" layout (#83) * General Content * Review view layout complete * Cleanup * Update utils file name and export * Change LogFlow > Flow Log * default_values.js files * Route constants (#111) * Route constants * Fixing conflicts * Simple Clickthrough without API Convert to Input Clickable Skeleton All Steps Cleanup Yarn Route constants Fixing conflicts * Correct import path * Changing route object * Adjust Kinesis setup wording * Remove the unneeded * Fixing up rebase artifacts from squash of irrelevant commits * AWS lockfile * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * Setting up necessary hooks * Removing input validation steps to reduce PR * Rebasing * Rename StyledForm * DEFAULT_VALUES * cleanup hook returns * Moving some reducers to state and breaking apart the rest * Remove functions from Input values * Steps function cleanup * Remove commented import * FormDataProvider and Context * Steps Context & provider * log and advancedoptions moved to context and provider * FormWrap functional component * renaming things * lint error * PR feedback * Add validation to fields and form * beginning to restructure form validation * Convert state to hooks * error and dirty states * Error on submit * Update validation to new state hooks * remove unneeded defaults * Output error messages * text key instead of password * Remove field validation * Update comment blocks (#137) * Update comment blocks * Remove minor typos * Update version to 3.1.0-beta.2-SNAPSHOT * Implement AWS codec selector with tests (#116) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Remove erroneous comment * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump package.json version to 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * Authorize and Streams API * All API calls except Save * reverting to hooks for api calls * Regions hook * Streams and Logs * streams useFetch * log sample * generic loading states * move regions * adding some documentation * Adding all the files * removing uneeded * cleanup * ApiRoutes * import proper path * Changing up useFetch * need the values... * lint * Removed unneeded inline cURL API documentation (#143) No need for this docs duplication. The System > Nodes > API Browser already documents them. * Use standard field names for AWS key and secret (#144) * Use standard naming for AWS secret and access key ids for Save Input Use these standard names for AWS key and secret from the Save Input AWS endpoint (From the AWSRequest interface). All other AWS API requests should be using these already. `aws_access_key_id` `aws_secret_access_key` * Use central constant for Region also * Fix incorrectly set Region property name * Fix access and secret key method names to match the AWSRequest interface * rename useFetch file * better var naming in kinesisstreams: * Update useFetch props * AWS transport selector with tests (#117) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Remove duplicate AWSTestingUtils class * Add missing permissions for sample AWS policy (#122) * Migrate AWS policy to autovalue object This avoids embedding JSON in a Java constant, which is error prone. * Add missing permissions Added: - kinesis:ListStreams - logs:DescribeLogGroups * Define JSON policy field order that matches AWS examples * Return the policy JSON string to the UI as string This is consistent with how this was done before. The UI can then format and display this to the user on the Available Services page. This is the policy that the user will create the AWS user for the AWS integration with. * Specify the correct AWS policy version * Change AWS Policy to a list * Add missing permissions Based on this [sample KCL code](aws/aws-sdk-java-v2#1214 (comment)) * Update version to 3.1.0-beta.2-SNAPSHOT * Remove uneeded import * Add Available Services API response error documentation * Throw InternalServerErrorException instead of JSON exception This is a bit cleaner from the API consumption side. A nice short error message is now returned instead of an obscure JSON error. * API Save Input (#145) * Save after Review and redirect to Inputs * Save functional * Adding match function to pluginmanifest * custom component * Update version to 3.1.0-beta.3-SNAPSHOT * Update version to 3.1.0-beta.3-SNAPSHOT * Aws kinesis newstream (#129) * Add create new Kinesis stream method * Add API call for createNewKinesisStream and KinesisNewStream classes * Update createNewKinesisStream * Update version to 3.1.0-beta.2-SNAPSHOT * Add error logging to createNewKinesisStream method * Add unit test for createNewKinesisStream method * Update error logging for createNewKinesisStream * Update testCreateNewKinesisStream * Remove shard count variable and set default value * Update testCreateNewKinesisStream * Code clean up * Update api comment blocks * Code clean up * Update error handling in createNewKinesisStream * Update unit test testCreateNewKinesisStream * Unload Confirmation Dialog (#148) * Playing with onunload * use ConfirmLeaveDialog common component * Question * Update version to 3.1.0-beta.3-SNAPSHOT * lint * cleanup * Don't alert on last step. Save call failed if desc wasn't available * Remove emotion and use styled-components (#157) * Remove emotion and use styled-components * import styled-components * Move title and description to FormWrap for consistent layout * Handle validation per-step * fix aws key h4x * styled validatedinput * enable health check and review forms * Starting Error Handling * Output API Errors * remove some more debugging * Validate API Key & Secret and output styled errors * Error message if stream doesn't contain any logs * Return a 4xx response for an unsuccessful health check * Save API Error Handling * Remove `success` flag from HealthCheckResponse Failure is now tracked by throwing a `BadRequestException` resulting in an `ApiError` json response * Throw `BadRequestException` if no Kinesis streams are found * Throw `BadRequestException` if no Kinesis CloudWatch groups are found * Error message styling * wrap up api errors and adding styles * rebasing and cleanup * trim input values * remove defaults * Create input with togglable masking * Include stream name in error * Port of existing KinesisTransport (#140) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Remove unused imports and fix formatting * Update version to 3.1.0-beta.3-SNAPSHOT * Merge branch 'aws' into aws-transport-kinesis and resolve conflicts # Conflicts: # src/main/java/org/graylog/integrations/aws/service/KinesisService.java * Upgrade to latest Kinesis Client version (#151) * Implement AWS metacodec handler with tests Closes #70 #27 * Rename AWSMetaCodec to AWSCodec * Add AWS transport selection handler * Remove erroneous comment * Add transport selection test for AWSTransport * Fix log message that referred to the codec instead of transport * More log cleanup of log entries * Fix log entries - Change info > debug - Use consistent starting/stopping wording. - These log entries may be removed later. They are helpful to verify that the AWSTransport is selecting the correct transport based on the AWSMessageType enum. * Finalize variables * Finalize variables * Update version to 3.1.0-beta.2-SNAPSHOT * Update version to 3.1.0-beta.2-SNAPSHOT * Rename codec constant AWSMetaCodec -> AWSCodec Co-Authored-By: Bernd Ahlers <bernd@users.noreply.github.com> * Move integrations tests to actual folder path Moved from test/java/org.graylog.integrations to test/java/org/graylog/integrations * First cut of migrating the existing Kinesis client Migrate Kinesis client v1.10 from the existing AWS integration to the new. * Short-circuit usage of multi-AWSAuthProvider Now, the AWS credentials are directly provided to the KinesisTransport. This will likely be improved in the future. See Graylog2/graylog-plugin-integrations#139 * Add missing name in codec * Add processor for Kinesis transport This is responsible for handling the kinesis payload (decompress if from CloudWatch, or convert bytes to string if not) and converting it into an a list of raw messages. * Use KinesisTransportProcessor in the KinesisTransport * Add code comments * Adjustments to get KinesisTransport running The main change is to migrate the AWS_MESSAGE_TYPE config prop to the codec, since the codec can only access config properties it owns (due to per-message instantiation and configs being encoded with each raw message). The config prop is still accessible from the transport. * Improve comments * Code and comments cleanup * Add KinesisPayloadDecoder tests * Add message timestamp coverage to KinesisPayloadDecoder tests * Update Kinesis Client version * Add Kinesis client library v2 Consumer sample * Migrate KinesisConsumer to Kinesis Client Library v2 * Remove unused imports and fix formatting * Finish migration of Kinesis client to new version * Complete upgrade to Kinesis Client Library v2 * Bump KCL version * Migrate shard processor to its own class * Bump version to 3.1.0-beta.3-SNAPSHOT * Add batch size limit * Code cleanup * Remove unneeded throttle time limit reached shutdown * Remove kinesis_max_throttled_wait_ms save input field This field is no longer needed, since the new Kinesis Consumer appears to correctly handle longer throttling and pausing in `processRecords` without making the consumer unhealthy. * Temporarily remove Assume Role Arn auth This will be added back later. See #29 * Adjust logging levels * Remove sample class * Ignore unmapped properties due to removed max_throttled_wait field See Graylog2/graylog-plugin-integrations#156 * Update Throttling advanced option (#169) * Update Throttling advanced option * messages/ms * Remove throttle number * Update label * Shells/mocks for Kinesis Auto setup API endpoints (#167) * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Update version to 3.1.0-beta.4-SNAPSHOT * Code cleanup - Remove unneeded imports - Replace hyphens with underscores * Add `aws` to root URL path for consistency All other AWS routes are prefixed with this, so do the same here. * Change subscription path Change `add_subscription` route to `create_subscription` * Fix linter warning was introduced during merge conflict resolution. * Add RolePermissions for Kinesis stream (#163) * Add checkKinesisStreamStatus and setRolePermissions method * Add getNewRolePermissions and autoKinesisPermissionRequired method * Rename autoKinesisPermissionsRequired method and add error handling * Code clean up * Refactor autoSetup, split up creating a role and attaching a policy * Add error handling for creating a new role * Update autoKinesisPermissionsRequired logic and add comment block * Update version to 3.1.0-beta.4-SNAPSHOT * Delete checkKinesisStreamStatus method * Update error logging messages * Add temporary main class for testing, update error handling logic * Delete temporary main, update autoKinesisPermissionsRequired * Change exceptions thrown and code clean up * Update version to 3.1.0-rc.2-SNAPSHOT * Add addSubscriptionFilter to CloudwatchService (#160) * Add addSubscriptionFilter method to KinesisService * Move addSubscriptionFilter method to CloudWatchService * Code clean up, change exception errors * Bump version to 3.1.0-rc.3-SNAPSHOT * Kinesis Auto-setup (#186) * Updating yarn lockfile. (#119) * Updating yarn lockfile. (#127) Due to recent transitive dependency updates (#6097, Graylog2/graylog2-server##6072, #6069, #6065), an update of the yarn lockfile for core and all plugins is required. * Bump js-yaml from 3.13.0 to 3.13.1 Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 3.13.0 to 3.13.1. - [Release notes](https://github.com/nodeca/js-yaml/releases) - [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md) - [Commits](nodeca/js-yaml@3.13.0...3.13.1) Signed-off-by: dependabot[bot] <support@github.com> * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Revert version to 3.1.0-SNAPSHOT * Bump package.json version to 3.1.0-beta.1 * [graylog-plugin-integrations] prepare release 3.1.0-beta.1 * Bump package.json version to 3.1.0-beta.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * [Security] Bump lodash from 4.17.11 to 4.17.14 (#131) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.11 to 4.17.14. **This update includes security fixes.** - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.11...4.17.14) Signed-off-by: dependabot-preview[bot] <support@dependabot.com> * Bump package.json version to 3.1.0-beta.2 * [graylog-plugin-integrations] prepare release 3.1.0-beta.2 * Bump package.json version to 3.1.0-beta.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Add auto-setup API shell methods with mocked responses * Adjustments to auto-setup fields and wording * Bump package.json version to 3.1.0-beta.3 * [graylog-plugin-integrations] prepare release 3.1.0-beta.3 * Bump package.json version to 3.1.0-beta.4-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Render Setup screen * cleanup * Allowing toggle to get from Streams to Setup * Toggle back to Streams & cleanup Group on Region change * Add Kinesis auto-setup routes * Add initial `create_stream` request * Define fetchs for auto Kinesis setup * Finish create_stream request * Add all auto-setup API requests * Specify actual region and request properties * Improve presentation for transition to automated Kinesis setup * Improve on-screen documentation for the automatic Kinesis setup * Bump package.json version to 3.1.0-rc.1 * [graylog-plugin-integrations] prepare release 3.1.0-rc.1 * Bump package.json version to 3.1.0-rc.2-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Update version to 3.1.0-rc.2-SNAPSHOT * Add KinesisSetupStep shell * Add KinesisSetupSteps container * Update createKinesisStream to return streamArn * Update createKinesisStream and testCreateNewKinesisStream * Add edge case error handling to createNewKinesisStream * Add preliminary code for KinesisFullSetup method * Use await for synchronous auto setup steps * Migrate to map for auto-setup state management * Render auto-setup progress to UI * Include success field and initial delay for testing * Delete KinesisFullSetup method * Migrate Kinesis setup to map to allow setup step iteration * Refactor autoKinesisPermissions parameters and return response * Rework setup logic to use individual state fields Individual state fields are needed for each component to update independently. * Add progress icons * Improve icon and step styles * Add auto-setup error handling * Refactor addSubscriptionFilter to return CreateLogSubscriptionResponse * Update api calls in KinesisFullSetup * Update createNewKinesisStream and createPolicies api call * Add flow control for setup steps * Clean up api methods * Add mocking for UI testing Also remove full setup, since the individual steps will be used. * Sync field mappings between UI and backend * Add auto-setup integration unit test * Add auto-setup test assertions * Rename `explanation` -> `result` * Display result to the user for each auto-setup step * Use unique role ane role policy names Since these are only needed in the auto-setup flow, we can automatically generate and assign them (as long as we provide them to the user in the response). Customizing these names is possible, and can be considered in the future. * Adjustments to get auto-setup working * Add support subscription retries to resolve IAM eventual consistency IAM sometimes takes 10+ seconds to initialize, so retrying allows the subscription attempt to be attempted multiple times, so that once IAM is available, then the subscription will be created successfully. * Cleanup retry logic * Add new page for auto-setup steps * Adjust wording * Add auto-setup TODO for adding warning * TODOs and cleanup * Formatting cleanup * Support ability to continue adding input after auto-setup * Bump package.json version to 3.1.0-rc.2 * [graylog-plugin-integrations] prepare release 3.1.0-rc.2 * Bump package.json version to 3.1.0-rc.3-SNAPSHOT * [graylog-plugin-integrations] prepare for next development iteration * Disable Continue Setup button while auto-setup is in progress * Include warning that messages might take time to arrive in new stream * Require explicit agreement from user before starting auto-setup * Bump version to 3.1.0-rc.3-SNAPSHOT * Adjustments to auto-setup buttons, labels, and formatting * Skip CloudWatch control messages in Health Check Fixes Graylog2/graylog-plugin-integrations#178 * Add comment * Add support for the "flow_log_" prefix Closes Graylog2/graylog-plugin-integrations#120 * Fixing lint errors * Return mock responses * DEV COMMIT * Remove unnecessary Row and Col now that we're rendering a sidebar * sidebar styles * Add checkbox for adding Flow Log field prefix * Auto-Setup Sidebar * lint * Adjust wording * Correct label for AWS Message Type * Consolidate Kinesis payload decompression logic * Agree TOS in Modal * Return actual stream name in create stream mock * Modal Styled * Don't include the policy name in response, since it's inline * Remove DEV COMMIT changes * Update header description * Fix failing unit test * Mock responses * Add default setting for prefix field * Add support for assuming an ARN role (#188) * Add support for assuming an ARN role Fixes Graylog2/graylog-plugin-integrations#29 * Add Assume Role ARN config field * Remove unneeded imports * Fix formatting * Record when users agree to auto-setup (#194) * Add log message recording that a user agreed to auto-setup * Remove unneeded import * Fixing select default value (#199) * Adding pattern to Stream Name input (#200) * Adding pattern to Stream Name input * Don't submit if field has error * Add API request to get permissions (#196) * Add request to retireve Kinesis permissions Includes permissions for both the full setup and the auto-setup. Closes #174 * Alphabetize permissions, add comments, remove unused imports * Add unit test * Make auto-setup unit test pass when mocking enabled * Remove unneeded create log group/stream permissions * Minor code clean up (#201) * Minor code clean up * Fix typo * Proposed wording changes We'll pick this up again in Hamburg * Bump version to 3.2.0-SNAPSHOT * Wording adjustments * Change "Setup" -> "Set Up" * Fix lint errors * More wording changes * Removing empty doc link * Updating Review log type output

lennartkoopmann closed this as completed May 4, 2013

lennartkoopmann mentioned this issue May 4, 2013

Upgrade to RabbitMQ 3.1.0 #138

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade to latest RabbitMQ client library version #131

Upgrade to latest RabbitMQ client library version #131

lennartkoopmann commented Apr 14, 2013

joschi commented May 1, 2013

lennartkoopmann commented May 4, 2013

Upgrade to latest RabbitMQ client library version #131

Upgrade to latest RabbitMQ client library version #131

Comments

lennartkoopmann commented Apr 14, 2013

joschi commented May 1, 2013

lennartkoopmann commented May 4, 2013