Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Apache Flink runtime for Kinesis Analytics #11652

Merged
merged 129 commits into from
Oct 21, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
129 commits
Select commit Hold shift + click to select a range
b94eba4
Update usage of kinesisanalytics packge to kinesisanalyticsv2
blake-wilson Jan 10, 2020
0f6103f
Add required field for service execution role
blake-wilson Jan 11, 2020
f38a807
Begin adding support for Flink runtime with KDA
blake-wilson Jan 12, 2020
e021caf
Add skeleton for Flink configuration
blake-wilson Jan 12, 2020
9b6bb8f
Fix some differences with SQL and Flink runtime configs
blake-wilson Jan 12, 2020
372e430
Make more fixes for differing runtimes
blake-wilson Jan 13, 2020
6cf164b
Merge remote-tracking branch 'upstream/master' into add-flink-support
blake-wilson Jan 13, 2020
f9e7c75
Merge remote-tracking branch 'upstream/master' into add-flink-support
blake-wilson Jan 15, 2020
eb40579
Begin adding tests for flink application provisioning
blake-wilson Jan 15, 2020
7a11f3a
Start fleshing out create logic
blake-wilson Jan 16, 2020
15e5656
Add monitoring config
blake-wilson Jan 16, 2020
d6fdc5d
Add parallelism config
blake-wilson Jan 16, 2020
7f6549f
Test monitoring and parallelism
blake-wilson Jan 16, 2020
77b4fca
Continue work on testing kinesisanalyticsv2 provider
blake-wilson Jan 17, 2020
222c49a
Fix more tests
blake-wilson Jan 17, 2020
246e79d
Fix version being incremented when no changes occurred
blake-wilson Jan 17, 2020
98c1854
Fix test config issues
blake-wilson Jan 17, 2020
20249c7
Merge remote-tracking branch 'upstream/master' into add-flink-support
blake-wilson Jan 17, 2020
175675c
Fix update
blake-wilson Jan 17, 2020
19c84ce
Set cloudwatch logging option updates in update application options
blake-wilson Jan 17, 2020
cdcb549
Fix import for Flink application resources. Cover more with test
blake-wilson Jan 17, 2020
ef628fc
Fix bug updating Flink appplications. Add update test
blake-wilson Jan 17, 2020
c26cdbd
Add validation for configuration type and metrics level
blake-wilson Jan 18, 2020
b7ff649
Add validation for log level
blake-wilson Jan 18, 2020
b81d25e
Merge remote-tracking branch 'upstream/master' into add-flink-support
blake-wilson Jan 18, 2020
d1ac79b
Add optional S3 object version to schema
blake-wilson Jan 18, 2020
4623182
Add tests for new validators
blake-wilson Jan 18, 2020
c71d79e
Remove unnecessary type conversion
blake-wilson Jan 18, 2020
ae63460
Add test for kinesisanalytics configuration type validation
blake-wilson Jan 19, 2020
f7f862c
Add support for property groups
blake-wilson Jan 19, 2020
b20abea
Begin reworking schema to make more semantically similar to AWS API
blake-wilson Jan 20, 2020
c727498
Continue refactoring for more AWS-like schema
blake-wilson Jan 20, 2020
917cddf
Fix some tests
blake-wilson Jan 20, 2020
4026120
Populate empty application config at read time
blake-wilson Jan 20, 2020
2f0e384
Use existing utility function for runtime comparison
blake-wilson Jan 20, 2020
6e6b653
Remove ineffectual assignment
blake-wilson Jan 20, 2020
d31eeab
Separate logic from excessively long function into other functions
blake-wilson Jan 21, 2020
84df0d8
Fix flink config expansion so empty values are not expanded
blake-wilson Jan 21, 2020
f58a82c
Merge remote-tracking branch 'upstream/master' into add-flink-support
blake-wilson Jan 26, 2020
f3b0de1
Handle computed diffs better by using computed for map schemas
blake-wilson Jan 29, 2020
6b24c09
Merge remote-tracking branch 'upstream/master' into add-flink-support
blake-wilson Jan 29, 2020
aaf4109
Simplify return statement
blake-wilson Jan 29, 2020
1e53f0d
Merge remote-tracking branch 'upstream/master' into add-flink-support
blake-wilson Jan 31, 2020
772c3cc
Split kinesis analytics application created with v2 API into separate
blake-wilson Mar 13, 2020
ba74ed0
Merge remote-tracking branch 'upstream/master' into add-flink-support
blake-wilson Mar 13, 2020
4b62816
Merge branch 'add-flink-support' into add-flink-support-split-v2-reso…
blake-wilson Mar 13, 2020
17b00a5
Merge remote-tracking branch 'upstream/master' into add-flink-support
blake-wilson Mar 29, 2020
20f1ce2
Fix reference to v2 test config
blake-wilson Mar 30, 2020
47e31a4
Rename KinesisAnalyticsApplicationV2 -> KinesisAnalyticsV2Application
blake-wilson May 9, 2020
c123e22
Finish renaming aws_kinesis_analytics_application_v2 to
blake-wilson May 9, 2020
190549d
Merge remote-tracking branch 'upstream/master' into add-flink-support
blake-wilson May 9, 2020
0b926c6
Use TypeSet instead of TypeMap for Flink's checkpoint_configuration,
blake-wilson May 10, 2020
e6e9c92
Namespace kinesisanalyticsv2 validation functions
blake-wilson May 10, 2020
6816b31
Simplify validators with StringInSlice
blake-wilson May 10, 2020
d400a28
Fix metrics level validation
blake-wilson May 10, 2020
823d95e
Begin work to revise schema to more closely match AWS API
blake-wilson May 25, 2020
3f91224
Continue work on schema revisions
blake-wilson May 27, 2020
42e700d
Continue schema revisions
blake-wilson May 27, 2020
39d3f81
Continue work on schema revisions
blake-wilson May 27, 2020
403e796
Continue work on schema revisions
blake-wilson May 27, 2020
014efdc
Continue revising schema
blake-wilson May 28, 2020
d9ce938
Fix updating
blake-wilson May 28, 2020
f62e6d4
Continue revising schema
blake-wilson Jun 12, 2020
0cfaf71
Continue revising schema
blake-wilson Jun 12, 2020
2d381b4
Continue revising schema and patching up tests
blake-wilson Jun 20, 2020
a608ae0
Fix more tests. Fix issue with code config update not applied
blake-wilson Jun 20, 2020
a2a39b5
Fix another test
blake-wilson Jun 20, 2020
604be2a
Fix another test
blake-wilson Jun 20, 2020
8d8c309
Fix another test
blake-wilson Jun 20, 2020
ea9e809
Fix remaining tests
blake-wilson Jun 21, 2020
72445ac
Add test for application disappears
blake-wilson Jun 21, 2020
04509f0
Use composeConfig for combining test configs
blake-wilson Jun 21, 2020
75da72e
Remove debugging print statements
blake-wilson Jun 21, 2020
09f2bd7
Merge remote-tracking branch 'upstream/master' into add-flink-support
blake-wilson Jun 21, 2020
81dd75e
Namespace hash functions to kinesisanalyticsv2 application
blake-wilson Jun 21, 2020
afa0920
Convert length one sets to lists (fix linter S018 warnings)
blake-wilson Jun 21, 2020
b6dad4d
Fix linter errors
blake-wilson Jun 21, 2020
ee6aa28
Merge remote-tracking branch 'upstream/master' into add-flink-support
blake-wilson Sep 12, 2020
d7a0bf5
Clean up assertion of set contents
blake-wilson Sep 12, 2020
a64290e
Merge remote-tracking branch 'upstream/master' into add-flink-support
blake-wilson Sep 12, 2020
6bf9891
Merge branch 'master' into f-aws_kinesisanalyticsv2_application
ewbankkit Sep 25, 2020
39c2507
r/aws_kinesisanalyticsv2_application: Rename source files.
ewbankkit Sep 25, 2020
c83a212
r/aws_kinesisanalyticsv2_application: 'aws_kinesis_analyticsv2_applic…
ewbankkit Sep 25, 2020
faa346b
r/aws_kinesisanalyticsv2_application: 'resName' -> 'resourceName'.
ewbankkit Sep 25, 2020
3421214
r/aws_kinesisanalyticsv2_application: Remove 'testAccCheckKinesisAnal…
ewbankkit Sep 25, 2020
2120062
r/aws_kinesisanalyticsv2_application: Add and use internal 'finder' p…
ewbankkit Sep 25, 2020
fa19de6
r/aws_kinesisanalyticsv2_application: Add and use internal 'waiter' p…
ewbankkit Sep 25, 2020
f0df67d
r/aws_kinesisanalyticsv2_application: Add test sweeper.
ewbankkit Sep 25, 2020
ad3d88a
r/aws_kinesisanalyticsv2_application: Tidy up acceptance tests.
ewbankkit Sep 26, 2020
88fca01
r/aws_kinesisanalyticsv2_application: Start to add documentation.
ewbankkit Sep 27, 2020
c329eca
r/aws_kinesisanalyticsv2_application: Call Read after Create, not Upd…
ewbankkit Sep 28, 2020
0f99be0
r/aws_kinesisanalyticsv2_application: Add a true 'Basic' configuration.
ewbankkit Sep 28, 2020
b1c0251
r/aws_kinesisanalyticsv2_application: Update 'service_execution_role'.
ewbankkit Sep 28, 2020
34a61bb
r/aws_kinesisanalyticsv2_application: Standardize tags tests.
ewbankkit Sep 28, 2020
76227c9
r/aws_kinesisanalyticsv2_application: Fix 'TestAccAWSKinesisAnalytics…
ewbankkit Sep 28, 2020
fea2ede
r/aws_kinesisanalyticsv2_application: Make 'application_configuration…
ewbankkit Sep 28, 2020
bf37f22
r/aws_kinesisanalyticsv2_application: Handle 'cloudwatch_logging_opti…
ewbankkit Sep 28, 2020
117bead
r/aws_kinesisanalyticsv2_application: Handle 'application_code_config…
ewbankkit Sep 29, 2020
d1c1118
r/aws_kinesisanalyticsv2_application: Handle 'application_snapshot_co…
ewbankkit Sep 29, 2020
0779a13
r/aws_kinesisanalyticsv2_application: Handle 'flink_application_confi…
ewbankkit Sep 29, 2020
c51c8e7
r/aws_kinesisanalyticsv2_application: Handle 'environment_properties'…
ewbankkit Sep 29, 2020
6198885
r/aws_kinesisanalyticsv2_application: Handle 'sql_application_configu…
ewbankkit Sep 30, 2020
0fdc924
r/aws_kinesisanalyticsv2_application: Add 'TestAccAWSKinesisAnalytics…
ewbankkit Oct 4, 2020
267ab44
r/aws_kinesisanalyticsv2_application: Add 'TestAccAWSKinesisAnalytics…
ewbankkit Oct 5, 2020
5be8d9c
r/aws_kinesisanalyticsv2_application: Add 'TestAccAWSKinesisAnalytics…
ewbankkit Oct 6, 2020
ec859f5
r/aws_kinesisanalyticsv2_application: Add 'TestAccAWSKinesisAnalytics…
ewbankkit Oct 6, 2020
b8b0d62
Merge branch 'master' into f-aws_kinesisanalyticsv2_application
ewbankkit Oct 6, 2020
8cea2b8
Move TimedOut check inside 'kinesisAnalyticsV2RetryIAMEventualConsist…
ewbankkit Oct 6, 2020
2b543d8
r/aws_kinesisanalyticsv2_application: Add 'TestAccAWSKinesisAnalytics…
ewbankkit Oct 6, 2020
97c8bdd
r/aws_kinesisanalyticsv2_application: Add 'TestAccAWSKinesisAnalytics…
ewbankkit Oct 6, 2020
71f5a5d
r/aws_kinesisanalyticsv2_application: Add 'TestAccAWSKinesisAnalytics…
ewbankkit Oct 6, 2020
1544dc9
r/aws_kinesisanalyticsv2_application: Add 'TestAccAWSKinesisAnalytics…
ewbankkit Oct 6, 2020
b67f637
Remove unused code.
ewbankkit Oct 7, 2020
5921efd
r/aws_kinesisanalyticsv2_application: Reorder tests.
ewbankkit Oct 7, 2020
7fc789d
r/aws_kinesisanalyticsv2_application: Add 'TestAccAWSKinesisAnalytics…
ewbankkit Oct 7, 2020
2002dbe
r/aws_kinesisanalyticsv2_application: Add 'TestAccAWSKinesisAnalytics…
ewbankkit Oct 7, 2020
18d79c5
r/aws_kinesisanalyticsv2_application: Add 'TestAccAWSKinesisAnalytics…
ewbankkit Oct 8, 2020
b149d0d
r/aws_kinesisanalyticsv2_application: Add 'TestAccAWSKinesisAnalytics…
ewbankkit Oct 8, 2020
6eb2296
r/aws_kinesisanalyticsv2_application: Add 'TestAccAWSKinesisAnalytics…
ewbankkit Oct 8, 2020
ee5afac
Acceptance test output:
ewbankkit Oct 8, 2020
754879d
Modify documentation sub-category to reflect recent product repositio…
ewbankkit Oct 8, 2020
d451533
r/aws_kinesis_analytics_application: Change documentation sub-category.
ewbankkit Oct 8, 2020
4a8d644
Add documentation notes on usage of r/aws_kinesis_analytics_applicati…
ewbankkit Oct 8, 2020
002d1cb
r/aws_kinesisanalyticsv2_application: Add documentation examples.
ewbankkit Oct 8, 2020
8c28c62
r/aws_kinesisanalyticsv2_application: Move some code around.
ewbankkit Oct 8, 2020
687d4ae
Fix golangci-lint errors.
ewbankkit Oct 14, 2020
bc744d4
Fix 'Warning: List items should be accessed using square brackets (te…
ewbankkit Oct 14, 2020
74b4fa3
Fix markdown-lint errors.
ewbankkit Oct 14, 2020
1885acc
Acceptance test output:
ewbankkit Oct 14, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
20 changes: 20 additions & 0 deletions aws/internal/service/kinesisanalyticsv2/finder/finder.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
package finder

import (
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/kinesisanalyticsv2"
)

// ApplicationByName returns the application corresponding to the specified name.
func ApplicationByName(conn *kinesisanalyticsv2.KinesisAnalyticsV2, name string) (*kinesisanalyticsv2.ApplicationDetail, error) {
input := &kinesisanalyticsv2.DescribeApplicationInput{
ApplicationName: aws.String(name),
}

output, err := conn.DescribeApplication(input)
if err != nil {
return nil, err
}

return output.ApplicationDetail, nil
}
31 changes: 31 additions & 0 deletions aws/internal/service/kinesisanalyticsv2/waiter/status.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
package waiter

import (
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/kinesisanalyticsv2"
"github.com/hashicorp/aws-sdk-go-base/tfawserr"
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource"
"github.com/terraform-providers/terraform-provider-aws/aws/internal/service/kinesisanalyticsv2/finder"
)

const (
applicationStatusNotFound = "NotFound"
applicationStatusUnknown = "Unknown"
)

// ApplicationStatus fetches the Application and its Status
func ApplicationStatus(conn *kinesisanalyticsv2.KinesisAnalyticsV2, name string) resource.StateRefreshFunc {
return func() (interface{}, string, error) {
application, err := finder.ApplicationByName(conn, name)

if tfawserr.ErrCodeEquals(err, kinesisanalyticsv2.ErrCodeResourceNotFoundException) {
return nil, applicationStatusNotFound, nil
}

if err != nil {
return nil, applicationStatusUnknown, err
}

return application, aws.StringValue(application.ApplicationStatus), nil
}
}
26 changes: 26 additions & 0 deletions aws/internal/service/kinesisanalyticsv2/waiter/waiter.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
package waiter

import (
"time"

"github.com/aws/aws-sdk-go/service/kinesisanalyticsv2"
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource"
)

// ApplicationDeleted waits for an Application to return Deleted
func ApplicationDeleted(conn *kinesisanalyticsv2.KinesisAnalyticsV2, name string, timeout time.Duration) (*kinesisanalyticsv2.ApplicationDetail, error) {
stateConf := &resource.StateChangeConf{
Pending: []string{kinesisanalyticsv2.ApplicationStatusDeleting},
Target: []string{},
Refresh: ApplicationStatus(conn, name),
Timeout: timeout,
}

outputRaw, err := stateConf.WaitForState()

if v, ok := outputRaw.(*kinesisanalyticsv2.ApplicationDetail); ok {
return v, err
}

return nil, err
}
3 changes: 2 additions & 1 deletion aws/provider.go
Original file line number Diff line number Diff line change
Expand Up @@ -699,9 +699,10 @@ func Provider() *schema.Provider {
"aws_iot_topic_rule": resourceAwsIotTopicRule(),
"aws_iot_role_alias": resourceAwsIotRoleAlias(),
"aws_key_pair": resourceAwsKeyPair(),
"aws_kinesis_analytics_application": resourceAwsKinesisAnalyticsApplication(),
"aws_kinesisanalyticsv2_application": resourceAwsKinesisAnalyticsV2Application(),
"aws_kinesis_firehose_delivery_stream": resourceAwsKinesisFirehoseDeliveryStream(),
"aws_kinesis_stream": resourceAwsKinesisStream(),
"aws_kinesis_analytics_application": resourceAwsKinesisAnalyticsApplication(),
"aws_kinesis_video_stream": resourceAwsKinesisVideoStream(),
"aws_kms_alias": resourceAwsKmsAlias(),
"aws_kms_external_key": resourceAwsKmsExternalKey(),
Expand Down