Upgrade/aws sdk to v2 #1
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Change-Id: Ic8dc81a437d9c72e841048359745e245c6c4895b
This is to create configuration and client interface in order to give user an overview on how the Kinesis client library works. In order not to reinvent wheel, the api is designed closely aligned with Amazon Kinesis Client Library in Java. add errors. remove @throws and use @error instead. https://jira.eng.vmware.com/browse/CNA-614 Change-Id: I78a269b328c14df37f878eccef192ff022a669cc
This is the first part of implementing shard lease for Kinesis Client library. It creates dynamoDB table for managing Kinesis stream shard lease. https://jira.eng.vmware.com/browse/CNA-636 Adjust error code value range. Change-Id: I16565fa15332843101235fb14545ee69c2599f2f
Fix code bug for removing cyclic dependency and fix unit test. Test: hmake test Change-Id: Ib4d4ba416d0133542e6459459ddf43079ff53ab8
This is the core part of KCL by implementing worker. It has exactly the same interface as Amazon's KCL. Internally, it uses code from GoKini in order to get the library functionaly quickly. This is a working version. The test code worker_test.go shows how to use this library. Dynamic resharding feature is out of the scope of M4. Test: 1. A Kinesis stream named "kcl-test" has been created under photon-infra account. 2. Download your AWS Credential from IAM user page. 3. Modify the worker_test.go to fill in your aws credential. 4. hmake test Jira CNA-637 Change-Id: I886d255bab9adaf7a13bca11bfda51bedaacaaed
This change fixed the bug of not finding checkpoint when process restart. It also adds missing call to record processor for notifying the shard info and checkpoint when application first started. Test: Run hmake test and verify the log. Change-Id: I4bdf21ac10c5ee988a0860c140991f7d05975541
Add support for handling child/parent shard. When processing child shard, it has to wait until parent shard finished before processing itself. Change-Id: I8bbf104c22ae93409d856be9c6829988c1b2d7eb
go languaage doesn't like all-caps on const. Since KCL is mainly from Amazon's KCL, we'd like the constant to have the exactly same name as Amazon's KCL. Thefore, skip the lint check. Change-Id: Ib8a2f52a8f4b44d814eda264f62fdcd53cccc2a7
This change enables metrics reporting and fixes a few bug in metrics reporting. The current metrics reporting is quite limited. Will add more metrics in next cr. Tested with both prometheus and cloudwatch. Jira CNA-702 Change-Id: I678b3f8a372d83f7b8adc419133c14cd10884f61
This changes fixed cloudwatch metrics publishing by adding long running go routine to periodically publish cloudwatch metrics. Also, shutdown metrics publishing when KCL is shutdown. Test: Run hmake test and verified cloudwatch metrics has been published via AWS cloudwatch console. Jira CNA-702 Change-Id: I78b347cd12939447b0daf93f51acf620d18e2f49
Need to remove shard not longer existed in Kinesis from shardStatus cache. Change-Id: I09b4a4c3c6480b8300fa937e6073dcd578156b29
Organize the folder structure in order to support imported as submodule for other services. Jira CNA-701 Change-Id: I1dda27934642bb8a7755df07dc4a5048449afc86
Need to remove lease entry in dynamodb table when shard has been removed by Kinesis. This happens when doing shard splitting and parent shard will be moved by Kinesis after its retention period (normally after 24 hours). Change-Id: I70a5836436ac0698110085d46d9438fcaf539cd2
cascade-kinesis-client will be used as a submodule of other projects, so it should not have "src/vmware.com/cascade-kinesis-client" in its path. To build this project locally, please manually create the parent folders. Change-Id: I8844e6a0e32aae65b28496915d8507e9fb1058c6
Fix bug when doing shard sync which removing shard info. Jira ID: CNA-612 Change-Id: Ibaf55fffa39b793abbfe3bd57999e5d37f82a52f
Change-Id: I164a3e551331464a020e82ad305294e5a659ab6c
There might be verious reason for shard iterator to expire, such as: not enough data in shard or process even takes more than 5 minutes which cause shard iterator not refreshing enough. This change removes log.Fatal which causes panic. Panic inside go routine will bring down the whole app. Therefore, just log error and exit the go routine instead. Jira ID: CNA-1072 Change-Id: I34a8d9af7258f3ea75465e2245bbc25c2fafee35
The processing Kinesis gets stuck after splitting shard. The
reason is that the app doesn't do mandatory checkpoint.
KCL document states:
// When the value of {@link ShutdownInput#getShutdownReason()} is
// {@link com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownReason#TERMINATE} it is required that you
// checkpoint. Failure to do so will result in an IllegalArgumentException, and the KCL no longer making progress.
Also, fix shard lease to prevent one host takes more shard than
its configuration allowed.
Jira CNA-1701
Change-Id: Icbdacaf347c7a67b5793647ad05ff93cca629741
All source should be prepared in a manner that reflects comments that VMware would be comfortable sharing with the public. Documentation only. No functional change. Update the license to MIT to be consistent with approved OSSTP product tracking ticket: https://osstp.vmware.com/oss/#/upstreamcontrib/project/1101391 Jira CNA-1117 Change-Id: I3fe31f10db954887481e3b21ccd20ec8e39c5996
gas is now gosec. Need to update security scan and fix security issue as needed. No functional change. Jira CNA-2022 Change-Id: I36f2a204114f3f13e2ed05579c04a9c89f528f9a
Fix the random number generator by adding seed. https://stackoverflow.com/questions/12321133/golang-random-number-generator-how-to-seed-properly Jira CNA-1119 Change-Id: Idfe23d84f31a47dcf43c8025632ff6f115614d34
After a few days of shard splitting, the parent shard will be deleted by Kinesis system. KCL should ignore the error caused by deleted parent shared and move on. Test: Manuall split shard on kcl-test stream in photon-infra account Currently, shard3 is the parent shard of shard 4 and 5. Shard 3 has a parent shard 0 which has been deleted already. Verified the test can run and not stuck in waiting for parent shard. Jira CNA-2089 Change-Id: I15ed0db70ff9836313c22ccabf934a2a69379248
Current, KCL doesn't release shard when returning on error which causes the worker cannot get any shard because it has the maximum number of shard already. This change makes sure releasing shard when return. update the log message. Test: Integration test by forcing error on reading shard to simulate Kinesis Internal error and make sure the KCL will not stop processing. Jira CNA-1995 Change-Id: Iac91579634a5023ab5ed73c6af89e4ff1a9af564
Update the readme and contributing doc before publishing to github repo. https://github.com/vmware/vmware-go-kcl Jira CNA-2036 Change-Id: Idd8cfd8c89d3202613ff1d3018a584945ad30e4a
Update import path in files when switching to github.
Fix the calculation of exponential backoff. ^ is the XOR in golang. Replaced it with math.exp2().
Adding min/max retry and throttle delay for the retryer. Also, increase the max retries to 10 which is inline with dynamodb default retry count. Signed-off-by: Tao Jiang <taoj@vmware.com>
Add support for Kinesis aggregation format to consume record published by KPL. Note: current implementation need to checkpoint the whole batch of the de-aggregated records instead of just portion of them. Add cache entry and exit time. Signed-off-by: Tao Jiang <taoj@vmware.com>
Minor fix on constant naming convention. Signed-off-by: Tao Jiang <taoj@vmware.com>
* clientlibrary/checkpoint: convert ErrLeaseAcquired to struct Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com> * clientlibrary/checkpoint: add context to ErrLeaseNotAcquired Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com> * Use errors.As to check for ErrLeaseNotAcquired error Signed-off-by: Aurélien Rainone <aurelien.rainone@gmail.com>
* Implement enhanced fan-out consumer Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com> * Add test cases Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com> * Small adjustments in fan-out consumer Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>
* Use ApplicationName as default for EnhancedFanOutConsumerName Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com> * Add tests Signed-off-by: Ilia Cimpoes <ilia.cimpoes@ellation.com>
Fixes #4 Signed-off-by: Connor McKelvey <connormckelvey@gmail.com> Signed-off-by: Ali Hobbs <alisuehobbs@gmail.com> Co-authored-by: Ali Hobbs <alisuehobbs@gmail.com> Co-authored-by: Ali Hobbs <alisuehobbs@gmail.com>
* improve log event Signed-off-by: lucarin91 <lucarin@protonmail.com> * use %+v in template string Signed-off-by: lucarin91 <lucarin@protonmail.com>
Signed-off-by: Tao Jiang <taoj@vmware.com>
* Added RS Zerolog logging adapter Signed-off-by: Mate Lang <mate.lang@welthee.com> * Fixed pull request review comments Signed-off-by: Mate Lang <mate.lang@welthee.com>
* upgrade to golang 1.17 Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com> # Conflicts: # go.mod # go.sum * improve after shell lint Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com> * improve after upgrade docker image (used by the build system) Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com> * remove not needed variable Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com> * apply fixes after security scan (hmake test) Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com> * add missing package after merge with latest master branch code. Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com> * improve docker layer Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com> * upgrade packages Signed-off-by: Fabiano Graças <fabiano.gracas@faro.com> Co-authored-by: Fabiano Graças <fabiano.gracas@faro.com>
… will not merge the PR soon.
…icitly bellow to avoid lint msgs.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Bring over the change from https://github.com/fafg/vmware-go-kcl/tree/upgrade/aws-sdk-to-v2 as the starting point of this repo.