[TT-1574] Multiple analytics keys in slaves gateway #3487
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/release to release-3 |
|
/release to release-3.2 |
|
Working on it! Note that it can take a few minutes. |
|
/release to release-3.0.5 |
|
Working on it! Note that it can take a few minutes. |
tykbot bot
pushed a commit
that referenced
this pull request
Mar 11, 2021
…ys (#3487) <!-- Provide a general summary of your changes in the Title above --> ## Description <!-- Describe your changes in detail --> Extends #3483 Modified the logic for rpc_analytics_purger so it can read from keys from multiple keys. It's coded in full backward-compatible mode, which means that it's going to look for `tyk-system-analytics` key and then for `tyk-system-analytics_X `where X goes from 0 to 9. After reading, it will send the analytics records to sink through RPC. If tyk-sink has enable_multiple_analytics_keys = true, it will write the records in multiple keys and if it's false, it will write them in only one key. ## Related Issue <!-- This project only accepts pull requests related to open issues --> <!-- If suggesting a new feature or change, please discuss it in an issue first --> <!-- If fixing a bug, there should be an issue describing it with steps to reproduce --> <!-- Please link to the issue here --> https://tyktech.atlassian.net/browse/TT-1574 ## Motivation and Context <!-- Why is this change required? What problem does it solve? --> Reduce Redis overload and have a better analytics distribution across the cluster for slave gateways. ## How This Has Been Tested <!-- Please describe in detail how you tested your changes --> <!-- Include details of your testing environment, and the tests you ran to see how your change affects other areas of the code, etc. --> ## Screenshots (if appropriate) ## Types of changes <!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: --> - [ ] Bug fix (non-breaking change which fixes an issue) - [X] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Refactoring or add test (improvements in base code or adds test coverage to functionality) ## Checklist <!-- Go over all the following points, and put an `x` in all the boxes that apply --> <!-- If you're unsure about any of these, don't hesitate to ask; we're here to help! --> - [X] Make sure you are requesting to **pull a topic/feature/bugfix branch** (right side). If pulling from your own fork, don't request your `master`! - [X] Make sure you are making a pull request against the **`master` branch** (left side). Also, you should start *your branch* off *our latest `master`*. - [] My change requires a change to the documentation. - [ ] If you've changed APIs, describe what needs to be updated in the documentation. - [ ] If new config option added, ensure that it can be set via ENV variable - [ ] I have updated the documentation accordingly. - [ ] Modules and vendor dependencies have been updated; run `go mod tidy && go mod vendor` - [ ] When updating library version must provide reason/explanation for this update. - [ ] I have added tests to cover my changes. - [X] All new and existing tests passed. - [X] Check your code additions will not fail linting checks: - [X] `go fmt -s` - [X] `go vet` (cherry picked from commit d0cf603)
|
@buger Succesfully merged |
|
Working on it! Note that it can take a few minutes. |
tykbot bot
pushed a commit
that referenced
this pull request
Mar 11, 2021
…ys (#3487) <!-- Provide a general summary of your changes in the Title above --> ## Description <!-- Describe your changes in detail --> Extends #3483 Modified the logic for rpc_analytics_purger so it can read from keys from multiple keys. It's coded in full backward-compatible mode, which means that it's going to look for `tyk-system-analytics` key and then for `tyk-system-analytics_X `where X goes from 0 to 9. After reading, it will send the analytics records to sink through RPC. If tyk-sink has enable_multiple_analytics_keys = true, it will write the records in multiple keys and if it's false, it will write them in only one key. ## Related Issue <!-- This project only accepts pull requests related to open issues --> <!-- If suggesting a new feature or change, please discuss it in an issue first --> <!-- If fixing a bug, there should be an issue describing it with steps to reproduce --> <!-- Please link to the issue here --> https://tyktech.atlassian.net/browse/TT-1574 ## Motivation and Context <!-- Why is this change required? What problem does it solve? --> Reduce Redis overload and have a better analytics distribution across the cluster for slave gateways. ## How This Has Been Tested <!-- Please describe in detail how you tested your changes --> <!-- Include details of your testing environment, and the tests you ran to see how your change affects other areas of the code, etc. --> ## Screenshots (if appropriate) ## Types of changes <!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: --> - [ ] Bug fix (non-breaking change which fixes an issue) - [X] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Refactoring or add test (improvements in base code or adds test coverage to functionality) ## Checklist <!-- Go over all the following points, and put an `x` in all the boxes that apply --> <!-- If you're unsure about any of these, don't hesitate to ask; we're here to help! --> - [X] Make sure you are requesting to **pull a topic/feature/bugfix branch** (right side). If pulling from your own fork, don't request your `master`! - [X] Make sure you are making a pull request against the **`master` branch** (left side). Also, you should start *your branch* off *our latest `master`*. - [] My change requires a change to the documentation. - [ ] If you've changed APIs, describe what needs to be updated in the documentation. - [ ] If new config option added, ensure that it can be set via ENV variable - [ ] I have updated the documentation accordingly. - [ ] Modules and vendor dependencies have been updated; run `go mod tidy && go mod vendor` - [ ] When updating library version must provide reason/explanation for this update. - [ ] I have added tests to cover my changes. - [X] All new and existing tests passed. - [X] Check your code additions will not fail linting checks: - [X] `go fmt -s` - [X] `go vet` (cherry picked from commit d0cf603)
|
@buger Succesfully merged |
tykbot bot
pushed a commit
that referenced
this pull request
Mar 11, 2021
…ys (#3487) <!-- Provide a general summary of your changes in the Title above --> ## Description <!-- Describe your changes in detail --> Extends #3483 Modified the logic for rpc_analytics_purger so it can read from keys from multiple keys. It's coded in full backward-compatible mode, which means that it's going to look for `tyk-system-analytics` key and then for `tyk-system-analytics_X `where X goes from 0 to 9. After reading, it will send the analytics records to sink through RPC. If tyk-sink has enable_multiple_analytics_keys = true, it will write the records in multiple keys and if it's false, it will write them in only one key. ## Related Issue <!-- This project only accepts pull requests related to open issues --> <!-- If suggesting a new feature or change, please discuss it in an issue first --> <!-- If fixing a bug, there should be an issue describing it with steps to reproduce --> <!-- Please link to the issue here --> https://tyktech.atlassian.net/browse/TT-1574 ## Motivation and Context <!-- Why is this change required? What problem does it solve? --> Reduce Redis overload and have a better analytics distribution across the cluster for slave gateways. ## How This Has Been Tested <!-- Please describe in detail how you tested your changes --> <!-- Include details of your testing environment, and the tests you ran to see how your change affects other areas of the code, etc. --> ## Screenshots (if appropriate) ## Types of changes <!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: --> - [ ] Bug fix (non-breaking change which fixes an issue) - [X] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Refactoring or add test (improvements in base code or adds test coverage to functionality) ## Checklist <!-- Go over all the following points, and put an `x` in all the boxes that apply --> <!-- If you're unsure about any of these, don't hesitate to ask; we're here to help! --> - [X] Make sure you are requesting to **pull a topic/feature/bugfix branch** (right side). If pulling from your own fork, don't request your `master`! - [X] Make sure you are making a pull request against the **`master` branch** (left side). Also, you should start *your branch* off *our latest `master`*. - [] My change requires a change to the documentation. - [ ] If you've changed APIs, describe what needs to be updated in the documentation. - [ ] If new config option added, ensure that it can be set via ENV variable - [ ] I have updated the documentation accordingly. - [ ] Modules and vendor dependencies have been updated; run `go mod tidy && go mod vendor` - [ ] When updating library version must provide reason/explanation for this update. - [ ] I have added tests to cover my changes. - [X] All new and existing tests passed. - [X] Check your code additions will not fail linting checks: - [X] `go fmt -s` - [X] `go vet` (cherry picked from commit d0cf603)
|
@buger Succesfully merged |
|
@tbuchaillot would be helpful to have information about how was this tested |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Extends #3483
Modified the logic for rpc_analytics_purger so it can read from keys from multiple keys. It's coded in full backward-compatible mode, which means that it's going to look for
tyk-system-analyticskey and then fortyk-system-analytics_Xwhere X goes from 0 to 9.After reading, it will send the analytics records to sink through RPC. If tyk-sink has enable_multiple_analytics_keys = true, it will write the records in multiple keys and if it's false, it will write them in only one key.
Related Issue
https://tyktech.atlassian.net/browse/TT-1574
Motivation and Context
Reduce Redis overload and have a better analytics distribution across the cluster for slave gateways.
How This Has Been Tested
Screenshots (if appropriate)
Types of changes
Checklist
fork, don't request your
master!masterbranch (left side). Also, you should startyour branch off our latest
master.go mod tidy && go mod vendorgo fmt -sgo vet