-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] Streaming POC #6291
base: master
Are you sure you want to change the base?
[WIP] Streaming POC #6291
Conversation
Based on #6288
PR Description updated to latest commit (d2ca309) |
API Changes --- prev.txt 2024-06-30 18:43:35.438528688 +0000
+++ current.txt 2024-06-30 18:43:32.546505401 +0000
@@ -5591,6 +5591,8 @@
// OAS holds the configuration for various OpenAPI-specific functionalities
OAS OASConfig `json:"oas_config"`
+
+ Labs labsConfig `json:"labs"`
}
Config is the configuration object used by Tyk to set up various parameters.
@@ -7425,6 +7427,9 @@
The name for event handlers as defined in the API Definition JSON/BSON
format
+const (
+ ExtensionTykStreaming = "x-tyk-streaming"
+)
const ListDetailed = "detailed"
const LoopScheme = "tyk"
const MaxBackoffRetries = 4
@@ -7692,6 +7697,9 @@
func CloneAPI(a *APISpec) *APISpec
+func (s *APISpec) AddUnloadHook(hook func())
+ AddUnloadHook adds a function to be called when the API spec is unloaded
+
func (a *APISpec) CheckSpecMatchesStatus(r *http.Request, rxPaths []URLSpec, mode URLStatus) (bool, interface{})
CheckSpecMatchesStatus checks if a url spec has a specific status
@@ -7707,9 +7715,6 @@
func (a *APISpec) Init(authStore, sessionStore, healthStore, orgStore storage.Handler)
-func (s *APISpec) Release()
- Release releases all resources associated with API spec
-
func (a *APISpec) RequestValid(r *http.Request) (bool, RequestStatus)
RequestValid will check if an incoming request has valid version data and
return a RequestStatus that describes the status of the request
@@ -7723,6 +7728,9 @@
func (a *APISpec) URLAllowedAndIgnored(r *http.Request, rxPaths []URLSpec, whiteListStatus bool) (RequestStatus, interface{})
URLAllowedAndIgnored checks if a url is allowed and ignored.
+func (s *APISpec) Unload()
+ Release releases all resources associated with API spec
+
func (s *APISpec) Validate(oasConfig config.OASConfig) error
Validate returns nil if s is a valid spec and an error stating why the spec
is not valid.
@@ -7815,6 +7823,8 @@
FireEvent is added to the BaseMiddleware object so it is available across
the entire stack
+func (t *BaseMiddleware) GetSpec() *APISpec
+
func (t *BaseMiddleware) Init()
func (t *BaseMiddleware) Logger() (logger *logrus.Entry)
@@ -7829,6 +7839,8 @@
func (t *BaseMiddleware) SetRequestLogger(r *http.Request)
+func (t *BaseMiddleware) Unload()
+
func (t *BaseMiddleware) UpdateRequestSession(r *http.Request) bool
type BaseTykResponseHandler struct {
@@ -10057,6 +10069,25 @@
type StatsDSinkSanitizationFunc func(*bytes.Buffer, string)
+type StreamingMiddleware struct {
+ *BaseMiddleware
+
+ // Has unexported fields.
+}
+ StreamingMiddleware is a middleware that handles streaming functionality
+
+func (s *StreamingMiddleware) EnabledForSpec() bool
+
+func (s *StreamingMiddleware) Init()
+ Init initializes the middleware
+
+func (s *StreamingMiddleware) Name() string
+
+func (s *StreamingMiddleware) ProcessRequest(w http.ResponseWriter, r *http.Request, _ interface{}) (error, int)
+ ProcessRequest will handle the streaming functionality
+
+func (s *StreamingMiddleware) Unload()
+
type StripAuth struct {
*BaseMiddleware
}
@@ -10287,6 +10318,10 @@
ProcessRequest(w http.ResponseWriter, r *http.Request, conf interface{}) (error, int) // Handles request
EnabledForSpec() bool
Name() string
+
+ GetSpec() *APISpec
+
+ Unload()
}
type TykOsinServer struct { |
PR Review 🔍
Code feedback:
|
PR Code Suggestions ✨
|
💥 CI tests failed 🙈git-stateall ok Please look at the run or in the Checks tab. |
💥 CI tests failed 🙈git-stateall ok Please look at the run or in the Checks tab. |
💥 CI tests failed 🙈git-stateall ok Please look at the run or in the Checks tab. |
💥 CI tests failed 🙈git-stateall ok Please look at the run or in the Checks tab. |
1 similar comment
💥 CI tests failed 🙈git-stateall ok Please look at the run or in the Checks tab. |
💥 CI tests failed 🙈git-stateall ok Please look at the run or in the Checks tab. |
💥 CI tests failed 🙈git-stateall ok Please look at the run or in the Checks tab. |
…isconnected would cause a send on closed channel
💥 CI tests failed 🙈git-stateall ok Please look at the run or in the Checks tab. |
1 similar comment
💥 CI tests failed 🙈git-stateall ok Please look at the run or in the Checks tab. |
💥 CI tests failed 🙈git-stateall ok Please look at the run or in the Checks tab. |
💥 CI tests failed 🙈git-stateall ok Please look at the run or in the Checks tab. |
Quality Gate failedFailed conditions |
…ddleware This commit introduces a major change in the streaming architecture: - Replace shared Redis streams with direct user-to-stream connections - Each user now has their own dedicated stream, improving isolation and performance Additional improvements include: - Refactor StreamingMiddleware to use consumer group managers - Update StreamManager to use sync.Map for thread-safe operations - Remove Redis dependency and related functionality - Implement garbage collection for unused consumer groups - Update tests to reflect new structure and functionality - Add support for Kafka in streaming tests - Improve error handling and logging throughout - Update go.mod and go.sum with new dependencies These changes significantly enhance the scalability and efficiency of the streaming system by establishing direct connections between users and their respective streams, eliminating the need for a shared Redis-based message queue.
This commit refactors the `internal/streaming/manager.go` file to update the `Stream` struct and its methods. The main changes include: 1. Remove unused imports: `crypto/sha256` and `sync`. 2. Update `addMetadata`, `GetHTTPPaths`, `removeUnsafe`, and `removeConsumerGroup` methods to work with the `Stream` struct instead of `StreamManager`. 3. Remove references to `StreamManager` and update method receivers to use `Stream`. 4. Update `GetHTTPPaths` to use the `streamConfig` field of the `Stream` struct instead of loading from a map. These changes align the `Stream` struct with the new design, allowing it to manage a single stream configuration and its associated operations.
This commit enhances the Stream.Stop() method to make it more robust and prevent nil pointer dereferences. The changes include: 1. Adding a nil check for the stream before attempting to stop it. 2. Implementing a timeout mechanism using context to prevent indefinite hanging. 3. Using a goroutine and channel for asynchronous stream stopping. 4. Improving logging to provide more detailed information about the stopping process. These modifications aim to resolve issues related to stopping non-existent streams and make the overall process more reliable.
This commit improves the `testAsyncAPIHttp` function in the `gateway/mw_streaming_test.go` file to increase test reliability and provide more detailed logging. The changes include: 1. Increase initial WebSocket connection delay from 1 to 2 seconds 2. Extend overall timeout from 10 to 30 seconds 3. Increase inactivity timeout from 2 to 5 seconds 4. Add final log message showing total received messages These modifications aim to give the test more time to receive messages and offer more comprehensive logging information, facilitating better diagnosis of potential issues in message reception.
This commit improves the `testAsyncAPIHttp` function in the `mw_streaming_test.go` file to provide better debugging information and increase the chances of successful message reception. The changes include: 1. Increase initial WebSocket connection stabilization delay to 5 seconds 2. Add logging for WebSocket connection stabilization 3. Extend overall timeout to 60 seconds 4. Increase inactivity timeout to 10 seconds 5. Add a 5-second delay before closing WebSocket connections 6. Improve logging throughout the test execution These modifications aim to address potential timing issues and provide more detailed information about the test's progress, facilitating easier debugging of any remaining issues.
### **User description** <!-- Provide a general summary of your changes in the Title above --> ## Description Based off this POC [here](#6291) [TT-12893](https://tyktech.atlassian.net/browse/TT-12893) <!-- Describe your changes in detail --> ## Related Issue <!-- This project only accepts pull requests related to open issues. --> <!-- If suggesting a new feature or change, please discuss it in an issue first. --> <!-- If fixing a bug, there should be an issue describing it with steps to reproduce. --> <!-- OSS: Please link to the issue here. Tyk: please create/link the JIRA ticket. --> ## Motivation and Context <!-- Why is this change required? What problem does it solve? --> ## How This Has Been Tested <!-- Please describe in detail how you tested your changes --> <!-- Include details of your testing environment, and the tests --> <!-- you ran to see how your change affects other areas of the code, etc. --> <!-- This information is helpful for reviewers and QA. --> ## Screenshots (if appropriate) ## Types of changes <!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: --> - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Refactoring or add test (improvements in base code or adds test coverage to functionality) ## Checklist <!-- Go over all the following points, and put an `x` in all the boxes that apply --> <!-- If there are no documentation updates required, mark the item as checked. --> <!-- Raise up any additional concerns not covered by the checklist. --> - [ ] I ensured that the documentation is up to date - [ ] I explained why this PR updates go.mod in detail with reasoning why it's required - [ ] I would like a code coverage CI quality gate exception and have explained why [TT-12893]: https://tyktech.atlassian.net/browse/TT-12893?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ ___ ### **PR Type** Enhancement, Tests ___ ### **Description** - Implemented `StreamingMiddleware` to handle streaming functionality, including stream management and integration with API specifications. - Added comprehensive tests for streaming API, covering single and multiple client scenarios, asynchronous API, and WebSocket connections. - Developed `Stream` struct for managing streaming configurations, with methods for starting, stopping, and resetting streams. - Introduced `PortalClient` for interacting with the developer portal, including methods to list webhook credentials and fetch app details. - Implemented `portalOutput` for sending messages to webhooks, with configuration and connection handling. - Enhanced middleware interface to support unloading functionality, with hooks for unloading middleware when API specs change. - Updated dependencies to support new streaming and portal functionalities, including Kafka, NATS, and Benthos integration. ___ ### **Changes walkthrough** 📝 <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><details><summary>3 files</summary><table> <tr> <td> <details> <summary><strong>mw_streaming_test.go</strong><dd><code>Add comprehensive tests for streaming API functionality</code> </dd></summary> <hr> gateway/mw_streaming_test.go <li>Added tests for streaming API with single and multiple clients.<br> <li> Implemented setup functions for streaming API tests.<br> <li> Included test cases for asynchronous API and WebSocket connections.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6496/files#diff-a0d1bd0196a741537a3c850e340225c8993e49d709c838af0f1b48b9893af1da">+670/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>manager_test.go</strong><dd><code>Implement tests for Stream management and safety checks</code> </dd></summary> <hr> internal/streaming/manager_test.go <li>Added tests for starting and stopping streams.<br> <li> Tested removal and whitelisting of unsafe components.<br> <li> Verified stream configuration handling.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6496/files#diff-c3ed576ae7d6430b9ac57b46c39d64981c274188c807ff4f0cfc4f15753e67af">+151/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>portal_test.go</strong><dd><code>Add tests for PortalClient webhook credential listing</code> </dd></summary> <hr> internal/portal/portal_test.go <li>Added mock server for testing portal client interactions.<br> <li> Implemented tests for listing webhook credentials.<br> <li> Verified correct handling of multiple apps and webhooks.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6496/files#diff-2d5d9f02d0c6c149d531f5471b69936ccbf414a02d977813803fc3eafe15052d">+76/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Enhancement</strong></td><td><details><summary>6 files</summary><table> <tr> <td> <details> <summary><strong>mw_streaming.go</strong><dd><code>Implement StreamingMiddleware for API streaming functionality</code></dd></summary> <hr> gateway/mw_streaming.go <li>Implemented <code>StreamingMiddleware</code> for handling streaming functionality.<br> <li> Added methods for initializing, creating, and removing streams.<br> <li> Integrated stream management with API specifications.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6496/files#diff-6f565750150d990575c808f1ca8f38483160dc6edf05f1534cd0bedb27c2e6c8">+298/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>manager.go</strong><dd><code>Develop Stream struct for managing streaming configurations</code></dd></summary> <hr> internal/streaming/manager.go <li>Created <code>Stream</code> struct for managing streaming configurations.<br> <li> Added methods for starting, stopping, and resetting streams.<br> <li> Implemented safety checks for removing unsafe components.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6496/files#diff-8cda02479026085d17a6c5559a15bf59638174d16fb5f3ad65bebcecb25ad8d8">+236/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>portal_client.go</strong><dd><code>Add PortalClient for developer portal interactions</code> </dd></summary> <hr> internal/portal/portal_client.go <li>Introduced <code>PortalClient</code> for interacting with the developer portal.<br> <li> Added methods to list webhook credentials and fetch app details.<br> <li> Defined structures for app and webhook details.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6496/files#diff-0752a118f5d1513887f5ebf9930ae9c382a16798c3a4a301dac34523b5010289">+152/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>portal_output.go</strong><dd><code>Implement portalOutput for webhook message delivery</code> </dd></summary> <hr> internal/portal/portal_output.go <li>Implemented <code>portalOutput</code> for sending messages to webhooks.<br> <li> Added configuration and connection handling for portal output.<br> <li> Registered output plugin with Benthos service.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6496/files#diff-5225b7c3a3d3677eed2edb98be754e5cc70a124b0af26485511a0528f535048b">+128/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>api_loader.go</strong><dd><code>Integrate StreamingMiddleware into API loading process</code> </dd></summary> <hr> gateway/api_loader.go <li>Integrated <code>StreamingMiddleware</code> into the API loading process.<br> <li> Adjusted logic for unloading API specifications.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6496/files#diff-cdf0b7f176c9d18e1a314b78ddefc2cb3a94b3de66f1f360174692c915734c68">+9/-5</a> </td> </tr> <tr> <td> <details> <summary><strong>middleware.go</strong><dd><code>Enhance middleware interface with unload functionality</code> </dd></summary> <hr> gateway/middleware.go <li>Enhanced middleware interface with unload functionality.<br> <li> Added hooks for unloading middleware when API specs change.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6496/files#diff-703054910891a4db633eca0f42ed779d6b4fa75cd9b3aa4c503e681364201c1b">+15/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>1 files</summary><table> <tr> <td> <details> <summary><strong>config.go</strong><dd><code>Add StreamingConfig to global configuration settings</code> </dd></summary> <hr> config/config.go <li>Added <code>StreamingConfig</code> struct to configuration.<br> <li> Enabled streaming configuration in global settings.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6496/files#diff-fe44f09c4d5977b5f5eaea29170b6a0748819c9d02271746a20d81a5f3efca17">+26/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Dependencies</strong></td><td><details><summary>1 files</summary><table> <tr> <td> <details> <summary><strong>go.mod</strong><dd><code>Update dependencies for streaming and portal integration</code> </dd></summary> <hr> go.mod <li>Updated dependencies for streaming and portal functionalities.<br> <li> Added new modules for Kafka, NATS, and Benthos integration.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6496/files#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6">+307/-23</a></td> </tr> </table></details></td></tr><tr><td><strong>Additional files (token-limit)</strong></td><td><details><summary>1 files</summary><table> <tr> <td> <details> <summary><strong>go.sum</strong><dd><code>...</code> </dd></summary> <hr> go.sum ... </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6496/files#diff-3295df7234525439d778f1b282d146a4f1ff6b415248aaac074e8042d9f42d63">+1134/-65</a></td> </tr> </table></details></td></tr></tr></tbody></table> ___ > 💡 **PR-Agent usage**: >Comment `/help` on the PR to get a list of all available PR-Agent tools and their descriptions --------- Co-authored-by: Leonid Bugaev <leonsbox@gmail.com> Co-authored-by: Martin Buhr <martin@tyk.io>
### **User description** <!-- Provide a general summary of your changes in the Title above --> ## Description Based off this POC [here](#6291) [TT-12893](https://tyktech.atlassian.net/browse/TT-12893) ## Related Issue <!-- This project only accepts pull requests related to open issues. --> <!-- If suggesting a new feature or change, please discuss it in an issue first. --> <!-- If fixing a bug, there should be an issue describing it with steps to reproduce. --> <!-- OSS: Please link to the issue here. Tyk: please create/link the JIRA ticket. --> ## Motivation and Context <!-- Why is this change required? What problem does it solve? --> ## How This Has Been Tested <!-- Please describe in detail how you tested your changes --> <!-- Include details of your testing environment, and the tests --> <!-- you ran to see how your change affects other areas of the code, etc. --> <!-- This information is helpful for reviewers and QA. --> ## Screenshots (if appropriate) ## Types of changes <!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: --> - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Refactoring or add test (improvements in base code or adds test coverage to functionality) ## Checklist <!-- Go over all the following points, and put an `x` in all the boxes that apply --> <!-- If there are no documentation updates required, mark the item as checked. --> <!-- Raise up any additional concerns not covered by the checklist. --> - [ ] I ensured that the documentation is up to date - [ ] I explained why this PR updates go.mod in detail with reasoning why it's required - [ ] I would like a code coverage CI quality gate exception and have explained why [TT-12893]: https://tyktech.atlassian.net/browse/TT-12893?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ ___ ### **PR Type** Enhancement, Tests ___ ### **Description** - Implemented a new `StreamingMiddleware` to handle streaming API functionality, including stream initialization and request processing. - Added comprehensive tests for streaming API scenarios, including single and multiple client interactions. - Developed a `PortalClient` for managing webhook credentials and interacting with the developer portal. - Introduced a `Stream` struct for managing stream configurations and handling unsafe components. - Enhanced the middleware interface to support unload functionality and integrated it into the API loader. - Updated the configuration to include streaming settings and adjusted dependencies to support new features. ___ ### **Changes walkthrough** 📝 <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><details><summary>3 files</summary><table> <tr> <td> <details> <summary><strong>mw_streaming_test.go</strong><dd><code>Add comprehensive tests for streaming API functionality</code> </dd></summary> <hr> gateway/mw_streaming_test.go <li>Added tests for single and multiple client streaming API scenarios.<br> <li> Implemented setup functions for streaming API configurations.<br> <li> Included test cases for asynchronous API and WebSocket connections.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-a0d1bd0196a741537a3c850e340225c8993e49d709c838af0f1b48b9893af1da">+671/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>manager_test.go</strong><dd><code>Add unit tests for stream management functionality</code> </dd></summary> <hr> internal/streaming/manager_test.go <li>Added tests for starting and stopping streams.<br> <li> Implemented tests for removing and whitelisting unsafe components.<br> <li> Verified stream configuration handling and error scenarios.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-c3ed576ae7d6430b9ac57b46c39d64981c274188c807ff4f0cfc4f15753e67af">+152/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>portal_test.go</strong><dd><code>Add tests for portal client webhook credential retrieval</code> </dd></summary> <hr> internal/portal/portal_test.go <li>Added mock server for testing portal client interactions.<br> <li> Implemented tests for listing webhook credentials across multiple <br>apps.<br> <li> Verified correct handling of webhook data retrieval.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-2d5d9f02d0c6c149d531f5471b69936ccbf414a02d977813803fc3eafe15052d">+76/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Enhancement</strong></td><td><details><summary>6 files</summary><table> <tr> <td> <details> <summary><strong>mw_streaming.go</strong><dd><code>Implement streaming middleware for API gateway</code> </dd></summary> <hr> gateway/mw_streaming.go <li>Implemented <code>StreamingMiddleware</code> for handling streaming functionality.<br> <li> Added methods for initializing and managing streams.<br> <li> Introduced a mechanism to process requests and manage stream <br>lifecycles.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-6f565750150d990575c808f1ca8f38483160dc6edf05f1534cd0bedb27c2e6c8">+298/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>manager.go</strong><dd><code>Develop stream management and configuration handling</code> </dd></summary> <hr> internal/streaming/manager.go <li>Created <code>Stream</code> struct for managing stream configurations.<br> <li> Added methods to start, stop, and reset streams.<br> <li> Implemented functionality to remove unsafe components from stream <br>configurations.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-8cda02479026085d17a6c5559a15bf59638174d16fb5f3ad65bebcecb25ad8d8">+236/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>portal_client.go</strong><dd><code>Add portal client for webhook credential management</code> </dd></summary> <hr> internal/portal/portal_client.go <li>Introduced <code>PortalClient</code> for interacting with the developer portal.<br> <li> Added methods to list webhook credentials and fetch app details.<br> <li> Implemented client initialization with base URL and secret.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-0752a118f5d1513887f5ebf9930ae9c382a16798c3a4a301dac34523b5010289">+152/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>portal_output.go</strong><dd><code>Implement portal output for webhook integration</code> </dd></summary> <hr> internal/portal/portal_output.go <li>Implemented portal output for sending messages to webhooks.<br> <li> Added configuration for portal URL, secret, and event types.<br> <li> Developed methods for connecting and writing to webhooks.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-5225b7c3a3d3677eed2edb98be754e5cc70a124b0af26485511a0528f535048b">+128/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>api_loader.go</strong><dd><code>Integrate streaming middleware into API loader</code> </dd></summary> <hr> gateway/api_loader.go <li>Integrated <code>StreamingMiddleware</code> into the API loading process.<br> <li> Adjusted logic for unloading and reloading API specs.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-cdf0b7f176c9d18e1a314b78ddefc2cb3a94b3de66f1f360174692c915734c68">+9/-5</a> </td> </tr> <tr> <td> <details> <summary><strong>middleware.go</strong><dd><code>Enhance middleware interface with unload capabilities</code> </dd></summary> <hr> gateway/middleware.go <li>Enhanced middleware interface with unload functionality.<br> <li> Updated middleware creation to include unload hooks.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-703054910891a4db633eca0f42ed779d6b4fa75cd9b3aa4c503e681364201c1b">+15/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>1 files</summary><table> <tr> <td> <details> <summary><strong>config.go</strong><dd><code>Introduce streaming configuration in main config</code> </dd></summary> <hr> config/config.go <li>Added <code>StreamingConfig</code> struct to manage streaming settings.<br> <li> Integrated streaming configuration into the main config structure.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-fe44f09c4d5977b5f5eaea29170b6a0748819c9d02271746a20d81a5f3efca17">+26/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Dependencies</strong></td><td><details><summary>1 files</summary><table> <tr> <td> <details> <summary><strong>go.mod</strong><dd><code>Update dependencies for streaming and portal integration</code> </dd></summary> <hr> go.mod <li>Updated dependencies to include streaming and portal-related packages.<br> <li> Adjusted versions for existing dependencies.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6">+307/-23</a></td> </tr> </table></details></td></tr><tr><td><strong>Additional files (token-limit)</strong></td><td><details><summary>1 files</summary><table> <tr> <td> <details> <summary><strong>go.sum</strong><dd><code>...</code> </dd></summary> <hr> go.sum ... </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-3295df7234525439d778f1b282d146a4f1ff6b415248aaac074e8042d9f42d63">+1134/-65</a></td> </tr> </table></details></td></tr></tr></tbody></table> ___ > 💡 **PR-Agent usage**: >Comment `/help` on the PR to get a list of all available PR-Agent tools and their descriptions --------- Co-authored-by: Leonid Bugaev <leonsbox@gmail.com> Co-authored-by: Martin Buhr <martin@tyk.io> Co-authored-by: Burak Sezer <burak.sezer.developer@gmail.com>
User description
Based on #6288
The main changes that Streaming configuration is now part of OAS API definition "x-tyk-streaming", and can look like this:
PR Type
Enhancement, Tests, Documentation
Description
StreamingMiddleware
for handling streaming functionality with WebSocket and SSE support.StreamManager
for managing stream configurations and Redis integration.PortalClient
andportalOutput
for interacting with the developer portal and posting messages to webhooks.StreamingMiddleware
into the API loading process and added unload hooks toAPISpec
.Changes walkthrough 📝
8 files
mw_streaming.go
Introduce Streaming Middleware with WebSocket and SSE Support
gateway/mw_streaming.go
StreamingMiddleware
for handling streaming functionality.manager.go
Implement StreamManager for Stream Configuration and Management
internal/streaming/manager.go
StreamManager
for managing stream configurations.portal_client.go
Implement PortalClient for Developer Portal Interaction
internal/portal/portal_client.go
PortalClient
for interacting with the developer portal.portal_output.go
Add PortalOutput for Posting Messages to Webhooks
internal/portal/portal_output.go
portalOutput
for posting messages to webhooks.PortalClient
to fetch webhook credentials.api_loader.go
Integrate StreamingMiddleware into API Loading Process
gateway/api_loader.go
StreamingMiddleware
into the API loading process.middleware.go
Enhance TykMiddleware Interface with Spec Retrieval and Unloading
gateway/middleware.go
TykMiddleware
interface for spec retrieval andunloading.
api_definition.go
Add Unload Hooks to APISpec
gateway/api_definition.go
APISpec
.Release
method to call unload hooks.mw_context_vars.go
Modify EnabledForSpec Method in MiddlewareContextVars
gateway/mw_context_vars.go
EnabledForSpec
method to always return true.5 files
mw_streaming_test.go
Add Tests for StreamingMiddleware WebSocket and SSE Handling
gateway/mw_streaming_test.go
StreamingMiddleware
including WebSocket and SSEhandling.
manager_test.go
Add Unit Tests for StreamManager Functionality
internal/streaming/manager_test.go
StreamManager
.portal_test.go
Add Tests for PortalClient and Webhook Credential Listing
internal/portal/portal_test.go
PortalClient
and webhook credential listing.portal_output_test.go
Add Unit Tests for PortalOutput Plugin
internal/portal/portal_output_test.go
portalOutput
plugin.testutil.go
Update Test Utility for API Spec Unloading
gateway/testutil.go
1 files
config.go
Add StreamingConfig to Global Configuration
config/config.go
StreamingConfig
to global configuration.1 files
go.mod
Update Dependencies for Streaming and Portal Integration
go.mod
benthos
andgo-redis
.