[TT-10532] Update taskfile, CI tests

[titpetric]

This PR implements:

Taskfile.yml, .taskfiles/test.yml, taskfiles/deps/Taskfile.yml:

• added test:e2e and test:e2e-combined. By default they run without -race.
• included new CI dependencies in deps taskfile (provided with tykio/ci-tools for github CI, this is for local development)

ci-tests.yml:

• CI runs task test:e2e-combined args="-race -timeout=15m"
• CI runs task test:coverage to print per-package coverage report
• CI runs task test:report at to print slowest tests (only available in e2e-combined)
• CI Pipeline changes: run test environment from taskfiles
  • Cache moved up to happen before make lint (restore cache first)
  • Using tykio/ci-tools to get task, golangci-lint and other CI tooling, silence golangcilint config
  • Removed github comment notification, it's noise - it's a required pipeline

Pipeline for sonarcloud is now pinned to redis7, there is some unnecessary duplication due to the redis matrix and some actions like golangci-lint unfortunately run twice. Our next step should be to make the redis 7 pipeline required, and then remove the redis 5 pipeline, and then sanitize the state in a future review.

storage: test fix for a data race
gateway: rpc_test.go, agreed marking RPC test as flaky due to sensitivity to -race (ok'd with @sredxny)
docker/services: support REDIS_IMAGE to override for matrix test

https://tyktech.atlassian.net/browse/TT-10532

[github-actions]

API Changes

no api changes detected

[github-actions]

PR Reviewer Guide 🔍

⏱️ Estimated effort to review [1-5]	4
🧪 Relevant tests	Yes
🔒 Security concerns	No
⚡ Key issues to review	CI Configuration: The PR sets CI: 'true' which might skip flaky tests as mentioned in the PR description. This could potentially mask issues that only appear under certain conditions.
	Test Coverage: The use of head -n -2 in the test commands might inadvertently trim important output from the test results, potentially hiding errors or important warnings.
	Race Condition Handling: The PR includes separate tasks for running tests with and without the -race flag. It's crucial to ensure that tests which should be run with the race detector are indeed being executed in that mode, especially since RPC tests are mentioned to be problematic with -race.
	Dependency Management: The PR includes updates to task dependencies and introduces new tasks. It's important to ensure these dependencies are correctly managed to avoid issues in the build pipeline.

[github-actions]

PR Code Suggestions ✨

Category	Suggestion	Score
Best practice	Add error handling to shell commands in variable definitions to enhance robustness Consider adding error handling for the shell commands used in variable definitions to prevent failures that could terminate the task unexpectedly. For example, the go mod edit -json command could fail if not run within a Go module. .taskfiles/test.yml [39] -sh: go mod edit -json | jq .Module.Path -r +sh: go mod edit -json | jq .Module.Path -r || echo "Error: Not a Go module" Suggestion importance[1-10]: 8 Why: Adding error handling to shell commands is a good practice that enhances the robustness of the script. This suggestion addresses potential failures that could terminate the task unexpectedly, making it a valuable improvement.	8
	Use a more robust and readable method for string manipulation in the packages variable Replace the sed command with a more robust method for string manipulation in the packages variable. Using sed can be error-prone and less readable. Consider using a more direct method of string replacement that might be provided by the language or framework you are using. .taskfiles/test.yml [41] -sh: go list ./... | sed -e 's|{{.package}}|.|g' +sh: go list ./... | awk '{gsub("{{.package}}", ".", $0); print $0}' Suggestion importance[1-10]: 6 Why: The suggestion to replace sed with awk for string manipulation is valid as awk can be more readable and less error-prone. However, both commands are commonly used, and the change is more about preference and readability rather than fixing a critical issue.	6
Enhancement	Clarify the descriptions of similar tasks to avoid confusion and provide better context The integration and integration-combined tasks have very similar descriptions. It might be beneficial to differentiate these descriptions more clearly to avoid confusion and to better describe the specific purpose of each task. .taskfiles/test.yml [31] -desc: "Run e2e/integration tests" +desc: "Run individual e2e/integration tests" Suggestion importance[1-10]: 7 Why: Differentiating the descriptions of similar tasks can help avoid confusion and provide better context. This is a useful enhancement for maintainability and clarity, though it is not critical.	7
	Improve the default value for the tags variable to enhance clarity and context Consider using a more specific default value for the tags variable. The current default value 'goplugin' might not be descriptive enough for all scenarios where this Taskfile is used. A more specific default could help in understanding the context or purpose of the tests without needing to specify it every time. .taskfiles/test.yml [14] -tags: '{{ .tags | default "goplugin" }}' +tags: '{{ .tags | default "default-test-tag" }}' Suggestion importance[1-10]: 5 Why: While the suggestion to use a more specific default value for the tags variable could enhance clarity, it is not crucial and does not address any major issues. The current default value 'goplugin' is functional, and changing it to 'default-test-tag' is a minor improvement.	5

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud