ci: Merge changes from v2 to release 2.8 branch (for 2.8.1) #5436
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…5381) Only return success = true for a pipeline create operation if the pipeline has started and the corresponding kafka streams are running A concurrency issue remains because if running the code on multiple coroutines these may interleave at suspension points. This is not an issue atm, as we are executing on one coroutine. If/when we introduce more parallelism, this will need to be dealt with. **Fixed issues**: - #INFRA-830 (internal): pipeline might be marked ready before is actually running
…files (#5390) * add license generator script * add command to generate dependencies in txt * copy license files for scheduler * copy license files for hodometer * copy licesne for operator container * add a script to generate txt file for dataflow dependency licenses * generate licenses in txt * point to v2 branch for v2 license * map to v2 ref * add a mapping for v2 branch * always update go-license-tools * remove refs to seldon-core * add branch refs to the get github license info command * fix path for branch refs * gen licenses with v2 * styling fix * copy components/tls licenses in
…Es (#5402) In the dataflow component we use `io.kubernetes:client-java` for fetching k8s secrets needed for Kafka SaaS auth. The library has `org.apache.commons:commons-compress` as a dependency. This dependency, at the version used by the k8s client java lib, has two high CVEs that we would like to fix. The fix forces `org.apache.commons:commons-compress` update from `1.24.0` to `1.26.0` by defining an explicit gradle build dependency constraint See https://docs.gradle.org/current/userguide/dependency_constraints.html for a description of how gradle may handle transitive dependency updates. This comes with its own disadvantages: - because we force the update ourselves, we have to test that our dependency indeed works with the updated package and delivers the same functionality we need - we need to remove the build constraint once the dependency updates its dependency, so that we don't "pin" that to an old version unnecessarily (this is why we have the following TODO) TODO(future): remove gradle constraint when io.kubernetes:client-java gets updated to a version directly depending on 1.26.0 or higher **Fixes**: - [CVE-2024-26308](GHSA-4265-ccf5-phj5) (high) - [CVE-2024-25710](GHSA-4g9r-vxhx-9pgx) (high)
* bump envoy to 1.29.1 * run go mod tidy * Revert "run go mod tidy" This reverts commit 1b9a714.
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.61.1 to 1.62.0. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.61.1...v1.62.0) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* update manifests to use mlsever 1.5.0 * remove lightgbm for now from generation in script * update onnx link in script * update models to use newly generated ones
…untime (#5404) * add example for overriding podSpec in SeldonRuntime * rephrase
…ct (#5405) Previously, we only caught a StreamsException. However, the creation might fail for many reasons (for example, incorrect configuration). We want to catch any exception so that we mark the pipeline creation as failed and we don't stop the connection to the scheduler. Previously, on configuration errors, the exception would be bubbled to the PipelineSubscriber event loop, and the connection to the scheduler would be broken. We would try to reconnect, but on reconnect the scheduler would try to re-init the problematic pipeline (with the same id). This then led to an error about existing uncleaned KafkaStreams state in /tmp. This latter error was being handled cleanly (i.e not breaking the connection to the scheduler anymore), but would mask the real reason for the failure when looking at the pipeline status (via k8s or seldon cli).
#5406) * add missing unlock call * increase eventhub to 1000 pending msgs * refactor variable name
…5395) Bumps [go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc](https://github.com/open-telemetry/opentelemetry-go) from 1.22.0 to 1.24.0. - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go@v1.22.0...v1.24.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* allow connection to be passed to LoadModel and tidy up code * fix caller based on new signature of LoadModel * add docstring * wire up reloading models on reconnect * add logging * remove spurious break * mark some logging as debug
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.62.0 to 1.62.1. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.62.0...v1.62.1) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.62.0 to 1.62.1. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.62.0...v1.62.1) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps ubi9/ubi-micro from 9.3-13 to 9.3-15. --- updated-dependencies: - dependency-name: ubi9/ubi-micro dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps ubi9/ubi-micro from 9.3-13 to 9.3-15. --- updated-dependencies: - dependency-name: ubi9/ubi-micro dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps grafana/grafana from 10.3.3 to 10.4.0. --- updated-dependencies: - dependency-name: grafana/grafana dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps envoyproxy/envoy from v1.29.1 to v1.29.2. --- updated-dependencies: - dependency-name: envoyproxy/envoy dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps ubi9/ubi-minimal from 9.3-1552 to 9.3-1612. --- updated-dependencies: - dependency-name: ubi9/ubi-minimal dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.62.0 to 1.62.1. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.62.0...v1.62.1) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps rclone/rclone from 1.65.2 to 1.66.0. --- updated-dependencies: - dependency-name: rclone/rclone dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…5424) Bumps google.golang.org/protobuf from 1.32.0 to 1.33.0. --- updated-dependencies: - dependency-name: google.golang.org/protobuf dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
) Bumps [github.com/golang/protobuf](https://github.com/golang/protobuf) from 1.5.3 to 1.5.4. - [Release notes](https://github.com/golang/protobuf/releases) - [Commits](golang/protobuf@v1.5.3...v1.5.4) --- updated-dependencies: - dependency-name: github.com/golang/protobuf dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…5421) Bumps google.golang.org/protobuf from 1.32.0 to 1.33.0. --- updated-dependencies: - dependency-name: google.golang.org/protobuf dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…nfluentinc/confluent-kafka-go/v2/kafka/splunkkafka (#5420) Bumps [github.com/signalfx/splunk-otel-go/instrumentation/github.com/confluentinc/confluent-kafka-go/v2/kafka/splunkkafka](https://github.com/signalfx/splunk-otel-go) from 1.13.0 to 1.14.0. - [Release notes](https://github.com/signalfx/splunk-otel-go/releases) - [Changelog](https://github.com/signalfx/splunk-otel-go/blob/main/CHANGELOG.md) - [Commits](signalfx/splunk-otel-go@v1.13.0...v1.14.0) --- updated-dependencies: - dependency-name: github.com/signalfx/splunk-otel-go/instrumentation/github.com/confluentinc/confluent-kafka-go/v2/kafka/splunkkafka dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
) The number of partitions with which model-specific kafka topics is created needs to be the same as the number of partitions for other topics (pipelines/experiments/etc). This is a Kafka Streams constraint in order for joins to work correctly. Until now, an incorrect environment variable was being passed to modelgateway, which meant that it was always creating topics with 1 partition. This in turn led to dataflow processing getting stuck whenever other topics were created with multiple partitions **Fixed issues**: - INFRA-822: Pipelines get stuck on triggers/joins
When a new model is added, modelgateway creates quite a few connections to the kafka cluster, in quick succession (producer, consumer, admin). When using Confluent Cloud Kafka with OAUTHBEARER/OIDC token auth, the KafkaAdmin client failed to fetch the OIDC token sometimes. In tests we were using MS Entra ID, unsure if there is some rate-limiting on fetching tokens. Because we're not using the admin and producer connections concurrently (admin mostly used to create topics on model add), here we're letting the kafka admin client reuse the producer connection (and the same token). **Fixed issues**: - INFRA-867: Modelgateway sometimes fails to fetch OIDC tokens
Bumps ubi9/openjdk-17-runtime from 1.17 to 1.18. --- updated-dependencies: - dependency-name: ubi9/openjdk-17-runtime dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
- bump io.grpc 1.61.1 -> 1.62.2 - bump google.protobuf 3.25.2 -> 3.25.3 - bump io.kubernetes:client-java 19.0.0 -> 20.0.0 minor API deprecation fixes were required for the k8s client-java version bump
…er -> controller) (#5434) * add missing send experiment statues on reconnect * update api go mod * update go for tls component * update go.sum with missing deps for tls * go mod tidy for scheduler / operator * add timeout for status send * add unit test for status send * add stress tester script * refactor tests into separate files
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes for release 2.8.1 (rc1)