ci: Merge changes from v2 to release 2.8 branch #5705

sakoush · 2024-06-21T14:55:12Z

No description provided.

Bumps grafana/grafana from 10.4.0 to 10.4.1. --- updated-dependencies: - dependency-name: grafana/grafana dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [github.com/onsi/gomega](https://github.com/onsi/gomega) from 1.31.1 to 1.32.0. - [Release notes](https://github.com/onsi/gomega/releases) - [Changelog](https://github.com/onsi/gomega/blob/master/CHANGELOG.md) - [Commits](onsi/gomega@v1.31.1...v1.32.0) --- updated-dependencies: - dependency-name: github.com/onsi/gomega dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* add test for rolling update for pipelines * add case for no new version * add test case for pipeline create / delete * fix lint * be specific about the error status message

…5483) The version bump required small changes in the way we use the API.

…5490) * update copyright * cleanup

…5488) Bumps [github.com/cenkalti/backoff/v4](https://github.com/cenkalti/backoff) from 4.2.1 to 4.3.0. - [Commits](cenkalti/backoff@v4.2.1...v4.3.0) --- updated-dependencies: - dependency-name: github.com/cenkalti/backoff/v4 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…#5487) Bumps [github.com/cenkalti/backoff/v4](https://github.com/cenkalti/backoff) from 4.2.1 to 4.3.0. - [Commits](cenkalti/backoff@v4.2.1...v4.3.0) --- updated-dependencies: - dependency-name: github.com/cenkalti/backoff/v4 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…lint issues (#5491) * configure ktlint * add make commands * fix path to exclude * configure ktlint * run ktlintFormat * run ktlintFormat * add readme for lint check and format * ignore vscode * clean up * add --continue flag in make command * modify lint rules * unpack import statements to follow no-wildcard-imports * fix lint issues in test * fix lint issues around property names * fix inline comment lint issues * fix max chars lint issue * add linting for jvm * add dataflow lint command in workflows * use data-flow make command for lint target in scheduler

* add pipeline version to kafka header * add doc for pipeline version

* fix note * add ability to check loading state * change func name to be more reflective

* make tfsimple_pipeline test create random models underneath * update alpine base image and install xk6 * make namespace configurable in shell script * add xk6 to .gitignore * add xk6 install as make target * adjust wrapper to use k6 with xk6 * add note in readme regarding xk6 * add k8s test script scenario * adjust namespace variable * ability to set namespace in kustomize * update model script * add newline in file * add NAMESPACE as envar * wire up namespace * parse pod namespace from envar * use NAMESPACE env var * add service account and remove unnecessary secret * have main scenario back as default * review comments

Bumps grafana/grafana from 10.4.1 to 10.4.2. --- updated-dependencies: - dependency-name: grafana/grafana dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.62.1 to 1.63.2. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.62.1...v1.63.2) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps envoyproxy/envoy from v1.29.2 to v1.29.3. --- updated-dependencies: - dependency-name: envoyproxy/envoy dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.62.1 to 1.63.2. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.62.1...v1.63.2) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…taflow (#5522) Update confirmed working in kind, with simple pipeline inferencing test

Introduce the ability to declaratively specify, via an ansible-playbook extra vars file, sets of components that should be built locally from the workdir and pushed into kind. The kind Seldon Core v2 install is then configured to use those dev images. Full documentation included in `README.dev.md`

* deal with multi model env vars * define const before usage * add memory size handling

Co-authored-by: sakoush <5911512+sakoush@users.noreply.github.com>

…o previous contributions (#5555)

* upgrade to golang 1.21 * go mod tidy * upgrade linter to v1.57.2 * upgrade lint worfklow to v1.57.2 * increase timeout for lint workflow

Bumps envoyproxy/envoy from v1.29.3 to v1.30.1. --- updated-dependencies: - dependency-name: envoyproxy/envoy dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [github.com/onsi/gomega](https://github.com/onsi/gomega) from 1.32.0 to 1.33.1. - [Release notes](https://github.com/onsi/gomega/releases) - [Changelog](https://github.com/onsi/gomega/blob/master/CHANGELOG.md) - [Commits](onsi/gomega@v1.32.0...v1.33.1) --- updated-dependencies: - dependency-name: github.com/onsi/gomega dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…#5554) Bumps [go.opentelemetry.io/otel/sdk](https://github.com/open-telemetry/opentelemetry-go) from 1.24.0 to 1.26.0. - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go@v1.24.0...v1.26.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/otel/sdk dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…nes (#5550) This fixes a dataflow-engine bug triggered when the scheduler sends or re-sends pipeline creation messages after a restart (because it's not aware of pipeline statuses across various components). Previous behaviour: Dataflow-engine, on receiving a command to create a pipeline, would first create a new Kafka Streams application for this pipeline, before checking if one already exists and it's running. Because of this, triggering a control-plane restart of the scheduler would result in dataflow errors for pipelines that kept internal state (mostly pipelines making use of triggers/joins). Kafka Streams would complain about an existing application using the same state directory, fail the newly created pipeline and inform the scheduler about this. However, in actuality the old pipeline, if it was previously running ok, would continue doing so inside dataflow. This meant that a disconnect between the state of dataflow-engine and what the scheduler knew about it was being created New behaviour: The introduced changes mean that dataflow-engine first checks if a pipeline with the same id is already running. If its state is ok, dataflow simply informs the scheduler that the pipeline is created, without taking further action. If a pipeline with the same id already exists but is in a failed or stopped state, the local Kafka Streams state is first cleaned, then an attempt is made to re-create it, with the corresponding status being sent to the scheduler. - adds `PipelineState` tests - test `StreamStopped(prevState)` with `prevState=StreamStopped` avoids nesting - update test names for clarity - update build config to print test PASS/SKIP/FAIL results - add `test` target to Makefile **Which issue(s) this PR fixes:** - INFRA-978 (internal issue) Pipelines fail to start because Kafka Streams state hasn't been cleared

* upgrade controller-runtime in scheduler * add error handling for AddEventHandler * upgrade controller-runtime in tls * upgrade controller-runtime in operator * upgrade controller-runtime in apis * replace deprecated pointer pkg * go mod tidy hodometer * add a comment to consider passing a context in the future

Bumps ubi9/ubi-minimal from 9.3-1612 to 9.4-949. --- updated-dependencies: - dependency-name: ubi9/ubi-minimal dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…5628) Bumps google.golang.org/protobuf from 1.34.0 to 1.34.1. --- updated-dependencies: - dependency-name: google.golang.org/protobuf dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ux/otelmux (#5627) Bumps [go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux](https://github.com/open-telemetry/opentelemetry-go-contrib) from 0.49.0 to 0.51.0. - [Release notes](https://github.com/open-telemetry/opentelemetry-go-contrib/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go-contrib/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go-contrib@zpages/v0.49.0...zpages/v0.51.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Adds a k6 test that arbitrarily creates/updates/deletes models in each VU. Starting from the scenario defined in `scenarios/core2_qa_control_plane_ops.js`, we can run a variety of control plane tests to check that Core 2 responds as expected to a wide array of commands/situations, including concurrent operations. New configuration options introduced for k6 control-plane scenarios: - `modelMaxReplicas` (set via environment var MODEL_MAX_REPLICAS, array) each model type may be updated during the test to a maximum of replicas defined here - `maxMemUpdateFraction` (set via environment var MAX_MEM_UPDATE_FRACTION, float) each model type may be updated by having, in the worst case, this much memory relative to its default setting either added or removed. Example: For a model being assigned 8M in `MODEL_MEMORY_BYTES`, a value of 0.5 in `maxMemUpdateFraction` means that the model may be created, at random with a memory requirement between 4M and 12M. - `maxNumModelsHeadroom` (set via environment var MAX_NUM_MODELS_HEADROOM, array) models beyond `maxNumModels` (which are all created during the setup phase) which are allowed during control-plane testing - `createUpdateDeleteBias` (set via environment var MODEL_CREATE_UPDATE_DELETE_BIAS, array) 3 element array defining the relative frequency with which CREATE/UPDATE/DELETE operations happen during control-plane tests - `maxCreateOpsPerVU` (set via environment var MAX_CREATE_OPS_PER_VU, int) because each VU runs independently, we set aside a namespace of this size for model IDs created by that VU. For example, if maxCreateOpsPerVU is 100, preloaded models get ids [0, 99], models created by VU 1 get ids [100, 199], by VU 2 get ids [200, 299], etc. - `k8sDelaySecPerVU` (set via environment var K8S_DELAY_SECONDS_PER_VU, int) the following delay is observed by each VU before starting a new iteration Minor k6 tests fixes: - allow 0 as an element of MAX_NUM_MODELS - check model/pipeline status via k8s rather than relying on connection to scheduler - fix kustomize warnings **Which issue(s) this PR fixes**: - INFRA-1002 (internal issue) Control-plane tests

* add note about pipeline name * fix scenario * add pipelines to control plane ops * add pipelines to control operations tests * add note in code about pipeline assumptions * deal with errors

Fix a bug where defining multiple custom secrets in an extra-vars file without explicit namespaces would add the default namespace only to the last one, and fail the playbook.

- bump kotlin language version 1.8.20 -> 1.9.21 - bump kotlinx-coroutines-core 1.7.3 -> 1.8.1 - bump io.grpc:grpc-* 1.63.0 -> 1.64.0 The only remaining update that we would like to make to dataflow is bumping JDK 17 -> JDK 21. However, that update is currently blocked by the requirements of kafka-streams; The confluent platform libraries that we're using (kafka-streams:7.6.1-ccs) have only been tested up to JDK 17 according to: https://docs.confluent.io/platform/current/installation/versions-interoperability.html#java

Bumps ubi9/ubi-micro from 9.4-6 to 9.4-6.1716471860. --- updated-dependencies: - dependency-name: ubi9/ubi-micro dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps ubi9/ubi-minimal from 9.4-949.1714662671 to 9.4-949.1716471857. --- updated-dependencies: - dependency-name: ubi9/ubi-minimal dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [github.com/hashicorp/go-retryablehttp](https://github.com/hashicorp/go-retryablehttp) from 0.7.6 to 0.7.7. - [Changelog](https://github.com/hashicorp/go-retryablehttp/blob/main/CHANGELOG.md) - [Commits](hashicorp/go-retryablehttp@v0.7.6...v0.7.7) --- updated-dependencies: - dependency-name: github.com/hashicorp/go-retryablehttp dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps ubi9/ubi-micro from 9.4-6 to 9.4-6.1716471860. --- updated-dependencies: - dependency-name: ubi9/ubi-micro dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps grafana/grafana from 10.4.3 to 11.0.0. --- updated-dependencies: - dependency-name: grafana/grafana dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…er (#5640) Bumps [go.opentelemetry.io/otel/trace](https://github.com/open-telemetry/opentelemetry-go) from 1.26.0 to 1.27.0. - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go@v1.26.0...v1.27.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/otel/trace dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps ubi9/ubi-micro from 9.4-6 to 9.4-6.1716471860. --- updated-dependencies: - dependency-name: ubi9/ubi-micro dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ux/otelmux (#5679)

…#5677)

Bumps rclone/rclone from 1.66.0 to 1.67.0. --- updated-dependencies: - dependency-name: rclone/rclone dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…er (#5696) Bumps ubi9/ubi-minimal from 9.4-949.1717074713 to 9.4-1134. --- updated-dependencies: - dependency-name: ubi9/ubi-minimal dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps ubi9/ubi-micro from 9.4-6.1716471860 to 9.4-9. --- updated-dependencies: - dependency-name: ubi9/ubi-micro dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…5691) Bumps google.golang.org/protobuf from 1.34.1 to 1.34.2. --- updated-dependencies: - dependency-name: google.golang.org/protobuf dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [github.com/go-logr/logr](https://github.com/go-logr/logr) from 1.4.1 to 1.4.2. - [Release notes](https://github.com/go-logr/logr/releases) - [Changelog](https://github.com/go-logr/logr/blob/master/CHANGELOG.md) - [Commits](go-logr/logr@v1.4.1...v1.4.2) --- updated-dependencies: - dependency-name: github.com/go-logr/logr dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps ubi9/ubi-micro from 9.4-6.1716471860 to 9.4-9. --- updated-dependencies: - dependency-name: ubi9/ubi-micro dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* deal with `PipelineTerminating` when rebalancing and dataflow exists * add tests * increasing debug level in test

CLAassistant · 2024-06-21T14:55:20Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 3 committers have signed the CLA.

✅ ramonpzg
✅ paulb-seldon
❌ KengoA
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

lc525

LGTM

sakoush · 2024-06-21T15:00:42Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.2 out of 3 committers have signed the CLA.✅ ramonpzg✅ paulb-seldon❌ KengoAYou have signed the CLA already but the status is still pending? Let us recheck it.

All changes done as Seldon employees.

dependabot bot and others added 30 commits March 26, 2024 12:12

fix(scheduler): Add unit tests for chainer service (#5480)

55aea43

* add test for rolling update for pipelines * add case for no new version * add test case for pipeline create / delete * fix lint * be specific about the error status message

fix(deps): bump kotlin-retry 1.0.9 to 2.0.0 in /scheduler/data-flow (#…

7ec19c9

…5483) The version bump required small changes in the way we use the API.

build(dataflow): Add missing copyright headers for dataflow engine (#…

42f270c

…5490) * update copyright * cleanup

feat(dataflow): Add pipeline version to Kafka headers (#5493)

3ce7029

* add pipeline version to kafka header * add doc for pipeline version

fix: Handle unload too quick after load (#5504)

0b9ee7d

* fix note * add ability to check loading state * change func name to be more reflective

build(deps): bump io.grpc:grpc* from 1.62.2 to 1.63.0 in scheduler/da…

b02a400

…taflow (#5522) Update confirmed working in kind, with simple pipeline inferencing test

fix: Deal with multi model env vars (#5527)

a7aeca1

* deal with multi model env vars * define const before usage * add memory size handling

Re-generate license info (#5528)

a23fbc8

Co-authored-by: sakoush <5911512+sakoush@users.noreply.github.com>

Update CONTRIBUTING.md to add notice on licensing changes and links t…

47b141e

…o previous contributions (#5555)

fix: Upgrade go 1.21 and golint to 1.57 (#5556)

8067816

* upgrade to golang 1.21 * go mod tidy * upgrade linter to v1.57.2 * upgrade lint worfklow to v1.57.2 * increase timeout for lint workflow

dependabot bot and others added 25 commits May 20, 2024 17:04

feat: Add pipelines to core2 qa control plane tests (#5636)

b69ed59

* add note about pipeline name * fix scenario * add pipelines to control plane ops * add pipelines to control operations tests * add note in code about pipeline assumptions * deal with errors

fix(ansible): allow multiple custom secrets without namespaces (#5638)

bd0b2ed

Fix a bug where defining multiple custom secrets in an extra-vars file without explicit namespaces would add the default namespace only to the last one, and fail the playbook.

Bump go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/m…

a284d5c

…ux/otelmux (#5679)

Bump go.opentelemetry.io/otel/sdk from 1.26.0 to 1.27.0 in /scheduler (…

b5ae20f

…#5677)

Bump envoyproxy/envoy from v1.30.1 to v1.30.2 in /scheduler (#5668)

9d1e287

Bump ubi9/ubi-minimal in /scheduler (#5667)

c1c85de

fix(scheduler): Pipeline terminating fix on rebalance (#5703)

7389d07

* deal with `PipelineTerminating` when rebalancing and dataflow exists * add tests * increasing debug level in test

sakoush requested a review from lc525 as a code owner June 21, 2024 14:55

lc525 approved these changes Jun 21, 2024

View reviewed changes

sakoush merged commit 0a14aa0 into release-2.8 Jun 21, 2024
15 of 17 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: Merge changes from v2 to release 2.8 branch #5705

ci: Merge changes from v2 to release 2.8 branch #5705

sakoush commented Jun 21, 2024

CLAassistant commented Jun 21, 2024 •

edited

Loading

lc525 left a comment

sakoush commented Jun 21, 2024

ci: Merge changes from v2 to release 2.8 branch #5705

ci: Merge changes from v2 to release 2.8 branch #5705

Conversation

sakoush commented Jun 21, 2024

CLAassistant commented Jun 21, 2024 • edited Loading

lc525 left a comment

Choose a reason for hiding this comment

sakoush commented Jun 21, 2024

CLAassistant commented Jun 21, 2024 •

edited

Loading