FLPATH-1066: swf-builder: generate manifests using kn-workflow #1751

rgolangh · 2024-02-28T20:24:41Z

Generate manifests and save them in the runtime image.

Generating manifests is an opt-in as part of the build ran by /launch/build-app.sh
When setting GEN_MANIFESTS=true during container build the build-app.sh
script will call a new generate-manifests.sh script that will use
kn-workflow cli and will store the manifests under
$KOGITO_HOME/build/manifests

Later in the runtime image user can copy and store those manifests for
later use:

COPY --from=builder --chown=185 /home/kogito/build/manifests /deployments/manifests

The manifests can be extracted for example by echoing them from the image:

podman run builder:latest cat /deployment/manifests/* > manifests-all.yaml

or by podman cp:

podman cp $(podman create builder:latest):/deployment/manifests .

Fixes: #1739
Related-to: https://issues.redhat.com/browse/FLPATH-1066

Signed-off-by: Roy Golan rgolan@redhat.com

Many thanks for submitting your Pull Request ❤️!

Please make sure your PR meets the following requirements:

You have read the contributors guide
Pull Request title is properly formatted: [KOGITO|RHPAM-XYZ] Subject
Pull Request contains link to the JIRA issue
Pull Request contains description of the issue
Pull Request does not include fixes for issues other than the main ticket
Your feature/bug fix has a testcase that verifies it
You've tested the new feature/bug fix in an actual OpenShift cluster
You've added a RELEASE_NOTES.md entry regarding this change

How to retest this PR or trigger a specific build:

(Re)run Jenkins tests
Please add comment: Jenkins [test|retest] this

How to backport a pull request to a different branch?

In order to automatically create a backporting pull request please add one or more labels having the following format backport-<branch-name>, where <branch-name> is the name of the branch where the pull request must be backported to (e.g., backport-7.67.x to backport the original PR to the 7.67.x branch).

NOTE: backporting is an action aiming to move a change (usually a commit) from a branch (usually the main one) to another one, which is generally referring to a still maintained release branch. Keeping it simple: it is about to move a specific change or a set of them from one branch to another.

Once the original pull request is successfully merged, the automated action will create one backporting pull request per each label (with the previous format) that has been added.

If something goes wrong, the author will be notified and at this point a manual backporting is needed.

NOTE: this automated backporting is triggered whenever a pull request on main branch is labeled or closed, but both conditions must be satisfied to get the new PR created.

ricardozanini

Many thanks! A few comments if you don't mind. Also, can you please create one or more bats tests to validate this new module and the builder calling the new script?

modules/kogito-workflow-manifests/added/generate-manifests.sh

ricardozanini · 2024-02-29T15:05:18Z

modules/kogito-workflow-manifests/configure

+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+INSTALL_DIR=/usr/local/bin
+curl -L https://github.com/rgolangh/kie-tools/releases/download/0.0.2/kn-workflow-linux-amd64 -o "${INSTALL_DIR}"/kn-workflow


Can we use the official repo and uname to identify the OS and arch in this URL?

till now I'm using this private build because the latest fixes with ''--namespace" flag are still not published. @ederign is this still true?

@rgolangh not yet, but soon we will have kn workflow cli 1.32 published together with serverless release.

modules/kogito-workflow-manifests/install.sh

modules/kogito-workflow-manifests/module.yaml

ricardozanini · 2024-02-29T19:15:12Z

modules/kogito-workflow-manifests/tests/bats/kogito-workflow-manifest.bats

+    [ ! -f "${KOGITO_HOME}"/bin/kn-workflow_invocation.txt ] 
+}
+
+mock_bin() {


pretty cool!

Generate manifests and save them in the runtime image. Generating manifests is an opt-in as part of the build ran by /launch/build-app.sh When setting GEN_MANIFESTS=true during container build the build-app.sh script will call a new generate-manifests.sh script that will use _kn-workflow_ cli and will store the manifests under $KOGITO_HOME/build/manifests Later in the runtime image user can copy and store those manifests for later use: COPY --from=builder --chown=185 /home/kogito/build/manifests /deployments/manifests The manifests can be extracted for example by echoing them from the image: podman run builder:latest cat /deployment/manifests/* > manifests-all.yaml or by podman cp: podman cp $(podman create builder:latest):/deployment/manifests . Signed-off-by: Roy Golan <rgolan@redhat.com>

kie-ci3 · 2024-03-03T11:00:25Z

PR job #121 was: FAILURE
Possible explanation: Pipeline failure or project build failure

Please look here: https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/pullrequest/job/kogito-images.build-and-test/121/display/redirect
See console log:

Console Logs

[Pipeline] }
Build KIE » kogito » main » pullrequest » kogito-images.build-image PR #1751 - kogito-data-index-ephemeral: https://github.com//pull/1751 completed: SUCCESS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Build KIE » kogito » main » pullrequest » kogito-images.build-image PR #1751 - kogito-jobs-service-ephemeral: https://github.com//pull/1751 completed: SUCCESS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Build KIE » kogito » main » pullrequest » kogito-images.build-image PR #1751 - kogito-runtime-jvm: https://github.com//pull/1751 completed: SUCCESS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Build KIE » kogito » main » pullrequest » kogito-images.build-image PR #1751 - kogito-management-console: https://github.com//pull/1751 completed: SUCCESS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Build KIE » kogito » main » pullrequest » kogito-images.build-image PR #1751 - kogito-task-console: https://github.com//pull/1751 completed: SUCCESS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Build KIE » kogito » main » pullrequest » kogito-images.build-image PR #1751 - kogito-swf-builder: https://github.com//pull/1751 completed: SUCCESS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Build KIE » kogito » main » pullrequest » kogito-images.build-image PR #1751 - kogito-s2i-builder: https://github.com//pull/1751 completed: SUCCESS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Build KIE » kogito » main » pullrequest » kogito-images.build-image PR #1751 - kogito-jobs-service-allinone: https://github.com//pull/1751 completed: FAILURE
[Pipeline] error
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch kogito-jobs-service-allinone
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] script
[Pipeline] {
[Pipeline] sh
+ wget --no-check-certificate -qO - 'https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/pullrequest/job/kogito-images.build-and-test/121/api/json?depth=0'
[Pipeline] readJSON
[Pipeline] sh

domhanak · 2024-03-04T08:05:00Z

@tomasdavidorg fyi: concerns kn cli

ricardozanini · 2024-03-04T18:33:01Z

Errors in the CI are not related. These are tied to the recent renaming work we are doing on runtimes libraries.

ricardozanini · 2024-03-04T18:33:15Z

@jakubschwan @tomasdavidorg can you guys verify?

tomasdavidorg

@ricardozanini Is this still relevant with the latest planned changes? I mean the Unblocking and releasing Apache KIE 10. The kie-tools will be the last in the queue and it does not seems to be ok to include them here. But maybe I am missing something.

tomasdavidorg · 2024-03-22T06:54:29Z

modules/kogito-workflow-manifests/module.yaml

+
+artifacts:
+  - name: kn-workflow-x86_64
+    url: https://github.com/rgolangh/kie-tools/releases/download/0.0.2/kn-workflow-linux-amd64


The link does not seems to be correct as it points to the fork.

We will review this once we move kn-workflow to the operator.

is there a release I can use already?

No, there's not.

ricardozanini · 2024-03-22T14:20:20Z

@tomasdavidorg We will continue with this since we are moving to kie-tools just temporarily. After the release, we will fix the release stream.

Additionally, the workflow CLI will be in the operator repo.

kie-ci3 · 2024-05-06T20:48:09Z

PR job #1 was: FAILURE
Possible explanation: Pipeline failure or project build failure

Please look here: https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/pullrequest/job/kogito-images.build-and-test/1/display/redirect
See console log:

Console Logs

From https://github.com/rgolangh/incubator-kie-kogito-images
* branch add-kn-workflow-to-builder -> FETCH_HEAD
hint: You have divergent branches and need to specify how to reconcile them.
hint: You can do so by running one of the following commands sometime before
hint: your next pull:
hint:
hint: git config pull.rebase false # merge (the default strategy)
hint: git config pull.rebase true # rebase
hint: git config pull.ff only # fast-forward only
hint:
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
fatal: Need to specify how to reconcile divergent branches.
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] echo

-------------------------------------------------------------
[ERROR] Can't merge source into Target. Please rebase PR branch.
-------------------------------------------------------------
Source: git://github.com/rgolangh/incubator-kie-kogito-images add-kn-workflow-to-builder
Target: a595d8c Upgrade to and align with Quarkus 3.8.4 LTS release. (#1760)
-------------------------------------------------------------

[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Validate CeKit Image and Modules descriptors)
Stage "Validate CeKit Image and Modules descriptors" skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Build & Test Images)
Stage "Build & Test Images" skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] script
[Pipeline] {
[Pipeline] sh
+ wget --no-check-certificate -qO - 'https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/pullrequest/job/kogito-images.build-and-test/1/api/json?depth=0'
[Pipeline] readJSON
[Pipeline] sh

kie-ci3 · 2024-06-04T15:56:15Z

PR job #3 was: FAILURE
Possible explanation: Pipeline failure or project build failure

Please look here: https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/pullrequest/job/kogito-images.build-and-test/3/display/redirect
See console log:

Console Logs

From https://github.com/rgolangh/incubator-kie-kogito-images
* branch add-kn-workflow-to-builder -> FETCH_HEAD
hint: You have divergent branches and need to specify how to reconcile them.
hint: You can do so by running one of the following commands sometime before
hint: your next pull:
hint:
hint: git config pull.rebase false # merge (the default strategy)
hint: git config pull.rebase true # rebase
hint: git config pull.ff only # fast-forward only
hint:
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
fatal: Need to specify how to reconcile divergent branches.
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] echo

-------------------------------------------------------------
[ERROR] Can't merge source into Target. Please rebase PR branch.
-------------------------------------------------------------
Source: git://github.com/rgolangh/incubator-kie-kogito-images add-kn-workflow-to-builder
Target: 431715c Add @jstastny-cz and @rodrigonull to CODEOWNERS (#1771)
-------------------------------------------------------------

[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Validate CeKit Image and Modules descriptors)
Stage "Validate CeKit Image and Modules descriptors" skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Build & Test Images)
Stage "Build & Test Images" skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] script
[Pipeline] {
[Pipeline] sh
+ wget --no-check-certificate -qO - 'https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/pullrequest/job/kogito-images.build-and-test/3/api/json?depth=0'
[Pipeline] readJSON
[Pipeline] sh

rgolangh force-pushed the add-kn-workflow-to-builder branch from 87b04b1 to 23a12c8 Compare February 29, 2024 14:42

rgolangh marked this pull request as ready for review February 29, 2024 14:42

rgolangh changed the title ~~WIP: swf-builder generate manifests using kn-workflow~~ FLPATH-1066: swf-builder: generate manifests using kn-workflow Feb 29, 2024

ricardozanini requested review from ricardozanini, wmedvede and jakubschwan February 29, 2024 15:02

ricardozanini reviewed Feb 29, 2024

View reviewed changes

rgolangh force-pushed the add-kn-workflow-to-builder branch 3 times, most recently from e963f7f to eda46ff Compare February 29, 2024 15:26

apache deleted a comment from kie-ci3 Feb 29, 2024

rgolangh force-pushed the add-kn-workflow-to-builder branch from eda46ff to 094e03f Compare February 29, 2024 19:03

ricardozanini reviewed Feb 29, 2024

View reviewed changes

apache deleted a comment from kie-ci3 Feb 29, 2024

rgolangh force-pushed the add-kn-workflow-to-builder branch 3 times, most recently from e419825 to 86fedf8 Compare March 3, 2024 08:21

rgolangh force-pushed the add-kn-workflow-to-builder branch from 86fedf8 to a649be2 Compare March 3, 2024 08:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FLPATH-1066: swf-builder: generate manifests using kn-workflow #1751

FLPATH-1066: swf-builder: generate manifests using kn-workflow #1751

rgolangh commented Feb 28, 2024 •

edited

Loading

ricardozanini left a comment

ricardozanini Feb 29, 2024

rgolangh Feb 29, 2024

ederign Mar 5, 2024

ricardozanini Feb 29, 2024

kie-ci3 commented Mar 3, 2024

domhanak commented Mar 4, 2024

ricardozanini commented Mar 4, 2024

ricardozanini commented Mar 4, 2024

tomasdavidorg left a comment

tomasdavidorg Mar 22, 2024

ricardozanini Mar 22, 2024

rgolangh Jun 24, 2024

ricardozanini Jun 24, 2024

ricardozanini commented Mar 22, 2024

kie-ci3 commented May 6, 2024

kie-ci3 commented Jun 4, 2024

FLPATH-1066: swf-builder: generate manifests using kn-workflow #1751

Are you sure you want to change the base?

FLPATH-1066: swf-builder: generate manifests using kn-workflow #1751

Conversation

rgolangh commented Feb 28, 2024 • edited Loading

ricardozanini left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kie-ci3 commented Mar 3, 2024

domhanak commented Mar 4, 2024

ricardozanini commented Mar 4, 2024

ricardozanini commented Mar 4, 2024

tomasdavidorg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ricardozanini commented Mar 22, 2024

kie-ci3 commented May 6, 2024

kie-ci3 commented Jun 4, 2024

rgolangh commented Feb 28, 2024 •

edited

Loading