vdk-meta-jobs: exec job with arguments #1839
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
for more information, see https://pre-commit.ci
…1799) what: Updated the job builder images to work with temporary aws credentials. The new builders will work iff the session_token param is passed, behavior should be unchanged otherwise. Note that empty lines were removed by the git hook! why: this is a necessary change for this PR - #1787 which introduces abilities to generate temporary credentials. testing - local testing ci/cd Signed-off-by: mrMoZ1 <mzhivkov@vmware.com>
…ojects/control-service/projects (#1798) Bumps com.palantir.git-version from 2.0.0 to 3.0.0. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Apply needed changes for class names, imports, dirs structure, styling, and features parameterization for easier adoption in VMware projects and open-source community. Paved a road for editor configuration uniformity of formatting, where another Pull request will follow. Why: Make adoption in VMware projects is fully functional and fulfill all needs. Tests: unit tests 100% pass rate, manual integration with VMware projects with 100% pass rate in other projects unit tests and also 100% pass rate of e2e automation tests in other projects. --------- Signed-off-by: Goran Kokinovski <gkokinovski@vmware.com> Co-authored-by: Goran Kokinovski <gkokinovski@vmware.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: ivakoleva <ikoleva@vmware.com> Co-authored-by: github-actions <> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
) As part of my effort to stabilize our Nightly builds and the CICD cluster I am removing testing middle versions. We test with the oldest supported version to make sure we do not use a new feature added lately which would break compatability. We test with newest version to make sure that it runs with newest version. Although the python itself is very likely to be backwords compatible (for minor release), there are sometimes small changes that are not and same cannot be told for 3th party dependecies we may use which may not support latest version fully. Testing Done: this CI Signed-off-by: Antoni Ivanov <aivanov@vmware.com>
# Why Builds were failing because of the docker image we were using. # What I upgraded to the latest one. This is one we are using in more places in the the gitlab pipeline. # How has this been tested. In a previous commit I enabled those jobs on this PR and made sure they ran successfully. --------- Signed-off-by: murphp15 <murphp15@tcd.ie> Co-authored-by: Momchil Z <mzhivkov@vmware.com>
…o 2.2.9 in /projects/control-service/projects/pipelines_control_service (#1797) Bumps io.swagger.core.v3:swagger-annotations from 2.2.8 to 2.2.9. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Paul Murphy <murphp15@tcd.ie>
….9 in /projects/control-service/projects/pipelines_control_service (#1796) Bumps io.swagger.core.v3:swagger-models from 2.2.8 to 2.2.9. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Paul Murphy <murphp15@tcd.ie>
As outlined here - https://github.com/vmware/versatile-data-kit/blob/main/cicd/build.sh#L40 the top level projects are supposed to have build.sh that fully builds the project or provides intructions Adding build.sh for frontend Signed-off-by: Antoni Ivanov <aivanov@vmware.com>
…1800) what: Updated the helm chart to include variables for the service account pattern introduced: #1787 why: this is a necessary change for this PR - #1787 which introduces abilities to generate temporary credentials. Signed-off-by: mrMoZ1 <mzhivkov@vmware.com> Co-authored-by: Antoni Ivanov <aivanov@vmware.com>
what: added the ability to use an IAM user with Role permissions when the data job image registry type is ECR. Configuration through control-service properties. Previous functionality should remain unchanged if properties are unset. why: User needs of the control-service require the Service Pattern to be introduced. testing: Integration test that deploys a data job via the new method. --------- Signed-off-by: mrMoZ1 <mzhivkov@vmware.com> Co-authored-by: github-actions <> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Bumps [org.mockito:mockito-core](https://github.com/mockito/mockito) from 5.1.1 to 5.2.0. - [Release notes](https://github.com/mockito/mockito/releases) - [Commits](mockito/mockito@v5.1.1...v5.2.0) --- updated-dependencies: - dependency-name: org.mockito:mockito-core dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
What: Added server handler for the UI Deploy job component. Why: The functionality should be provided since the component does nothing currently. It is also linked to the issue #1320 Tests: manually tested Signed-off-by: Duygu Hasan [hduygu@vmware.com](mailto:hduygu@vmware.com)
There is a circular import issue between the vdk.internal.control.configuration.vdk_config and vdk.internal.control.command_groups.version_group.version modules. Specifically, vdk_config is importing version and version is importing cli_utils, which is importing vdk_config. This creates a cycle that cannot be resolved, which is causing the ImportError under certain circumstances. To fix this issue, I need to refactor the code to remove the circular import. I moved the methods for determining verison in version utils which is used by both vdk_config and version_group.version modules. --------- Signed-off-by: Antoni Ivanov <aivanov@vmware.com>
what: bumped versions.txt for job builders. why: previously pipeline was broken. This should force builders to be deployed. testing; ci/cd Signed-off-by: mrMoZ1 <mzhivkov@vmware.com>
Changing the helm configuraiton did not seem to fix the issue (PR 1813) so instead I will try to apply limit ranges for the whole namespace Testing Done: apply them manually and verified frontend builds passing Signed-off-by: Antoni Ivanov <aivanov@vmware.com>
latest version of sqllineage is removing `statements_parsed` method We are adopting our vdk-lineage plugin to support the latest release. This means we won't detect query type but that property was never used anywhere by vdk-lineage. Caught by nightly build tests https://gitlab.com/vmware-analytics/versatile-data-kit/-/jobs/4043903802 Testing Done: existing unit tests Signed-off-by: Antoni Ivanov <aivanov@vmware.com>
assert_click_status is both shorter and provides more details in case of failures . Hence using it to check status is better Testing Done: CI Signed-off-by: Antoni Ivanov <aivanov@vmware.com>
…cts/control-service/projects (#1833) Bumps com.palantir.docker from 0.34.0 to 0.35.0. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
<!--pre-commit.ci start--> updates: - [github.com/psf/black: 23.1.0 → 23.3.0](psf/black@23.1.0...23.3.0) - [github.com/Lucas-C/pre-commit-hooks: v1.4.2 → v1.5.1](Lucas-C/pre-commit-hooks@v1.4.2...v1.5.1) <!--pre-commit.ci end--> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Antoni Ivanov <aivanov@vmware.com>
…o 2.0.1 in /projects/control-service/projects/pipelines_control_service (#1836) Bumps [org.springframework.retry:spring-retry](https://github.com/spring-projects/spring-retry) from 2.0.0 to 2.0.1. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/spring-projects/spring-retry/releases">org.springframework.retry:spring-retry's releases</a>.</em></p> <blockquote> <h2>v2.0.1</h2> <h2>:star: New Features</h2> <ul> <li>Change default order for <code>@EnableRetry</code> <a href="https://redirect.github.com/spring-projects/spring-retry/pull/335">#335</a></li> <li>add notice and license to jar <a href="https://redirect.github.com/spring-projects/spring-retry/pull/330">#330</a></li> <li>Add default implementations to <code>RetryListener</code> <a href="https://redirect.github.com/spring-projects/spring-retry/pull/326">#326</a></li> <li>support custom RetryConfiguration.getOrder() via <code>@EnableRetry</code> like <code>@EnableAsync</code> <a href="https://redirect.github.com/spring-projects/spring-retry/issues/22">#22</a></li> </ul> <h2>:lady_beetle: Bug Fixes</h2> <ul> <li>Backoff annotation 'delayExpression' attribute is not always applied <a href="https://redirect.github.com/spring-projects/spring-retry/issues/332">#332</a></li> <li>[code error] in org.springframework.retry.annotation.RecoverAnnotationRecoveryHandler#isParameterizedTypeAssignable <a href="https://redirect.github.com/spring-projects/spring-retry/issues/328">#328</a></li> <li>AnnotationTypeMapping warning when using <code>@CIRCUITBREAKER</code> with Spring Framework 6 <a href="https://redirect.github.com/spring-projects/spring-retry/issues/324">#324</a></li> <li>AnnotationTypeMapping warning when using <code>@EnableRetry</code> with Spring Framework 6 <a href="https://redirect.github.com/spring-projects/spring-retry/pull/322">#322</a></li> </ul> <h2>:notebook_with_decorative_cover: Documentation</h2> <ul> <li>Added getting support links <a href="https://redirect.github.com/spring-projects/spring-retry/pull/334">#334</a></li> </ul> <h2>:hammer: Dependency Upgrades</h2> <ul> <li>Upgrade Spring Framework (6.0.7), aspectJ (1.9.19) <a href="https://redirect.github.com/spring-projects/spring-retry/pull/336">#336</a></li> </ul> <h2>:heart: Contributors</h2> <p>Thank you to all the contributors who worked on this release:</p> <p><a href="https://github.com/esivakumar18"><code>@esivakumar18</code></a>, <a href="https://github.com/garyrussell"><code>@garyrussell</code></a>, <a href="https://github.com/hpoettker"><code>@hpoettker</code></a>, <a href="https://github.com/tobi5775"><code>@tobi5775</code></a>, and <a href="https://github.com/xak2000"><code>@xak2000</code></a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/spring-projects/spring-retry/commit/e925ddbbede3541f1461d165ca0edcced80a0f6b"><code>e925ddb</code></a> Release v2.0.1</li> <li><a href="https://github.com/spring-projects/spring-retry/commit/11489d1f15a91c06d3de351a97fc3e9ccc1ae0b6"><code>11489d1</code></a> Upgrade Spring Framework (6.0.7), aspectJ (1.9.19)</li> <li><a href="https://github.com/spring-projects/spring-retry/commit/b9a2f94e78bc0b45eb7223245725cdc7fb7f8684"><code>b9a2f94</code></a> Update CI pipeline</li> <li><a href="https://github.com/spring-projects/spring-retry/commit/a53dddc70bdaf87a1e570054ea548698c38c843c"><code>a53dddc</code></a> Change default order for <a href="https://github.com/EnableRetry"><code>@EnableRetry</code></a></li> <li><a href="https://github.com/spring-projects/spring-retry/commit/debc4adc9f27a1a671af4ab6507aee10e31569ee"><code>debc4ad</code></a> <a href="https://redirect.github.com/spring-projects/spring-retry/issues/22">GH-22</a>: Add <a href="https://github.com/EnableRetry"><code>@EnableRetry</code></a>.order()</li> <li><a href="https://github.com/spring-projects/spring-retry/commit/0b37cc082ee8f236fb285d7d471dea47587d4595"><code>0b37cc0</code></a> Added getting support links into README</li> <li><a href="https://github.com/spring-projects/spring-retry/commit/864617f198fd88feaebf5f4c515cc796207362c7"><code>864617f</code></a> Add notice and license to jar</li> <li><a href="https://github.com/spring-projects/spring-retry/commit/4f61cd692817ae94677b6fb4a22a5ba20e71aa8b"><code>4f61cd6</code></a> Apply Java Format plugin for the latest code</li> <li><a href="https://github.com/spring-projects/spring-retry/commit/065c3091c4e4590cf89090f615446c15e1d5110c"><code>065c309</code></a> <a href="https://redirect.github.com/spring-projects/spring-retry/issues/328">GH-328</a>: Fix RecoverAnnRH return types matching</li> <li><a href="https://github.com/spring-projects/spring-retry/commit/87307292a56aecd0060538973eb1f34b1d227e35"><code>8730729</code></a> Deprecate RetryListenSupport for default methods</li> <li>Additional commits viewable in <a href="https://github.com/spring-projects/spring-retry/compare/v2.0.0...v2.0.1">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Antoni Ivanov <aivanov@vmware.com>
Added one more "m" in `vdk list -mmmm` to show execution details It is aligned somewhat how the UI looks. Generally this is useful and quick solution if UI is not available. Testing Done: unit tests --------- Signed-off-by: Antoni Ivanov <aivanov@vmware.com>
yonitoo
requested review from
antoniivanov
and removed request for
gorankokin and
hzhristova
yonitoo
changed the title
added 2 commits
April 6, 2023 03:41
Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
antoniivanov
reviewed
Apr 6, 2023
antoniivanov
reviewed
Apr 6, 2023
projects/vdk-plugins/vdk-meta-jobs/src/vdk/plugin/meta_jobs/api/meta_job.py
Outdated
Show resolved
Hide resolved
Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
ivakoleva
approved these changes
Apr 7, 2023
projects/vdk-plugins/vdk-meta-jobs/src/vdk/plugin/meta_jobs/dag_validator.py
Outdated
Show resolved
Hide resolved
projects/vdk-plugins/vdk-meta-jobs/src/vdk/plugin/meta_jobs/dag_validator.py
Outdated
Show resolved
Hide resolved
projects/vdk-plugins/vdk-meta-jobs/src/vdk/plugin/meta_jobs/api/meta_job.py
Show resolved
Hide resolved
Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
for more information, see https://pre-commit.ci
gageorgiev
approved these changes
Apr 11, 2023
Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
for more information, see https://pre-commit.ci
Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
for more information, see https://pre-commit.ci
doks5
pushed a commit
that referenced
this pull request
Apr 12, 2023
What:
Introduce execution of jobs with arguments (synchronously) via DAGs.
The way a job dict looks now:
{
"job_name": "name-of-job",
"team_name": "team-of-job",
"fail_meta_job_on_error": True or False,
**"arguments": \<json\>**,
"depends_on": [name-of-job1, name-of-job2]
}
GitHub Issue:
[#1502
Testing Done: existing tests + new unit test
Closes #1502
Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
mivanov1988
pushed a commit
that referenced
this pull request
Apr 13, 2023
What:
Introduce execution of jobs with arguments (synchronously) via DAGs.
The way a job dict looks now:
{
"job_name": "name-of-job",
"team_name": "team-of-job",
"fail_meta_job_on_error": True or False,
**"arguments": \<json\>**,
"depends_on": [name-of-job1, name-of-job2]
}
GitHub Issue:
[#1502
Testing Done: existing tests + new unit test
Closes #1502
Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What:
Introduce execution of jobs with arguments (synchronously) via DAGs.
The way a job dict looks now:
{
"job_name": "name-of-job",
"team_name": "team-of-job",
"fail_meta_job_on_error": True or False,
"arguments": <json>,
"depends_on": [name-of-job1, name-of-job2]
}
GitHub Issue: #1502
Testing Done: existing tests + new unit test
Closes #1502