Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE]: Add RemoveAfter tag on all UCX jobs, so that they're cleaned up by watchdog even if they are in the approved instance pool #977

Closed
1 task done
Tracked by #1528
nfx opened this issue Feb 26, 2024 · 0 comments
Closed
1 task done
Tracked by #1528

[FEATURE]: Add RemoveAfter tag on all UCX jobs, so that they're cleaned up by watchdog even if they are in the approved instance pool #977

nfx opened this issue Feb 26, 2024 · 0 comments
Assignees
@pritishpai
Labels
integration test failure tech debt chores and design flaws

Comments

@nfx
Copy link
Collaborator

nfx commented Feb 26, 2024 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Problem statement

as the number of installed UCX test assets grows bigger (10 pages of test workflows) ...
image

... the real assessment job progressively takes more time to complete, resulting in the nightly test flakiness.
image

Proposed Solution

Add RemoveAfter tag on all UCX jobs, so that they're cleaned up by watchdog even if they are in the approved instance pool

Additional Context

Relevant modifications have to be done in https://github.com/databrickslabs/watchdog/blob/main/scan/objects/compute.go#L140-L166
@nfx nfx added enhancement New feature or request needs-triage labels Feb 26, 2024
@nfx nfx added tech debt chores and design flaws integration test failure and removed needs-triage labels Mar 5, 2024
@HariGS-DB HariGS-DB mentioned this issue Mar 7, 2024
Merged
11 tasks
@nfx nfx moved this from Triage to Refined in UCX (roadmap) Apr 10, 2024
@nfx nfx removed the enhancement New feature or request label Apr 22, 2024
@nfx nfx mentioned this issue Apr 24, 2024
Open
12 tasks
@nfx nfx moved this from Refined to Month Backlog in UCX (roadmap) May 2, 2024
@pritishpai pritishpai self-assigned this May 15, 2024
@pritishpai pritishpai mentioned this issue May 15, 2024
Merged
11 tasks
github-merge-queue bot pushed a commit that referenced this issue May 17, 2024
@pritishpai
Add RemoveAfter tag on all UCX jobs for cleanup (#1699)
888e8fa 
## Changes
Add RemoveAfter tag on all UCX jobs, so that they're cleaned up by
watchdog even if they are in the approved instance pool

### Linked issues
Partial work from #977 

### Functionality 

- [ ] added relevant user documentation
- [ ] added new CLI command
- [ ] modified existing command: `databricks labs ucx ...`
- [ ] added a new workflow
- [ ] modified existing workflow: `...`
- [ ] added a new table
- [ ] modified existing table: `...`

### Tests
<!-- How is this tested? Please see the checklist below and also
describe any other relevant tests -->

- [ ] manually tested
- [ ] added unit tests
- [ ] added integration tests
- [ ] verified on staging environment (screenshot attached)
@pritishpai pritishpai mentioned this issue May 21, 2024
Merged
11 tasks
nfx pushed a commit that referenced this issue May 22, 2024
@pritishpai
[internal] Test job purge timeout to 1 hour and UTC time in tags (#1737)
47f8784 
## Changes
Changing the jobs timeout tag to 1 hour after creation. UTC time for the
time string.

### Linked issues
Partly introduces #977 

### Functionality 

- [ ] added relevant user documentation
- [ ] added new CLI command
- [ ] modified existing command: `databricks labs ucx ...`
- [ ] added a new workflow
- [ ] modified existing workflow: `...`
- [ ] added a new table
- [ ] modified existing table: `...`

### Tests
<!-- How is this tested? Please see the checklist below and also
describe any other relevant tests -->

- [ ] manually tested
- [ ] added unit tests
- [x] added integration tests
- [ ] verified on staging environment (screenshot attached)
@pritishpai pritishpai mentioned this issue May 23, 2024
Merged
11 tasks
nfx pushed a commit that referenced this issue May 23, 2024
@pritishpai
[internal] Add RemoveAfter tag/property to additional testing artifac…
f0b3341 
…ts (#1748)

## Changes
Add RemoveAfter tag to additional artifacts

Add it as tag to:

- clusters
- instance pools
- sql warehouse

Add it a property to:

- schemas
- tables

### Linked issues
Introduces #977 

### Functionality 

- [ ] added relevant user documentation
- [ ] added new CLI command
- [ ] modified existing command: `databricks labs ucx ...`
- [ ] added a new workflow
- [ ] modified existing workflow: `...`
- [ ] added a new table
- [ ] modified existing table: `...`

### Tests
<!-- How is this tested? Please see the checklist below and also
describe any other relevant tests -->

- [x] manually tested
- [ ] added unit tests
- [x] added integration tests
- [ ] verified on staging environment (screenshot attached)
@pritishpai pritishpai mentioned this issue May 23, 2024
Merged
11 tasks
nfx pushed a commit that referenced this issue May 24, 2024
@pritishpai
[internal] Add RemoveAfter property to additional testing artifacts (#…
44babc5 
…1755)

## Changes
Add RemoveAfter property to additional artifacts

Add it as property to:

tables
schemas

### Linked issues
Party introduces #977 

### Functionality 

- [ ] added relevant user documentation
- [ ] added new CLI command
- [ ] modified existing command: `databricks labs ucx ...`
- [ ] added a new workflow
- [ ] modified existing workflow: `...`
- [ ] added a new table
- [ ] modified existing table: `...`

### Tests
<!-- How is this tested? Please see the checklist below and also
describe any other relevant tests -->

- [x] manually tested
- [ ] added unit tests
- [x] added integration tests
- [ ] verified on staging environment (screenshot attached)
@nfx nfx closed this as completed Jun 3, 2024
@github-project-automation github-project-automation bot moved this from Month Backlog to Archive in UCX (roadmap) Jun 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pritishpai pritishpai

Labels
integration test failure tech debt chores and design flaws
Projects
UCX (roadmap)
Archived in project
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nfx @pritishpai