Skip to content

[AMORO-2714]Delete resource when optimizer instance expired #2715

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Yao-MR wants to merge 1 commit into from
Closed

[AMORO-2714]Delete resource when optimizer instance expired #2715

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ams/server/src/main/java/com/netease/arctic/server/DefaultOptimizingService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -548,6 +548,7 @@ public void run() {
.forEach(task -> retryTask(task, queue)));
if (isExpired) {
LOG.info("Optimizer {} has been expired, unregister it", keepingTask.getOptimizer());
deleteResource(keepingTask.getOptimizer().getResourceId());

@majin1102 majin1102 Apr 7, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The createResource() maybe called by user.
Is it proper that system auto deleting resources managed by user himself?

@Yao-MR Yao-MR Apr 7, 2024

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the createResource is actually called by user,
and the deleteResource is also will be called by user, if user want the release the optimzier,

but if the optimizer instance failed by some unexpected casuse, the system will delete the instance auto, but will
not clean the resource record in the record, and this resource will never be used again and will cause dead dirty data in the db forever.

so when the instance is expired and be deleted by system, the resource associate with this instance should also be cleand

@majin1102 majin1102 Apr 9, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't quite agree with that.

The record left in db has its meaning, cause when the optimizer came back, AMS could identify this resource.
What you said I think is a dashboard design issue, the optimizer page don't reveal inactive optimizers. I think we could improve in that way.
A good and simple princeple is who creates who deletes.

@Yao-MR Yao-MR Apr 10, 2024

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks your comment jin, agree with it, and will check the lifetime of resource again ,and get one graceful idea the resolve it.

unregisterOptimizer(token);
} else {
LOG.debug("Optimizer {} is being touched, keep it", keepingTask.getOptimizer());
Expand Down