static resources removed from the configuration may never be deleted

[ixdy]

Originally filed as kubernetes/test-infra#17282

I mentioned this tangentially in kubernetes/test-infra#16047 (comment), but I want to pull it to a separate issue to be more easily highlight it.

Boskos doesn't delete static resources that are removed from the configuration if they are in use, to ensure that jobs don't fail, and to ensure that such resources are properly cleaned up by the janitor.

Originally, this was a reasonable decision, since Boskos periodically synced its storage against the configuration, and most likely such resources would eventually be free and thus deleted from storage.

After kubernetes/test-infra#13990, Boskos only syncs its storage against the configuration when the configuration changes (or when Boskos restarts). As a result, it may take a long time for static resources to be deleted, if ever.

There was a similar issue for DRLCs that I addressed in kubernetes/test-infra#16021, effectively by putting the DRLCs into lame-duck mode.

There isn't a clear way to indicate that static resources are in lame-duck mode, though.

Possible ways to address this bug, in increasing order of complexity:

1. Just delete static resources, regardless of what state they're in.
2. Periodically sync storage against the config. It's probably less expensive now, due to the improvements around locking.
3. Somehow indicate that resources are in lame-duck mode to prevent them from being leased, and then delete them once free:
   a. Add a field into the UserData for static resources. (Currently UserData is not used for static resources.)
   b. Set an ExpirationDate on static resources. (Currently ExpirationDate is not used for static resources.)
   c. Add a new field on the ResourceStatus indicating resources are in lame-duck mode.

Workaround until this bug is fixed: admins with access to the cluster where Boskos is running can just delete the resources manually using kubectl.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ixdy]

/reopen
/remove-lifecycle rotten

[k8s-ci-robot]

@ixdy: Reopened this issue.

In response to this:

/reopen
/remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[spiffxp]

/remove-lifecycle stale
This may have been addressed (#54 (comment)) but need to verify

/assign
/sig testing

[spiffxp]

I'm going to test this out by shrinking the k8s-infra-prow-build pool temporarily, specifically choosing some resources that are currently leased and not leased

[spiffxp]

/close
Yeah this works now. I removed some resources from the configmap boskos reads, some of which were in busy state. A version of boskos prior to #54 landing left the resources in free once they were released. The latest version of boskos correctly removed them.

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close
Yeah this works now. I removed some resources from the configmap boskos reads, some of which were in busy state. A version of boskos prior to #54 landing left the resources in free once they were released. The latest version of boskos correctly removed them.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ixdy]

Thanks for confirming this has been fixed!