Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fast DDL sort path change to support multiple fast DDL jobs running simultaneously #52596

Closed
lance6716 opened this issue Apr 15, 2024 · 1 comment
Labels
affects-8.1 component/ddl This issue is related to DDL of TiDB. severity/moderate type/bug The issue is confirmed as a bug.

Comments

@lance6716
Copy link
Contributor

Enhancement

After walk the fast DDL sort path, there should be functionality that a) must remove all folders that won't be used in future, maybe caused by DDL owner switch and has finished this job. And b) optionally remove not running folders to release space for other tasks.

To support a), ddl can maintain a memory cache of all processing = 1 jobs of tidb_ddl_job table, if a folder's job ID does not exist in the cache, we can know it's finished by other DDL owner. The cache has some similarity with runningJobs which is the picked running job of this DDL owner, maybe we can reuse that structure.

To support b), we can reuse the runningJobs to know if the job is running. And limit the caller goroutine to not concurrently call b) and modify runningJobs

@lance6716 lance6716 added the type/enhancement The issue or PR belongs to an enhancement. label Apr 15, 2024
@lance6716
Copy link
Contributor Author

/cc @tangenta

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-8.1 component/ddl This issue is related to DDL of TiDB. severity/moderate type/bug The issue is confirmed as a bug.
Projects
None yet
Development

No branches or pull requests

3 participants