-
Notifications
You must be signed in to change notification settings - Fork 5.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make ddl operations for placement rule atomic #28159
Comments
You should add another option, retrying sending requests to PD until successful. API is not only atomic, but also idempotent. That means we can retry DDL job forever until it succeeds, i.e. blocking. It can be archived with a middle state. And that is what we wanted to do at the very beginning of the previous work plan. But we decided not to do. Because in the previous version, there is no persistence on TiKV. Infoschema is merely a cache. It can not be inconsistent. But, since now we have persistence, this plan becomes useful again. The reason that I did not choose to rollback: our DDL can not be canceled in many cases anyway. We are not good at rollback. It is hard. BTW, please add it in the tracking issue, in the rule engine section, maybe. |
Yes I agree. We should be retrying requests for |
We discussed this today on our weekly meeting. For direct placement options, it is more straight forward because we can retry the placement option.. except we need to be careful that the retrying is not in the DDL queue (which would block other DDL jobs). For altering a @AilinKid can you please share your suggestion? |
how about moving the sync pd logic out of ddl txn? I mean do the sync after the ddl txn is committed for sure. This comes from a similar logic for what gc does now, for example, drop a table, when the ddl is committed already, the following action is to write the table id into the gc_delete_range physical table, which in done in For the case here, sync PD is a kind of afterward action, which needs the ddl txn is done for sure. When ddl is done, the bundle can be found in the meta, which is truth already, syncing pd is just time problem. (which avoided the retry ddl job which is time-costy) |
I like the idea, but the order of DDL jobs still needs to be kept. Bundle does not come with monotonic id like key-ranes, which means the job either can not be executed parallel or must take care of the order. Otherwise, Serial execution does not make a difference. And taking care of the order may introduce new inconsistency. BTW, now tiflash works by pulling meta from tidb regularly(one persistence). |
Summary for some candidates:
|
The third way of mine is that: doing it in Or we can make it async (in a standalone goroutine), strictly control the syncing sequence of of what the ddl job done is like (another subtask queue?) |
It is trivial about where the third method is done. But it will be complex to ensure order and consistency.
We can tell if it is timeout or not. In case you lost the response, you could send it again to gain the correct response.
The hard part is to determine the state that needs to rollback to. |
Yes, we can tell it wether it is a network or request error. But what I mean is that if it is a request error, it is meaningless to retry it because it will always be failed. So we still need to rollback.
Actually, we need not to add extra rollback stage for placement. If a ddl operation does not support rollback before, we just need to request PD in first stage. |
So you can just retry if it is network, but fail if it is
What if it supports? What is the original state? |
|
Yeah, you could do so, if you can construct the old placement rule bundles. |
Enhancement
For now, the ddl operations for placement rule is not atomic. For the current implement, if it fails to send request to PD, meta operations will rollback:
tidb/ddl/partition.go
Lines 1131 to 1135 in 680de92
It works well in most times. However, it is possible that PD have received and processed the request successfully but the response packet is lost in the network. In this case, the meta operations will rollback and left an inconsistent state with PD.
Some ways to resolve the problem:
The text was updated successfully, but these errors were encountered: