[Feature]: co-exist of multiply kv connector

[maobaolong]

🚀 The feature, motivation and pitch

Now, there are two kind of kv connector.

1. Offload kv cache to kv connector for reuse purpose
2. Transfer KVCache from P to D

But we could only specific one kv connector, this make the two above kv connectors cannot co-exist, i propose to do an abstract or use a type, maybe {p2p, offload} to support to find the needed kv connector.

In my view, the ideal way is

1. P check for partial cache hit and do prefill for the uncache hit part.
2. P transfer the minimal necessary kv cache(which mean Decode instance cannot obtain it from offload kv connector)
3. D receive KV cache from offload connector and P and update the kvcache
4. D execute the decode step by step until finish.
5. D save the kv cache into offload kv connector for reuse purpose
6. Goto 1.

Any idea about this?

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[maobaolong]

@mgoin thanks for your pr #17564 , this issue has been done by your PR. Thanks!