The generate_schedules() missed the case of separating prefill and decode stages (on different GPUs)
In practice, we separate the two stages on different machines, i.e. one vm does prefill, the other does decoding. The paper mentioned it doesn't consider network topology, which is OK, but here it should still include the case of separating two stages onto different logical devices
The generate_schedules() missed the case of separating prefill and decode stages (on different GPUs)
In practice, we separate the two stages on different machines, i.e. one vm does prefill, the other does decoding. The paper mentioned it doesn't consider network topology, which is OK, but here it should still include the case of separating two stages onto different logical devices