Docs: LoRA Configuration Inconsistencies

[danehans]

The LoRA configuration for the CPU-based deployment and GPU-based deployment differ. Update the CPU-based deployment to match that of the GPU deployment, e.g., one LoRA adapter named food-review-1 and "--max-loras" "2".

@nirrozenbaum WDYT about removing --max-cpu-loras in the GPU deployment?

[nirrozenbaum]

yes, I was exactly looking at this when reviewing the getting started PR yesterday.

completely agree 👍🏻

@nirrozenbaum WDYT about removing --max-cpu-loras in the GPU deployment?

I think it was Abdullah who added this flag in GPU deployment , would be good to verify why and if removing it is ok.

more generally I think it is important that lora configuration looks exactly the same, but IMO it’s ok that other config flags differ.

[sathvikpurushotham]

/assign @sathvikpurushotham

[kfswain]

Why remove --max-cpu-loras? that dictates the maximum number of adapters allowed to be stored in CPU mem, and isn't necessarily related to the difference in the two deployments.

What is the definition of done with this issue? Just making sure that the CPU & GPU models exposed match for simpler E2E? I don't think we should try for complete parity between the 2. The GPU deployment uses the LoRA syncer to manage dynamic loading of its adapters

[danehans]

I agree that parity is not expected. I created the issue when I identified a few inconsistencies that appeared to be unrelated to CPU/GPU modes. The goal is to provide consistency for CLI args that should be equivalent across the deployment options to reduce the potential for future issues due to discrepancies.