Running Delphi in offline mode crashes due to vLLM incompatibility

[schraderSimon]

Hi,

I think there's a bug in delphi offline mode when using the newest vLLM:

We installed delphi a few days ago and run e2e.py in offline mode. A fresh installation of delphi installs vLLM==0.10.2 and we encounter two problems:

1. Default gpu_memory_utilization=0.9 causes OOM/crash with no easy way in the interface to override.
2. LLM.generate() rejects prompt_token_ids (TypeError), so offline pipeline breaks.

We patched this, making gpu_memory_utilization a parameter in RunConfig, and making a change to offline.py to pass text prompts, e.g. prompts = TokensPrompt.from_token_ids(prompts).

We fixed this in a forked repo https://github.com/schraderSimon/delphi/tree/main .

Is this the correct way to fix it, or do you recommend using an older version of vLLM (which one)?

[SrGonao]

Simon I replied via discord, I'm not sure which platform you prefer for the discussion, but I will write here my reply:

It is not clear to me that 1) is a bug with delphi. Can you give more details on what you are doing? e2e.py is supposed to be an example template on how to programatically use delphi, and as such should be modified to fit people's own usage. I agree with you that currently there's no way to change memory utilization (which now is at 0.85) if you use delphi with the CLI. Which numbers do you have to use for it to work, which gpus are you using and which models are you using for the explainer model? Are the gpus free when you run delphi?
2) seems to be a breaking change from vllm which I was unware of. Your change seems to be the correct one

[kmaherx]

Hi, I've been working with Simon on this.

We agree that 1) is not a bug, but we do think it's a lack of a useful feature:

• Here's a Colab notebook demonstrating the issue
• The GPU used doesnt have 90% of its memory available to run the explainer model, so the default max_memory value of 0.9 causes an error: (EngineCore_DP0 pid=5409) ERROR 10-02 19:12:55 [core.py:718] ValueError: Free memory on device (16.77/22.16 GiB) on startup is less than desired GPU memory utilization (0.9, 19.94 GiB). Decrease GPU memory utilization or reduce GPU memory used by other processes.
• The max_memory parameter already exists in the code, it's just hardcoded as 0.9 in __main__.py and doesn't make it to the CLI as an argument.
• Our edit simply adds max_memory to the run config, allowing one to specify it from the CLI. Seems like it works based on the notebook linked above, as we get output explanations and reasonable classification scores.

I made a fork with minimal changes addressing only these two points (max_memory and vLLM changes). Making a pull request now in case you think this all looks reasonable.

=======================================
Since we're here, two additional things visible in the above notebook that we may raise separate issues about in the future:

1. Had to install specific plotly and kaleido versions along with Chrome in order to make output plots. I've seen previous issues on this, but I figured I'd mention here because the specific versions we used in the notebook above appear to work.
2. Seems like processes are not terminated correctly: [rank0]:[W1002 19:50:27.270197722 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()). Ran into this on both Colab and my laptop.

[SrGonao]

Thank you for raising this issue and for suggesting a PR!