Address Phi modeling update

[Aakash-kaushik]

Closes #2422

Facing the error in:
vllm-openai docker image 0.2.7

 File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/workspace/vllm/entrypoints/openai/api_server.py", line 737, in <module>
    engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/workspace/vllm/engine/async_llm_engine.py", line 500, in from_engine_args
    engine = cls(parallel_config.worker_use_ray,
  File "/workspace/vllm/engine/async_llm_engine.py", line 273, in __init__
    self.engine = self._init_engine(*args, **kwargs)
  File "/workspace/vllm/engine/async_llm_engine.py", line 318, in _init_engine
    return engine_class(*args, **kwargs)
  File "/workspace/vllm/engine/llm_engine.py", line 111, in __init__
    self._init_workers()
  File "/workspace/vllm/engine/llm_engine.py", line 146, in _init_workers
    self._run_workers("load_model")
  File "/workspace/vllm/engine/llm_engine.py", line 795, in _run_workers
    driver_worker_output = getattr(self.driver_worker,
  File "/workspace/vllm/worker/worker.py", line 81, in load_model
    self.model_runner.load_model()
  File "/workspace/vllm/worker/model_runner.py", line 64, in load_model
    self.model = get_model(self.model_config)
  File "/workspace/vllm/model_executor/model_loader.py", line 65, in get_model
    model = model_class(model_config.hf_config, linear_method)
  File "/workspace/vllm/model_executor/models/phi_1_5.py", line 263, in __init__
    self.transformer = PhiModel(config, linear_method)
  File "/workspace/vllm/model_executor/models/phi_1_5.py", line 219, in __init__
    self.h = nn.ModuleList([
  File "/workspace/vllm/model_executor/models/phi_1_5.py", line 220, in <listcomp>
    PhiLayer(config, linear_method)
  File "/workspace/vllm/model_executor/models/phi_1_5.py", line 186, in __init__
    eps=config.layer_norm_epsilon)
  File "/usr/local/lib/python3.10/dist-packages/transformers/configuration_utils.py", line 265, in __getattribute__
    return super().__getattribute__(key)
AttributeError: 'PhiConfig' object has no attribute 'layer_norm_epsilon'. Did you mean: 'layer_norm_eps'?

It is infact layer_norm_eps even in the phi1.5 config on hugging face

[simon-mo]

nice catch!

[simon-mo]

please fix the formatting

[WoosukKwon]

This PR breaks the support for Phi-2, which uses layer_norm_epsilon instead of layer_norm_eps.

[Aakash-kaushik]

This PR breaks the support for Phi-2, which uses layer_norm_epsilon instead of layer_norm_eps.

does using layer_norm_eps for phi-1.5 make sense and for all other cases of phi if in future they do roll out something can be layer_norm_epsilon

[WoosukKwon]

Hi @Aakash-kaushik, can we do something like

eps = getattr(config, "layer_norm_eps", None)
if eps is None:
    eps = getattr(config, "layer_norm_epsilon", 1e-6)

[Aakash-kaushik]

Hi @Aakash-kaushik, can we do something like

eps = getattr(config, "layer_norm_eps", None)
if eps is None:
    eps = getattr(config, "layer_norm_epsilon", 1e-6)

hey, i have done something like this but instead of the last assumption i left it to fail, assuming epsilon if missing is not a great idea, my code looks like.

layer_norm_eps = getattr(PretrainedConfig, "layer_norm_eps", None)
if layer_norm_eps is None:
    layer_norm_eps = PretrainedConfig.layer_norm_epsilon

i am struggling with the format.sh script, if someone can format the code i can push it.
Error:

concurrent.futures.process._RemoteTraceback: 
'''
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.10/3.10.13_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/process.py", line 392, in wait_result_broken_or_wakeup
    result_item = result_reader.recv()
  File "/opt/homebrew/Cellar/python@3.10/3.10.13_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
TypeError: TomlDecodeError.__init__() missing 2 required positional arguments: 'doc' and 'pos'
'''

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "//bin/yapf", line 8, in <module>
    sys.exit(run_main())
  File "//lib/python3.10/site-packages/yapf/__init__.py", line 360, in run_main
    sys.exit(main(sys.argv))
  File "//lib/python3.10/site-packages/yapf/__init__.py", line 124, in main
    changed = FormatFiles(
  File "//lib/python3.10/site-packages/yapf/__init__.py", line 199, in FormatFiles
    changed |= future.result()
  File "/opt/homebrew/Cellar/python@3.10/3.10.13_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/opt/homebrew/Cellar/python@3.10/3.10.13_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending

[simon-mo]

I don't this this works? PretrainedConfig is the class instead of the instance.

[simon-mo]

ok i have a fix but now is running into

AttributeError: 'PhiConfig' object has no attribute 'rotary_dim'

This is #2422 and I will push to this PR to fix it.

Looks like Phi changed things...

[WoosukKwon]

@simon-mo The Phi 2 and Phi-1.5 models were recently (after this PR) updated to be compatible with HF transformers. Now we need to update the model code. I can do this.

[simon-mo]

I need to fix this to get CI working. Got it working now, running some tests and will ask for review.

[simon-mo]

I fixed the variable names, but currently facing weights naming mismatch (phi renamed some weights). I will skip this test in CI first and come back to this.

[simon-mo]

Superseded by #2428

[CHesketh76]

Was this resolved? I am also getting the same error when running this:

python -m vllm.entrypoints.openai.api_server --model TheBloke/phi-2-GPTQ --quantization gptq --trust-remote-code

[Aakash-kaushik]

Was this resolved? I am also getting the same error when running this:

python -m vllm.entrypoints.openai.api_server --model TheBloke/phi-2-GPTQ --quantization gptq --trust-remote-code

#2428 was merged to the main branch, there has been no release after that. You can use that branch with the fix