Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Transformers Loader: 4-bit and 8-bit loading of c4ai-command-r-plus results in nonsense responses. #5838

Closed
1 task done
RandomInternetPreson opened this issue Apr 9, 2024 · 8 comments
Closed
1 task done

Transformers Loader: 4-bit and 8-bit loading of c4ai-command-r-plus results in nonsense responses. #5838

RandomInternetPreson opened this issue Apr 9, 2024 · 8 comments
Labels
bug Something isn't working

Comments

@RandomInternetPreson
Copy link

Describe the bug

When I use the fp16 model of c4ai-command-r-plus (https://huggingface.co/CohereForAI/c4ai-command-r-plus) and load it via transformers 4-bit or 8-bit the resulting model only outputs repeating characters, often resulting in this type of output:

"authorauthorauthorauthorauthorauthorauthorauthorauthorauthorauthorauthorauthorauthorauthorauthorauthorauthorauthorauthorauthorbeginbeginauthor-sectionsectionauthor-sectionbegin-section"

I have quantized the model using exllamav2, installed the most recent build into textgen, and have successfully loaded the model with it functioning well using the exllamav2 loader.

Is there an existing issue for this?

  • I have searched the existing issues

Reproduction

Select the Transformers loader, and try loading the original fp16 model in either 4-bit or 8-bit (enabling trusted code is necessary for 8-bit, and it results in the same output type and constancy as 4-bit), the model will only output random strings of repeating words.

Screenshot

image

image

Logs

19:27:07-289121 INFO     Starting Text generation web UI                        

Running on local URL:  http://127.0.0.1:7860

19:27:19-767043 INFO     Loading "c4ai-command-r-plus"                          
19:27:19-771996 INFO     TRANSFORMERS_PARAMS=                                   
{   'low_cpu_mem_usage': True,
    'torch_dtype': torch.float16,
    'device_map': 'auto',
    'max_memory': {   0: '20000MiB',
                      1: '20000MiB',
                      2: '20000MiB',
                      3: '20000MiB',
                      4: '20000MiB',
                      5: '20000MiB',
                      6: '20000MiB',
                      'cpu': '99GiB'},
    'quantization_config': BitsAndBytesConfig {
  "_load_in_4bit": true,
  "_load_in_8bit": false,
  "bnb_4bit_compute_dtype": "float16",
  "bnb_4bit_quant_storage": "uint8",
  "bnb_4bit_quant_type": "nf4",
  "bnb_4bit_use_double_quant": false,
  "llm_int8_enable_fp32_cpu_offload": false,
  "llm_int8_has_fp16_weight": false,
  "llm_int8_skip_modules": null,
  "llm_int8_threshold": 6.0,
  "load_in_4bit": true,
  "load_in_8bit": false,
  "quant_method": "bitsandbytes"
}
}

Loading checkpoint shards: 100%|████████████████| 44/44 [00:33<00:00,  1.31it/s]
19:27:54-637659 INFO     LOADER: "Transformers"                                 
19:27:54-638494 INFO     TRUNCATION LENGTH: 8192                                
19:27:54-638974 INFO     INSTRUCTION TEMPLATE: "Command-R"                      
19:27:54-639401 INFO     Loaded the model in 34.87 seconds.                     
Output generated in 6.39 seconds (8.30 tokens/s, 53 tokens, context 91, seed 803219425)
19:28:11-452153 INFO     "Debug-deterministic" preset:                          
{   'top_k': 1,
    'do_sample': False}
19:28:15-583523 INFO     Deleted "logs/chat/Assistant/20240409-19-00-39.json".  
Output generated in 3.70 seconds (8.93 tokens/s, 33 tokens, context 63, seed 1377676542)

System Info

Ubuntu, 7xrtx 4090, msi, xeonW7
@RandomInternetPreson RandomInternetPreson added the bug Something isn't working label Apr 9, 2024
@Ph0rk0z
Copy link
Contributor

Ph0rk0z commented Apr 10, 2024

With latest transformers and bnb, does it do it too?

@RandomInternetPreson
Copy link
Author

@Ph0rk0z Good question! I installed the latest bitsandbytes and got the following install error:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. datasets 2.18.0 requires fsspec[http]<=2024.2.0,>=2023.1.0, but you have fsspec 2024.3.1 which is incompatible. torchvision 0.17.1+cu121 requires torch==2.2.1, but you have torch 2.2.2 which is incompatible. torchaudio 2.2.1+cu121 requires torch==2.2.1, but you have torch 2.2.2 which is incompatible.

The install did complete however, and when I tried loading the model via 4-bit I got this error:

09:50:24-061953 INFO Starting Text generation web UI

Running on local URL: http://127.0.0.1:7860

ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/applications.py", line 123, in call
await self.middleware_stack(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in call
raise exc
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in call
await self.app(scope, receive, _send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/route_utils.py", line 680, in call
await self.app(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 756, in call
await self.middleware_stack(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
await route.handle(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
await self.app(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
response = await func(request)
^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/routing.py", line 193, in run_endpoint_function
return await run_in_threadpool(dependant.call, **values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/concurrency.py", line 42, in run_in_threadpool
return await anyio.to_thread.run_sync(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 851, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/routes.py", line 377, in main
return templates.TemplateResponse(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/templating.py", line 229, in TemplateResponse
template = self.get_template(name)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/templating.py", line 143, in get_template
return self.env.get_template(name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/jinja2/environment.py", line 1010, in get_template
return self._load_template(name, globals)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/jinja2/environment.py", line 969, in _load_template
template = self.loader.load(self, name, self.make_globals(globals))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/jinja2/loaders.py", line 125, in load
source, filename, uptodate = self.get_source(environment, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/jinja2/loaders.py", line 206, in get_source
with open(filename, encoding=self.encoding) as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/modules/block_requests.py", line 46, in my_open
file_contents = file_contents.replace(b'\t\t<script\n\t\t\tsrc="https://cdnjs.cloudflare.com/ajax/libs/iframe-resizer/4.3.9/iframeResizer.contentWindow.min.js"\n\t\t\tasync\n\t\t></script>', b'')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: replace() argument 1 must be str, not bytes
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/applications.py", line 123, in call
await self.middleware_stack(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in call
raise exc
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in call
await self.app(scope, receive, _send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/route_utils.py", line 680, in call
await self.app(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 756, in call
await self.middleware_stack(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
await route.handle(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
await self.app(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
response = await func(request)
^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/routing.py", line 193, in run_endpoint_function
return await run_in_threadpool(dependant.call, **values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/concurrency.py", line 42, in run_in_threadpool
return await anyio.to_thread.run_sync(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 851, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/routes.py", line 377, in main
return templates.TemplateResponse(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/templating.py", line 229, in TemplateResponse
template = self.get_template(name)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/templating.py", line 143, in get_template
return self.env.get_template(name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/jinja2/environment.py", line 1010, in get_template
return self._load_template(name, globals)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/jinja2/environment.py", line 969, in _load_template
template = self.loader.load(self, name, self.make_globals(globals))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/jinja2/loaders.py", line 125, in load
source, filename, uptodate = self.get_source(environment, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/jinja2/loaders.py", line 206, in get_source
with open(filename, encoding=self.encoding) as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/modules/block_requests.py", line 46, in my_open
file_contents = file_contents.replace(b'\t\t<script\n\t\t\tsrc="https://cdnjs.cloudflare.com/ajax/libs/iframe-resizer/4.3.9/iframeResizer.contentWindow.min.js"\n\t\t\tasync\n\t\t></script>', b'')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: replace() argument 1 must be str, not bytes
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/applications.py", line 123, in call
await self.middleware_stack(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in call
raise exc
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in call
await self.app(scope, receive, _send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/route_utils.py", line 680, in call
await self.app(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 756, in call
await self.middleware_stack(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
await route.handle(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
await self.app(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
response = await func(request)
^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/routing.py", line 193, in run_endpoint_function
return await run_in_threadpool(dependant.call, **values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/concurrency.py", line 42, in run_in_threadpool
return await anyio.to_thread.run_sync(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 851, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/routes.py", line 377, in main
return templates.TemplateResponse(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/templating.py", line 229, in TemplateResponse
template = self.get_template(name)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/templating.py", line 143, in get_template
return self.env.get_template(name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/jinja2/environment.py", line 1010, in get_template
return self._load_template(name, globals)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/jinja2/environment.py", line 969, in _load_template
template = self.loader.load(self, name, self.make_globals(globals))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/jinja2/loaders.py", line 125, in load
source, filename, uptodate = self.get_source(environment, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/jinja2/loaders.py", line 206, in get_source
with open(filename, encoding=self.encoding) as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/modules/block_requests.py", line 46, in my_open
file_contents = file_contents.replace(b'\t\t<script\n\t\t\tsrc="https://cdnjs.cloudflare.com/ajax/libs/iframe-resizer/4.3.9/iframeResizer.contentWindow.min.js"\n\t\t\tasync\n\t\t></script>', b'')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: replace() argument 1 must be str, not bytes
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/applications.py", line 123, in call
await self.middleware_stack(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in call
raise exc
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in call
await self.app(scope, receive, _send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/route_utils.py", line 680, in call
await self.app(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 756, in call
await self.middleware_stack(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
await route.handle(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
await self.app(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
response = await func(request)
^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/routing.py", line 193, in run_endpoint_function
return await run_in_threadpool(dependant.call, **values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/concurrency.py", line 42, in run_in_threadpool
return await anyio.to_thread.run_sync(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 851, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/routes.py", line 377, in main
return templates.TemplateResponse(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/templating.py", line 229, in TemplateResponse
template = self.get_template(name)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/templating.py", line 143, in get_template
return self.env.get_template(name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/jinja2/environment.py", line 1010, in get_template
return self._load_template(name, globals)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/jinja2/environment.py", line 969, in _load_template
template = self.loader.load(self, name, self.make_globals(globals))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/jinja2/loaders.py", line 125, in load
source, filename, uptodate = self.get_source(environment, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/jinja2/loaders.py", line 206, in get_source
with open(filename, encoding=self.encoding) as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/modules/block_requests.py", line 46, in my_open
file_contents = file_contents.replace(b'\t\t<script\n\t\t\tsrc="https://cdnjs.cloudflare.com/ajax/libs/iframe-resizer/4.3.9/iframeResizer.contentWindow.min.js"\n\t\t\tasync\n\t\t></script>', b'')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: replace() argument 1 must be str, not bytes
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/applications.py", line 123, in call
await self.middleware_stack(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in call
raise exc
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in call
await self.app(scope, receive, _send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/route_utils.py", line 680, in call
await self.app(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 756, in call
await self.middleware_stack(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
await route.handle(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
await self.app(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
response = await func(request)
^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/fastapi/routing.py", line 193, in run_endpoint_function
return await run_in_threadpool(dependant.call, **values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/concurrency.py", line 42, in run_in_threadpool
return await anyio.to_thread.run_sync(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 851, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/routes.py", line 377, in main
return templates.TemplateResponse(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/templating.py", line 229, in TemplateResponse
template = self.get_template(name)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/starlette/templating.py", line 143, in get_template
return self.env.get_template(name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/jinja2/environment.py", line 1010, in get_template
return self._load_template(name, globals)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/jinja2/environment.py", line 969, in _load_template
template = self.loader.load(self, name, self.make_globals(globals))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/jinja2/loaders.py", line 125, in load
source, filename, uptodate = self.get_source(environment, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/jinja2/loaders.py", line 206, in get_source
with open(filename, encoding=self.encoding) as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/modules/block_requests.py", line 46, in my_open
file_contents = file_contents.replace(b'\t\t<script\n\t\t\tsrc="https://cdnjs.cloudflare.com/ajax/libs/iframe-resizer/4.3.9/iframeResizer.contentWindow.min.js"\n\t\t\tasync\n\t\t></script>', b'')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: replace() argument 1 must be str, not bytes
╭────────────────────────── Traceback (most recent call last) ───────────────────────────╮
│ /home/myself/Desktop/OobApril10NonDev/text-generation-webui/server.py:255 in │
│ │
│ 254 # Launch the web UI │
│ ❱ 255 create_interface() │
│ 256 while True: │
│ │
│ /home/myself/Desktop/OobApril10NonDev/text-generation-webui/server.py:161 in │
│ create_interface │
│ │
│ 160 with OpenMonkeyPatch(): │
│ ❱ 161 shared.gradio['interface'].launch( │
│ 162 max_threads=64, │
│ │
│ /home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/py │
│ thon3.11/site-packages/gradio/blocks.py:2210 in launch │
│ │
│ 2209 ): │
│ ❱ 2210 raise ValueError( │
│ 2211 "When localhost is not accessible, a shareable link must be cre │
╰────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: When localhost is not accessible, a shareable link must be created. Please set
share=True or check your proxy settings to allow access to localhost.
(/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env) myself@myself-System-Product-Name:~/Desktop/OobApril10NonDev/text-generation-webui$

Looks like someone else is getting the same error:
#5839

@RandomInternetPreson
Copy link
Author

When I set Share=True in the server.py code and have the latest bits and bytes installed I get this error:

0:07:44-841486 INFO Starting Text generation web UI

Running on local URL: http://127.0.0.1:7860

Running on public URL: https://6faaae8194bc7c77ae.gradio.live

This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run gradio deploy from Terminal to deploy to Spaces (https://huggingface.co/spaces)
10:08:06-715789 INFO Loading "c4ai-command-r-plus"
10:08:06-724076 INFO TRANSFORMERS_PARAMS=
{ 'low_cpu_mem_usage': True,
'torch_dtype': torch.float16,
'device_map': 'auto',
'max_memory': { 0: '20000MiB',
1: '20000MiB',
2: '20000MiB',
3: '20000MiB',
4: '20000MiB',
5: '20000MiB',
6: '20000MiB',
'cpu': '99GiB'},
'quantization_config': BitsAndBytesConfig {
"_load_in_4bit": true,
"_load_in_8bit": false,
"bnb_4bit_compute_dtype": "float16",
"bnb_4bit_quant_storage": "uint8",
"bnb_4bit_quant_type": "nf4",
"bnb_4bit_use_double_quant": false,
"llm_int8_enable_fp32_cpu_offload": false,
"llm_int8_has_fp16_weight": false,
"llm_int8_skip_modules": null,
"llm_int8_threshold": 6.0,
"load_in_4bit": true,
"load_in_8bit": false,
"quant_method": "bitsandbytes"
}
}

Loading checkpoint shards: 2%|▌ | 1/44 [00:01<00:47, 1.10s/it]
10:08:08-542802 ERROR Failed to load the model.
Traceback (most recent call last):
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/modules/ui_model_menu.py", line 245, in load_model_wrapper
shared.model, shared.tokenizer = load_model(selected_model, loader)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/modules/models.py", line 87, in load_model
output = load_func_maploader
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/modules/models.py", line 246, in huggingface_loader
model = LoaderClass.from_pretrained(path_to_model, **params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 563, in from_pretrained
return model_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/transformers/modeling_utils.py", line 3531, in from_pretrained
) = cls._load_pretrained_model(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/transformers/modeling_utils.py", line 3958, in _load_pretrained_model
new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/transformers/modeling_utils.py", line 814, in _load_state_dict_into_meta_model
hf_quantizer.create_quantized_param(model, param, param_name, param_device, state_dict, unexpected_keys)
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/transformers/quantizers/quantizer_bnb_4bit.py", line 219, in create_quantized_param
new_value = bnb.nn.Params4bit(new_value, requires_grad=False, **kwargs).to(target_device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/bitsandbytes/nn/modules.py", line 324, in to
return self._quantize(device)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/bitsandbytes/nn/modules.py", line 289, in _quantize
w_4bit, quant_state = bnb.functional.quantize_4bit(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myself/Desktop/OobApril10NonDev/text-generation-webui/installer_files/env/lib/python3.11/site-packages/bitsandbytes/functional.py", line 1206, in quantize_4bit
lib.cquantize_blockwise_fp16_nf4(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'cquantize_blockwise_fp16_nf4'

@oobabooga
Copy link
Owner

I haven't been able to run command-r-plus in 4-bit bnb format because I don't have enough memory, but the prequantized c4ai-command-r-v01-4bit works very well for me. Does c4ai-command-r-plus-4bit generate coherent results?

@RandomInternetPreson
Copy link
Author

:3 Hello! I'm currently downloading and will reply back with the result, but I suspect if it's working for you it will work for me too. I was reading the code at that link and it seemed like it's what textgen does too when doing 4 and 8 bit conversions on the fly with the transformers loader.

Maybe it's not textgen then?

Either way I'll let you know about the 4-bit version when it finishes downloading. Also, wanted to let you know I've got the new mixtral and dbrx models running in text gen with exllamav2 quants. Mixtral quantized without issue using the transformers loader on day one, and dbrx has a weird issue where it tries to load onto one gpu regardless of how one uses bitsandbytes. I am forced to run that one with exllamav2 quants only, but I've tried with the code from the databricks repo and the memory loading behavior is the same.

Ty so much for responding to the new model releases, I really appreciate your work <3

@RandomInternetPreson
Copy link
Author

Hmm, the model finished downloading and I tried loading it via the transformers loader, and I got the exact same response. I tried only loading the model across 3 gpus by exporting only 3 to the terminal, and still the same response. Also I'm using a version of oobabooga that is from April 10 (yestrday).

Closed
@RandomInternetPreson
Copy link
Author

RandomInternetPreson commented Apr 13, 2024 •
edited

I have figured out the solution, it was transformers. The current version is over 2 weeks old and it was definitely the issue for me. I did the following:

  1. Reinstalled textgen from scratch
  2. tried the prequantized 4-bit and it was still causing the gibberish error
  3. Installed the dev version of transformers pip install git+https://github.com/huggingface/transformers
  4. Tried the prequantized 4-bit and it was working

https://github.com/huggingface/transformers/tree/b109257f4fb8b1166e7c53cc5418632014ed53a5

This is the commit for the version of transformers that I ended up getting by installing the dev source today.

Works with fp16 quantized models on the fly using the transformers loader too!

@oobabooga
Copy link
Owner

The last released version is probably not reading the rope scale metadata correctly, causing the nonsense output. Good to know that the update fixed it, I'll update to the new version as soon as it comes out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@RandomInternetPreson @Ph0rk0z @oobabooga