Replies: 8 comments 26 replies
-
|
Try updating the web UI, you are at an older commit. For me it worked after installing einops: I could only get to 7000 input tokens with 24gb vram, and the generation was super slow:
Using triton + bf16 it gets a bit faster:
These changes were necessary to + config = transformers.AutoConfig.from_pretrained(
+ Path(f"{shared.args.model_dir}/{model_name}"),
+ trust_remote_code=True
+ )
+ config.attn_config['attn_impl'] = 'triton'
+
# Load the model in simple 16-bit mode by default
if not any([shared.args.cpu, shared.args.load_in_8bit, shared.args.wbits, shared.args.auto_devices, shared.args.disk, shared.args.gpu_memory is not None, shared.args.cpu_memory is not None, shared.args.deepspeed, shared.args.flexgen, shared.model_type in ['rwkv', 'llamacpp']]):
- model = LoaderClass.from_pretrained(Path(f"{shared.args.model_dir}/{model_name}"), low_cpu_mem_usage=True, torch_dtype=torch.bfloat16 if shared.args.bf16 else torch.float16, trust_remote_code=trust_remote_code)
+ model = LoaderClass.from_pretrained(Path(f"{shared.args.model_dir}/{model_name}"), low_cpu_mem_usage=True, torch_dtype=torch.bfloat16 if shared.args.bf16 else torch.float16, trust_remote_code=trust_remote_code, config=config)
+ print("yes")
Then start with |
Beta Was this translation helpful? Give feedback.
-
|
8bit mode also doesn't work |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @oobabooga , I've gotten to the einops missing library error. Where are you typing "pip install einops"? Are you setting a venv before running the command? |
Beta Was this translation helpful? Give feedback.
-
|
This worked for me (your path and env name may vary): cd ~/text-generation-webui Model loads and works fine after this. |
Beta Was this translation helpful? Give feedback.
-
|
You can use the cmd.bat script if you used the one click installer |
Beta Was this translation helpful? Give feedback.
-
|
i can not run it During handling of the above exception, another exception occurred: Traceback (most recent call last): |
Beta Was this translation helpful? Give feedback.
-
|
I was able to get this working by running
when I prompted it to write some stuff, both times it started out coherent, then started devolving into madness, eventually reaching a point where it just randomly spat out words one after the other: Is there anything we have to do to properly expand that token limit length? Maybe I'm not doing that correctly. |
Beta Was this translation helpful? Give feedback.
-
|
tried this and changing the repetition penality lower caused it to do even weird things with repetition. couldn't quite find a sweet spot. I basically gave it the start of a made up novel premise and chapter 1: and it always ends up degenerating into madness. even with the boilerplate python code on huggingface.. so not sure at the moment whats what. |
Beta Was this translation helpful? Give feedback.
-
|
Same observation as EnviralDesign on my end, I give it a large corpus (5k tokens) as starting point and the next 50 or so generated tokens kinda match the input I provided but then the model starts to hallucinate, it either repeats the same thing over and over again or switches to something completely different / incoherent. |
Beta Was this translation helpful? Give feedback.
-
|
Was hoping I had something setup wrong. Perhaps the mpt architecture needs something else or something different to work properly other than remote code |
Beta Was this translation helpful? Give feedback.
-
|
ValueError: Loading models\mpt-7b-storywriter-4bit-128g requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option trust_remote_code=True to remove this error. |
Beta Was this translation helpful? Give feedback.
-
|
Perhaps you can point me in the right direction?? "updated webui.py run command to this run_cmd("python server.py --model-menu --notebook --model mosaicml_mpt-7b-storywriter --trust-remote-code")" But there is a "thousand" places where yuo can put this.. how do we know where is goes?? Also the dokumentation says somthing like this...: Note: This model requires that trust_remote_code=True be passed to the from_pretrained method. This is because we use a custom model architecture that is not yet part of the transformers package. Byt there is no indication what so ever where you would put this code either... and further - would not know where to look for the right place to put/run this code?? I think i found it? in webui.py: The model loaded but CUDA ran out of memory.... so this is probably not for me? |
Beta Was this translation helpful? Give feedback.
-
|
How many memory to load thee model? INFO:Loading mosaicml_mpt-7b-storywriter... |
Beta Was this translation helpful? Give feedback.
-
|
It takes about 14 gb to get running and around 20+ if you want to actually have a decent length content window like more than most models. |
Beta Was this translation helpful? Give feedback.
-
|
Is it possible to load this into CPU and max out the context length and have it generate correctly? |
Beta Was this translation helpful? Give feedback.
-
Not sure. I may try doing that tomorrow or seeing if it’s possible to do at least for me because I have 64gb of ddr5 ram. I’m guessing it would run super slow though, but if I get it working I’ll let you know. In theory I just have to change the code that loads the model around a bit, but I’m not gonna make any promises as it could require too much work for me to do in a few hours. Technically it shouldn’t be much work at all, but every time I think something isn’t going to be much work I end up spending 8 hours debugging. I don’t really have the time to screw around with it for more than a few hours. |
Beta Was this translation helpful? Give feedback.
-
|
Same issue here, but with the chat model. If it goes no where, no worries but want to follow along none the less. |
Beta Was this translation helpful? Give feedback.
-
|
amazing stuff! |
Beta Was this translation helpful? Give feedback.
-
|
I ran an update this morning and then attempted to run with the following slightly diff model (still mosaic that requires the remote flag) |
Beta Was this translation helpful? Give feedback.
-
|
A 4-bit version is now available: https://huggingface.co/OccamRazor/mpt-7b-storywriter-4bit-128g EDIT: ERROR: Can't determine model type from model name. Please specify it manually using --model_type argument The model_type in the config.json file is listed as "mpt". Is that something that would need to be added to the base webui toolkit? |
Beta Was this translation helpful? Give feedback.
-
|
safetensors, would this need an updated GPTQ-for-llama? |
Beta Was this translation helpful? Give feedback.
-
|
ValueError: Loading models\mpt-7b-storywriter-4bit-128g requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option trust_remote_code=True to remove this error. |
Beta Was this translation helpful? Give feedback.
-
|
I tried to run the Dose it need to be changed anywhere else? |
Beta Was this translation helpful? Give feedback.
-
|
Hi Mate, what worked for me to resolve this issue has been: |
Beta Was this translation helpful? Give feedback.
-
|
I am also trying out the 4bit version of this model, I managed to get it to load with these settings: But on inference I get this error: Final bit of information, Windows 10 with 32GB of RAM, GTX 1080 Ti with 11GB of VRAM, used the 1click installer maybe about a week ago to do a fresh install of oobabooga, installation went a bit smoother then the previous times, was attempting to get the reeducator vicuna 13b model running but wasn't able to, but I could use both the gozfarb pygmalion 7b 4bit model and able to use the gpt4-x alpaca 13b 4bit model, so the install is working fine. |
Beta Was this translation helpful? Give feedback.
-
|
as of now, mpt does not support multiple gpus. on a server with 2 t4s it gave the same error. |
Beta Was this translation helpful? Give feedback.
-
|
@Erika-wby I managed to run the 4bit version, although it wasn't worth it. Storywriter just throws in some random bs, so I believe that we're not there yet with the support. Better wait few more days instead of fighting with the unknown. |
Beta Was this translation helpful? Give feedback.
-
|
@Erika-wby I was able to get it working on multiple GPUs by making these 2 changes to
|
Beta Was this translation helpful? Give feedback.
-
|
Hello @acon96 did this change help it to stop hallucinating, and generating random words? |
Beta Was this translation helpful? Give feedback.
-
|
@ekolawole not from what I can tell. I've had success when I set Lastly, make sure you give the model a decent bit of context to go off of. The more context you give it the more it seems to prevent it from going off the rails. You can try these presets which have worked for me: |
Beta Was this translation helpful? Give feedback.
-
Has anyone gotten this to load into the UI?
There's a new 65K token storywriter that was just released...
https://huggingface.co/mosaicml/mpt-7b-storywriter
I added --trust-remote-code to webui.py
But still get an error:
ValueError: Loading models\mosaicml_mpt-7b-storywriter requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option trust_remote_code=True to remove this error.
Beta Was this translation helpful? Give feedback.
All reactions