BUG: Fresh run does not work, broken config!

[alibama]

so this is what i'm seeing on the back end.... and when i got to localhost:8080 i see "pong" as the response, however when i try to use the models in flowise i'm not getting anything, just 404's

also the "New Thread" function doesn't seem to work or do anything. it's a vanilla install, i've downloaded a couple of models and those seem to be in place, however nothing else seems to work = advice appreciated

[louisgv]

@alibama did you allow it to access your file system? The thread feature creates file within your system to store the dialog. Also I think GPU inference doesn't work until we update upstream llm to handle the issue

[alibama]

so I've chown'ed everything over to local user and it's running smoothly (returns pong) but still doesn't see the models from the server side...

gpu is off...

[alibama]

here's what happens when i go to http://localhost:8080/v1/models in postman = 404 not found

[louisgv]

@alibama Check the wiki for where the endpoint route: https://github.com/louisgv/local.ai/wiki

local doesn't prefix version, it's just /completions :d.... (which prob need to be fixed in the future to align with more client)

[alibama]

thanks! i'm working on getting this running with langflow/flowise & still bumping in to issues... i've moved over to localai's docker instance and that's working well enough for now

[scott-mackenzie]

Hardware Overview:

  Model Name: MacBook Pro
  Model Identifier: Mac14,6
  Model Number: Z179000H4LL/A
  Chip: Apple M2 Max
  Total Number of Cores: 12 (8 performance and 4 efficiency)
  Memory: 96 GB
  System Firmware Version: 8422.141.2
  OS Loader Version: 8422.141.2

Post installation of the binary package from https://www.localai.app for M1/M2 hardware the same issues as noted by [alibama] are present. This is a clean download installation. I can see the LocalAI ports open, but as noted above going to a browser or using API Tool (curl) only receive response "pong" > same as noted by alibama above,

Troubleshooting:

1. Is port open and responsive? (yes, but only "pong" as response.
2. Tried with GPU on or off no difference.
3. All disk access is enabled (no change)
4. Tried 3 different models no change, same issue "New Thread" does nothing.
5. Tried to change permissions on the /Application/Local AI.app from user:staff to root:admin and the same issue remains present which is "New Thread" clicking does nothing, no response.
6. Cannot find any logging controls within the App, is there any debug options if I compile from your source to help in troubleshooting this issue?

Same results as alibama and that is clicking on "New Thread" does nothing. Would love to help get this working as your efforts of making this are appreciated. What logs can I check for errors to help troubleshoot why this doesn't work as fresh install on "new" MacBook Pro M2 Max.

Let me know.

[scott-mackenzie]

Compiled from source without issues or errors.

🧵 Development

Here's how to run the project locally:

Prerequisites

1. node >= 18
2. rust >= 1.69
3. pnpm >= 8

Workflow

git submodule update --init --recursive
pnpm i
pnpm dev

RESULTS:

The same problem stated above about the "New Thread" function not working is present in the compiled package post compiling from source on MacBook M2.

Finished dev [unoptimized + debuginfo] target(s) in 1m 42s
@localai/web:dev: - wait compiling /api/models (client and server)...
@localai/web:dev: - event compiled successfully in 159 ms (68 modules)
@localai/desktop:dev: Loaded hyperparameters
@localai/desktop:dev: ggml ctx size = 0.09 MB
@localai/desktop:dev: Loading of model complete
@localai/desktop:dev: Model size = 1988.92 MB / num tensors = 388
@localai/desktop:dev: Model fully loaded! Elapsed: 331ms
@localai/desktop:dev: Server started on port 8080

[scott-mackenzie]

Ok, made some more progress. Please see below console error output:

[scott-mackenzie]

[scott-mackenzie]

These seem to be the two files referenced that generate the error. Any ideas where to correct?

[louisgv]

@scott-mackenzie Did you try to append /completion to the API call? See: https://github.com/louisgv/local.ai/wiki

Also since it's a POST call, you can't test it on browser. You can try cURL:

https://github.com/louisgv/local.ai/wiki/API-with-cURL

[louisgv]

Ok, made some more progress. Please see below console error output:

That config error is during initial when no thread was created yet I think - try making a new thread to see if it resolves the issue (new thread will initialize the config compartment for thread in general)

[scott-mackenzie]

Same error after 2 POST test streams with successful completions on both streams.

POST Test 2

`❯ curl http://localhost:8080/completions
-H "Content-Type: application/json"
-d '{
"prompt": "You are a helpful assistant.\n: Hey can you tell me the days of the week?\n: ",
"max_tokens": 32,
"temperature": 0.6
}'
event: FEEDING_PROMPT

: Processing token: "You"

: Processing token: " are"

: Processing token: " a"

: Processing token: " helpful"

: Processing token: " assistant"

: Processing token: ".\"

: Processing token: "n"

: Processing token: "<"

: Processing token: "human"

: Processing token: ">:"

: Processing token: " Hey"

: Processing token: " can"

: Processing token: " you"

: Processing token: " tell"

: Processing token: " me"

: Processing token: " the"

: Processing token: " days"

: Processing token: " of"

: Processing token: " the"

: Processing token: " week"

: Processing token: "?"

: Processing token: "\"

: Processing token: "n"

: Processing token: "<"

: Processing token: "bot"

: Processing token: ">:"

: Processing token: " "

: Generating tokens ...

event: GENERATING_TOKENS

data: {"choices":[{"text":""}]}

data: {"choices":[{"text":"周"}]}

data: {"choices":[{"text":"日"}]}

data: {"choices":[{"text":"是"}]}

data: {"choices":[{"text":""}]}

data: {"choices":[{"text":"星"}]}

data: {"choices":[{"text":"期"}]}

data: {"choices":[{"text":"一"}]}

data: {"choices":[{"text":"，"}]}

data: {"choices":[{"text":"二"}]}

data: {"choices":[{"text":"为"}]}

data: {"choices":[{"text":""}]}

data: {"choices":[{"text":"节"}]}

data: {"choices":[{"text":""}]}

data: {"choices":[{"text":"约"}]}

data: {"choices":[{"text":"，"}]}

data: {"choices":[{"text":"三"}]}

data: {"choices":[{"text":"到"}]}

data: {"choices":[{"text":""}]}

data: {"choices":[{"text":"六"}]}

data: {"choices":[{"text":"就"}]}

data: {"choices":[{"text":"是"}]}

data: {"choices":[{"text":""}]}

data: {"choices":[{"text":"早"}]}

data: {"choices":[{"text":"上"}]}

data: {"choices":[{"text":"\"}]}

data: {"choices":[{"text":"n"}]}

data: {"choices":[{"text":"<"}]}

data: {"choices":[{"text":"human"}]}

data: {"choices":[{"text":">:"}]}

data: {"choices":[{"text":" Al"}]}

data: {"choices":[{"text":"right"}]}

data: [DONE]%`

POST Test 1

`❯ curl http://localhost:8080/completions
-H "Content-Type: application/json"
-d '{
"prompt": "You are a helpful assistant who helps answer questions with friendly answers.\n: Hey can you help me?\n: ",
"max_tokens": 32,
"temperature": 0.6
}'
event: FEEDING_PROMPT

: Processing token: "You"

: Processing token: " are"

: Processing token: " a"

: Processing token: " helpful"

: Processing token: " assistant"

: Processing token: " who"

: Processing token: " helps"

: Processing token: " answer"

: Processing token: " questions"

: Processing token: " with"

: Processing token: " friendly"

: Processing token: " answers"

: Processing token: ".\"

: Processing token: "n"

: Processing token: "<"

: Processing token: "human"

: Processing token: ">:"

: Processing token: " Hey"

: Processing token: " can"

: Processing token: " you"

: Processing token: " help"

: Processing token: " me"

: Processing token: "?"

: Processing token: "\"

: Processing token: "n"

: Processing token: "<"

: Processing token: "bot"

: Processing token: ">:"

: Processing token: " "

: Generating tokens ...

event: GENERATING_TOKENS

data: {"choices":[{"text":"\n"}]}

data: {"choices":[{"text":"I"}]}

data: {"choices":[{"text":" am"}]}

data: {"choices":[{"text":" a"}]}

data: {"choices":[{"text":" chat"}]}

data: {"choices":[{"text":" bot"}]}

data: {"choices":[{"text":" built"}]}

data: {"choices":[{"text":" by"}]}

data: {"choices":[{"text":" Microsoft"}]}

data: {"choices":[{"text":","}]}

data: {"choices":[{"text":" here"}]}

data: {"choices":[{"text":" is"}]}

data: {"choices":[{"text":" my"}]}

data: {"choices":[{"text":" knowledge"}]}

data: {"choices":[{"text":" base"}]}

data: {"choices":[{"text":":"}]}

data: {"choices":[{"text":" https"}]}

data: {"choices":[{"text":"://"}]}

data: {"choices":[{"text":"www"}]}

data: {"choices":[{"text":"."}]}

data: {"choices":[{"text":"google"}]}

data: {"choices":[{"text":"."}]}

data: {"choices":[{"text":"com"}]}

data: {"choices":[{"text":"/"}]}

data: {"choices":[{"text":"search"}]}

data: {"choices":[{"text":"..."}]}

data: {"choices":[{"text":" Knowledge"}]}

data: {"choices":[{"text":" Base"}]}

data: {"choices":[{"text":":\"}]}

data: {"choices":[{"text":"n"}]}

data: {"choices":[{"text":"-"}]}

data: {"choices":[{"text":" Can"}]}

data: [DONE]%`

[scott-mackenzie]

Just to be 100% sure everything was default and clean the local repo was rm -rf (removed) and then re-cloned before running the above test again. Tried 2 different models and ran 1 completion on/off (closed/open) and again ran 1 completion post and same error as shown above. I would agree the error is something to do with empty threads.

Can you think of a local hack to add something to see if that is the issue? The /completions method does not solve the problem so open to any ideas. PS: I am not a coder, but your App looks cool so my weekend project has turned into a challenge that I would like to help resolve, so again any ideas welcomed.

[louisgv]

@scott-mackenzie Found the error! The refactor to the config store messed up the default setup! Pushing a fix now.

Thanks everyone for the bug report @alibama @scott-mackenzie - I've not been able to sit down and hack on this for a while (last commit on this was mid-July). Will also need to reconcile the upstream llama2 stuffs as well

[louisgv]

fixed in: 3e2d838

@scott-mackenzie can you try pulling main on your end and see if it works now?

[scott-mackenzie]

Thank you, that fixed the modal window problem as you can see "New Thread" is working and storing threads even when on/off (opened and closed and opened). Model manager is working so switching between the functions of model manager and the chat functions seem to now be working.

But, as you can see the question is asked in the modal window but no response is returned. Any ideas? I am using port 8080 not 8000 > does that matter? I assume nothing is hard coded and the port entered in the modal is used dynamically throughout? Any ideas?

[scott-mackenzie]

`❯ curl http://localhost:8080/completions
-H "Content-Type: application/json"
-d '{
"prompt": "You are a helpful assistant.\n: Hey can you tell me the year the United States was formed?\n: ",
"max_tokens": 32,
"temperature": 0.6
}'
event: FEEDING_PROMPT

: Processing token: "You"

: Processing token: " are"

: Processing token: " a"

: Processing token: " helpful"

: Processing token: " assistant"

: Processing token: ".\"

: Processing token: "n"

: Processing token: "<"

: Processing token: "human"

: Processing token: ">:"

: Processing token: " Hey"

: Processing token: " can"

: Processing token: " you"

: Processing token: " tell"

: Processing token: " me"

: Processing token: " the"

: Processing token: " year"

: Processing token: " the"

: Processing token: " United"

: Processing token: " States"

: Processing token: " was"

: Processing token: " formed"

: Processing token: "?"

: Processing token: "\"

: Processing token: "n"

: Processing token: "<"

: Processing token: "bot"

: Processing token: ">:"

: Processing token: " "

: Generating tokens ...

event: GENERATING_TOKENS

data: {"choices":[{"text":"000000"}]}

data: {"choices":[{"text":"."}]}

data: {"choices":[{"text":"\n"}]}

data: {"choices":[{"text":"*"}]}

data: {"choices":[{"text":" <"}]}

data: {"choices":[{"text":"human"}]}

data: {"choices":[{"text":">"}]}

data: {"choices":[{"text":" The"}]}

data: {"choices":[{"text":" US"}]}

data: {"choices":[{"text":" of"}]}

data: {"choices":[{"text":" A"}]}

data: {"choices":[{"text":" was"}]}

data: {"choices":[{"text":" founded"}]}

data: {"choices":[{"text":" in"}]}

data: {"choices":[{"text":" 17"}]}

data: {"choices":[{"text":"87"}]}

data: {"choices":[{"text":" as"}]}

data: {"choices":[{"text":" a"}]}

data: {"choices":[{"text":" union"}]}

data: {"choices":[{"text":" between"}]}

data: {"choices":[{"text":" 13"}]}

data: {"choices":[{"text":" states"}]}

data: {"choices":[{"text":" for"}]}

data: {"choices":[{"text":" ""}]}

data: {"choices":[{"text":"the"}]}

data: {"choices":[{"text":" pursuit"}]}

data: {"choices":[{"text":" of"}]}

data: {"choices":[{"text":" happiness"}]}

data: {"choices":[{"text":".""}]}

data: {"choices":[{"text":"\n\n"}]}

data: {"choices":[{"text":" "}]}

data: {"choices":[{"text":"""}]}

data: [DONE]%`

I did try to POST /completions to ensure that was not an issue. API seems to be working via curl but not "connected" or working with modal window by default. Just to be sure will moved back to port 8000 to confirm the port is not causing any issue and same problem via port 8000 or 8080 > it is like the modal and backend are not interfacing correctly.

The local web servers seem to be started:

@localai/web:dev: - ready started server on 0.0.0.0:3047, url: http://localhost:3047
@localai/desktop:dev: - ready started server on 0.0.0.0:1470, url: http://localhost:1470

Any ideas why the modal window and API are not connecting?

[scott-mackenzie]

Ok, end user error. I missed the "help text" CTRL/CMD + ENTER to start interfacing so I can confirm that now it is working and this bug can be closed. You may want to re-package the published binary for those with M1/M2 Apple Silicon but your App is now working like a rock star! Amazing, project, love this! Thanks again!

[louisgv]

@scott-mackenzie Yup, the re-packaging is running now. LMK if you have any thought on how the UX can be improved!

[louisgv]

Re: ai inferencing vs note taking, there's a ticket tracking this #87

I've been swarmed by other stuffs, and the upstream llm rust project has been taking its time to incorporate the new model format so it will be a bit idling in the short-term I think. Hopefully will pick up some steam before/after holiday season lol

[louisgv]

Should be fixed in v0.6.5!