LocalAI v2.28 does not use the ENV file in docker.

[michieal]

LocalAI version:

2:47AM INF LocalAI version: v2.28.0 (56f44d4)
Docker, Radeon passed through. HipBlas/ROCm. Quay latest. as of ~ a week or so ago.

Environment, CPU architecture, OS, and Version:

Describe the bug

LocalAI seems to ignore the ENV settings. In this case it ignores the log level. (it ignores other ones too... I can put them in the comments or make separate issues for them.)

ENV file:

Enable debug mode

LOCALAI_LOG_LEVEL=info

DEBUG OUTPUT:
2:47AM DBG [startup] resolved local model: /aio/gpu-8g/embeddings.yaml
2:47AM DBG [startup] resolved local model: /aio/gpu-8g/rerank.yaml
2:47AM DBG [startup] resolved local model: /aio/gpu-8g/text-to-speech.yaml
2:47AM DBG [startup] resolved local model: /aio/gpu-8g/image-gen.yaml
2:47AM DBG [startup] resolved local model: /aio/gpu-8g/text-to-text.yaml
2:47AM DBG [startup] resolved local model: /aio/gpu-8g/speech-to-text.yaml
2:47AM DBG [startup] resolved local model: /aio/gpu-8g/vad.yaml
2:47AM DBG [startup] resolved local model: /aio/gpu-8g/vision.yaml
2:47AM DBG Model file loaded: granite-embedding-107m-multilingual-f16.gguf architecture=bert bosTokenID=0 eosTokenID=2 modelName="Granite Embedding 107m Multilingual"
2:47AM DBG guessDefaultsFromFile: family not identified
...

To Reproduce

I guess install the hipblas all in one container. try setting up an env file.

Expected behavior

I expect that LocalAI adheres to its documentation and to use the env file like it did last version. Not sure what changed, but I would think that if it was working (in v2.27), then it shouldn't be that difficult.

Logs

@@@@@
CPU info:
model name : QEMU Virtual CPU version 2.5+
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c hypervisor lahf_lm abm cpuid_fault pti bmi1 avx2 bmi2
CPU: AVX found OK
CPU: AVX2 found OK
CPU: no AVX512 found
@@@@@
2:47AM INF env file found, loading environment variables from file envFile=.env
2:47AM DBG Setting logging to debug
2:47AM INF Starting LocalAI using 32 threads, with models path: /build/models
2:47AM INF LocalAI version: v2.28.0 (56f44d4)

Additional context

Additional context: I am sick and tired of fighting with it to the point that I barely have the energy to make a freaking issue. I took a minute to look at the issues... and it seems like when one makes an issue, they are shouting into the void. Not... a good look.

[michieal]

I mean... I guess it's great to have the docker logs spammed so that they are 200+mb of text.
note that "kiri1" is Mistral-Nemo-Instruct-2407-Q4_K_M.gguf loaded using the llama-cpp backend, has a context window of 12k, and if I am lucky it does a full 1228 tokens counting prompt, context history, output, etc. 🤷‍♂

Let's you see that you are wasting your time:

3:02AM DBG Sending chunk: {"created":1745982085,"object":"chat.completion.chunk","id":"1e4f836d-57c6-4ab0-b1e0-3150e2485456","model":"kiri1","choices":[{"index":0,"finish_reason":"","delta":{"content":"o"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

3:02AM DBG Sending chunk: {"created":1745982085,"object":"chat.completion.chunk","id":"1e4f836d-57c6-4ab0-b1e0-3150e2485456","model":"kiri1","choices":[{"index":0,"finish_reason":"","delta":{"content":"u"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

3:02AM DBG Sending chunk: {"created":1745982085,"object":"chat.completion.chunk","id":"1e4f836d-57c6-4ab0-b1e0-3150e2485456","model":"kiri1","choices":[{"index":0,"finish_reason":"","delta":{"content":"t"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

3:02AM DBG Sending chunk: {"created":1745982085,"object":"chat.completion.chunk","id":"1e4f836d-57c6-4ab0-b1e0-3150e2485456","model":"kiri1","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

3:02AM DBG Sending chunk: {"created":1745982085,"object":"chat.completion.chunk","id":"1e4f836d-57c6-4ab0-b1e0-3150e2485456","model":"kiri1","choices":[{"index":0,"finish_reason":"","delta":{"content":"i"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

3:02AM DBG Sending chunk: {"created":1745982085,"object":"chat.completion.chunk","id":"1e4f836d-57c6-4ab0-b1e0-3150e2485456","model":"kiri1","choices":[{"index":0,"finish_reason":"","delta":{"content":"t"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

3:02AM DBG Sending chunk: {"created":1745982085,"object":"chat.completion.chunk","id":"1e4f836d-57c6-4ab0-b1e0-3150e2485456","model":"kiri1","choices":[{"index":0,"finish_reason":"","delta":{"content":"."}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

3:02AM DBG Sending chunk: {"created":1745982085,"object":"chat.completion.chunk","id":"1e4f836d-57c6-4ab0-b1e0-3150e2485456","model":"kiri1","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

3:02AM DBG Sending chunk: {"created":1745982085,"object":"chat.completion.chunk","id":"1e4f836d-57c6-4ab0-b1e0-3150e2485456","model":"kiri1","choices":[{"index":0,"finish_reason":"","delta":{"content":"H"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

3:02AM DBG Sending chunk: {"created":1745982085,"object":"chat.completion.chunk","id":"1e4f836d-57c6-4ab0-b1e0-3150e2485456","model":"kiri1","choices":[{"index":0,"finish_reason":"","delta":{"content":"o"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

3:02AM DBG Sending chunk: {"created":1745982085,"object":"chat.completion.chunk","id":"1e4f836d-57c6-4ab0-b1e0-3150e2485456","model":"kiri1","choices":[{"index":0,"finish_reason":"","delta":{"content":"w"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

[mudler]

I'd say that the tone you are opening issues is not really welcome here, and I invite you to re-read the way you are approaching to people. this is a free, open source project where anyone can contribute. Nobody is payed to work for LocalAI - this is not a company.

That being said, there were no changes regarding reading environment files in the latest release - and from your ticket it's not really clear what you did, and what you expected to happen.

[michieal]

I'd say that the tone you are opening issues is not really welcome here, and I invite you to re-read the way you are approaching to people. this is a free, open source project where anyone can contribute. Nobody is payed to work for LocalAI - this is not a company.

That being said, there were no changes regarding reading environment files in the latest release - and from your ticket it's not really clear what you did, and what you expected to happen.

You know, I get that. I do open-source projects a lot, so yeah, I am NOT approaching this as "fix it now" nor as "you're paid"... My Tone is frustration and defeat. point blank.

And, it doesn't matter what I change in the .env file - it doesn't change after restarting the docker. I've timed things, to be able to tell how long, say, the watchdog idle timeout actually waits. I have it set to 1m... and it still take between 5-8 minutes to kill the back end.

I try setting things to NOT be parallel requests, because I dislike a context window of 819 (when the context window in the model's yaml is 8192). and nothing changes. If I increase the yaml's context window to say... 12k the debug logs shows that it will go up to a total context of 1228 (counting a truncated prompt, history, and maybe 60 tokens of output. And, this is "new" to 2.28. before, in 2.27, It was doing sequential requests (until I changed it to be parallel) but even in parallel -- it didn't cut off the responses.
At this moment, I am lucky if it even gives a response.

if I set the local ai log level to info (which, according to the parameters in local-ai run --help is valid) I still get all of the debug level output. it floods the docker logs, which I have been trying to use to watch the watchdog... I had to set up log rotation because the logs are huge. So yeah - it doesn't matter what that is changed to, as it always does the same. dang. thing.

And, unlike most people, I am fairly determined to make things work, I don't give up at the first hurdle and scream "It's too hard!"
Can you understand why, after a couple of weeks of fighting with this daily, that I am frustrated?

what I expect to happen is that it reads in the .env file (I doubled checked inside of the container) and act upon those settings.
if you need something from me to help you see what's going on, tell me -- I'll put the information in here. I want my stuff to work.

(EDIT:)
@mudler

[Fr0d0Beutl1n]

I assume that you're using docker-compose like this:

    env_file:
      - env

This means, you're using the env_file keyword, which is a feature of docker-compose, not of LocalAI.

docker-compose does not reload the config, if you use restart. You need to use down and up, or use up --force-recreate in order for the config to be reloaded.

[michieal]

I'm using:

      env_file:
          - .env

And, restarting the container before has caused it to update... just... not with the new version.
I will try your suggestion and see what happens.

[michieal]

@Fr0d0Beutl1n thank you that worked. Apparently, AnythingLLM moves the .env fil inside of the container so that a simple restart works to change the settings. So, I confused how it works with how LocalAI should work... My misunderstanding.

@mudler Would you be terribly irritated if I made a feature request for the implementation of this style, so that the "env" can be updated on restart?