Cloudflared server not starting

[mnt4]

Describe the bug

Adding --public-api to the flags causes a traceback instead of starting a cloudflared server. flask-clouflared is installed separately, despite not being included in requirements.txt

Is there an existing issue for this?

• I have searched the existing issues

Reproduction

1. Install flask-cloudflared
2. Run python server.py --share --model_type llama --no-stream --api --public-api --wbits 4 --groupsize 128
3. Get a Traceback error

Screenshot

No response

Logs

Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/content/drive/MyDrive/text-generation-webui/extensions/api/util.py", line 69, in _start_cloudflared
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    raise Exception('Could not start cloudflared.')
Exception: Could not start cloudflared.
    self.run()
  File "/usr/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/content/drive/MyDrive/text-generation-webui/extensions/api/util.py", line 69, in _start_cloudflared
    raise Exception('Could not start cloudflared.')
Exception: Could not start cloudflared.

System Info

Google Colab

[TruthSearchers]

Same issue

[oobabooga]

The API extension hides the real error, which is

OSError: [Errno 8] Exec format error: '/tmp/cloudflared-linux-amd64'

It looks like cloudfare messed up their reverse proxy executable. It should fix itself eventually

[mnt4]

I made a fork of flask_cloudflared that fixes the issue temporarily
It can be used via pip install git+https://github.com/mnt4/flask-cloudflared

[TruthSearchers]

It's working fine now bro thanks ☺️

[laughinggaschambers]

I'm getting the same issue

I made a fork of flask_cloudflared that fixes the issue temporarily It can be used via pip install git+https://github.com/mnt4/flask-cloudflared

I'm using the one click install version, running the command doesn't fix it

[juliankolbe]

The real issue why this bug is happening is that both the blocking api and then streaming api start the _run_cloudflared more or less at the same time, and if this file '/tmp/cloudflared-linux-amd64' is not downloaded and ready fast enough, they collide as the file is still downloading. This is only a working theory, but on my host machine it had no problem, but as soon as this was running in docker it did.

TLDR: I changed the extensions/api/script.py and added a time.sleep(5) in between calls, fixed it for me.

    blocking_api.start_server(BLOCKING_PORT, share=shared.args.public_api)
    time.sleep(5)
    streaming_api.start_server(STREAMING_PORT, share=shared.args.public_api)

[dje3392]

I've tried adding time.sleep and installing the flask cloudflared fix with no success :(

[pato-pan]

for some users, you simply have to give execute permissions to /tmp/cloudflared-linux-amd64

like this
chmod +x /tmp/cloudflared-linux-amd64

This should be clear from the logs

PermissionError: [Errno 13] Permission denied: '/tmp/cloudflared-linux-amd64'

but that's easy to miss when you are also facing other errors. In my case I didn't have cloudflared installed on my os (Installing it on conda won't work, it has to be on the os) and I got the permission denied error, so when I installed it and the logs looked different I did not think of checking that file again, I assumed installing cloudflare would fix it and moved on to the next error I was facing. edit: also, make sure you have flask-cloudflared installed before trying anything suggested on this issue. pip install flask-cloudflared. Don't expect it to be on requirements.txt because it's not required.

[ChenMorSays]

Still doesnt work for me, any updates?

[adityasingh8010]

The real issue why this bug is happening is that both the blocking api and then streaming api start the _run_cloudflared more or less at the same time, and if this file '/tmp/cloudflared-linux-amd64' is not downloaded and ready fast enough, they collide as the file is still downloading. This is only a working theory, but on my host machine it had no problem, but as soon as this was running in docker it did.

TLDR: I changed the extensions/api/script.py and added a time.sleep(5) in between calls, fixed it for me.

    blocking_api.start_server(BLOCKING_PORT, share=shared.args.public_api)
    time.sleep(5)
    streaming_api.start_server(STREAMING_PORT, share=shared.args.public_api)

This hypothesis might be correct because sometimes it would run for me and sometimes it would crash. But adding in the 5 second sleep fixed the issue for me

[kushshrivastava]

I managed to run the API on one colab notebook and use it in another colab notebook using these steps

In first colab notebook
Clone text-generation-webui and install all requirements

!pip install flask-cloudflared
%cd /content/text-generation-webui
!python3.10 server.py --loader ExLlama --model /content/text-generation-webui/models/Llama-2-13b-Chat-GPTQ --public-api

this will return a url something like this
Starting non-streaming server at public url https://ABC-flag.trycloudflare.com/api

In second colab notebook
Go to api-examples/api-example.py
replace the above url in place of HOST

HOST = 'ABC-flag.trycloudflare.com'
URI = f'https://{HOST}/api/v1/generate'

Then run the file
!python3.10 api-example.py

[github-actions]

This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.