Container didn't respond to HTTP pings on port: 8000

[andresravinet]

Discussed in #285

^{Originally posted by andresravinet October 18, 2023}
Can anyone help me? I'm getting an Application Error when browsing to the web app. And the below is what is showing up in the log.
Container infoasst-web-6cq0e_0_17db8b46 didn't respond to HTTP pings on port: 8000, failing site start. See container logs for debugging.

[RandomDrew]

Hi, I've encountered the same issue I believe - does your web app report
:( Application Error
If you are the application administrator, you can access the diagnostic resources."

I looked through the logs and found this error being throw:

2023-10-18T23:03:41.225856898Z azure.core.exceptions.ClientAuthenticationError: DefaultAzureCredential failed to retrieve a token from the included credentials.
2023-10-18T23:03:41.225861298Z Attempted credentials:
2023-10-18T23:03:41.225865198Z EnvironmentCredential: Authentication failed: Unable to get authority configuration for https://login.microsoftonline.com/199679d1-f566-46c5-abdc-cd300e2e0dc9. Authority would typically be in a format of https://login.microsoftonline.com/your_tenant Also please double check your tenant name or GUID is correct.
2023-10-18T23:03:41.225869598Z To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.

and then I checked my Tenant ID, Subscription ID and the Secret key, all are correct. I also tried creating a new secret key and updating the config but the result was the same.

[andresravinet]

No, my error looks more like a port issue.

[RandomDrew]

OK. In my instance the authentication fails and then a few seconds later the ping error is reported.

2023-10-18T23:03:42.877Z INFO - Container infoasst-web-umvck_0_abb22b68_middleware for site infoasst-web-umvck initialized successfully and is ready to serve requests.
2023-10-18T23:03:42.941Z ERROR - Container infoasst-web-umvck_0_abb22b68 didn't respond to HTTP pings on port: 8000, failing site start. See container logs for debugging.

[dayland]

@andresravinet , Can you please check the following and post your logs here please.

1. Confirm the App Service Plan (named like infoasst-asp-xxxxx) configuration for your webapp. It should be deployed as 3 x B2 from main. We have changed this to 3 x S1 in the next release.
2. If you are still getting errors, please restart your App Service and then go to the "Log Stream" feature of your App Service and collect logs on startup. Most issues occur on startup of the container and the error only shows once in the log, then you get the above mentioned log error over and over again.

[mikeimmanuel]

Hi, I'm receiving the same error for the Delta release, I'm receiving the error: App Container Failed to Start, please see logs below:

Container infoasst-web-xmlub_0_8673eeee couldn't be started: Logs = 2023-11-16T01:26:46.775410689Z _____
2023-11-16T01:26:46.775450189Z / _ \ __________ _________ ____
2023-11-16T01:26:46.775454289Z / /\ \__ / | _ __ _/ __ \
2023-11-16T01:26:46.775457389Z / | / /| | /| | /\ /
2023-11-16T01:26:46.775460289Z _|__ /_____ _/ || __ >
2023-11-16T01:26:46.775463490Z / / /
2023-11-16T01:26:46.775466490Z A P P S E R V I C E O N L I N U X
2023-11-16T01:26:46.775469390Z
2023-11-16T01:26:46.775471990Z Documentation: http://aka.ms/webapp-linux
2023-11-16T01:26:46.775474890Z Python 3.10.12
2023-11-16T01:26:46.775477590Z Note: Any data outside '/home' is not persisted
2023-11-16T01:26:51.477492577Z Starting OpenBSD Secure Shell server: sshd.
2023-11-16T01:26:51.866919710Z App Command Line not configured, will attempt auto-detect
2023-11-16T01:26:53.210021618Z Starting periodic command scheduler: cron.
2023-11-16T01:26:53.210074619Z Launching oryx with: create-script -appPath /home/site/wwwroot -output /opt/startup/startup.sh -virtualEnvName antenv -defaultApp /opt/defaultsite
2023-11-16T01:26:53.510484202Z Found build manifest file at '/home/site/wwwroot/oryx-manifest.toml'. Deserializing it...
2023-11-16T01:26:53.520987869Z Build Operation ID: 39655595cb32b8e4
2023-11-16T01:26:53.531939044Z Oryx Version: 0.2.20230707.1, Commit: 0bd28e69919b5e8beba451e8677e3345f0be8361, ReleaseTagName: 20230707.1
2023-11-16T01:26:53.541451795Z Output is compressed. Extracting it...
2023-11-16T01:26:53.560351296Z Extracting '/home/site/wwwroot/output.tar.gz' to directory '/tmp/8dbe641779eb75a'...
2023-11-16T01:27:16.367939980Z App path is set to '/tmp/8dbe641779eb75a'
2023-11-16T01:27:16.419138691Z Detected an app based on Flask
2023-11-16T01:27:16.419169791Z Generating gunicorn command for 'app:app'
2023-11-16T01:27:16.469730393Z Writing output script to '/opt/startup/startup.sh'
2023-11-16T01:27:16.874434715Z Using packages from virtual environment antenv located at /tmp/8dbe641779eb75a/antenv.
2023-11-16T01:27:16.874464915Z Updated PYTHONPATH to '/opt/startup/app_logs:/tmp/8dbe641779eb75a/antenv/lib/python3.10/site-packages'
2023-11-16T01:27:24.974024665Z [2023-11-16 01:27:24 +0000] [74] [INFO] Starting gunicorn 20.1.0
2023-11-16T01:27:25.024370576Z [2023-11-16 01:27:25 +0000] [74] [INFO] Listening at: http://0.0.0.0:8000 (74)
2023-11-16T01:27:25.025378392Z [2023-11-16 01:27:25 +0000] [74] [INFO] Using worker: sync
2023-11-16T01:27:25.114986936Z [2023-11-16 01:27:25 +0000] [77] [INFO] Booting worker with pid: 77
2023-11-16T01:27:34.788820147Z Region configured (None) != region detected ('australiaeast')
2023-11-16T01:27:35.148042236Z [2023-11-16 01:27:35 +0000] [77] [ERROR] Exception in worker process
2023-11-16T01:27:35.148098636Z Traceback (most recent call last):
2023-11-16T01:27:35.148113637Z File "/opt/python/3.10.12/lib/python3.10/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
2023-11-16T01:27:35.148117437Z worker.init_process()
2023-11-16T01:27:35.148120637Z File "/opt/python/3.10.12/lib/python3.10/site-packages/gunicorn/workers/base.py", line 134, in init_process
2023-11-16T01:27:35.148124037Z self.load_wsgi()
2023-11-16T01:27:35.148176338Z File "/opt/python/3.10.12/lib/python3.10/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
2023-11-16T01:27:35.148180838Z self.wsgi = self.app.wsgi()
2023-11-16T01:27:35.148184338Z File "/opt/python/3.10.12/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi
2023-11-16T01:27:35.148187738Z self.callable = self.load()
2023-11-16T01:27:35.148190938Z File "/opt/python/3.10.12/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
2023-11-16T01:27:35.148194138Z return self.load_wsgiapp()
2023-11-16T01:27:35.148197638Z File "/opt/python/3.10.12/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
2023-11-16T01:27:35.148201038Z return util.import_app(self.app_uri)
2023-11-16T01:27:35.148204238Z File "/opt/python/3.10.12/lib/python3.10/site-packages/gunicorn/util.py", line 359, in import_app
2023-11-16T01:27:35.148207638Z mod = impo
=======[LONG LOG TRUNCATED BY DETECTOR]======
Container infoasst-web-xmlub_0_8673eeee didn't respond to HTTP pings on port: 8000, failing site start. See container logs for debugging.

---

I've also noticed the following error:

2023-11-16T02:57:24.321084250Z Updated PYTHONPATH to '/opt/startup/app_logs:/tmp/8dbe641779eb75a/antenv/lib/python3.10/site-packages'
2023-11-16T02:57:26.495346577Z [2023-11-16 02:57:26 +0000] [73] [INFO] Starting gunicorn 20.1.0
2023-11-16T02:57:26.519494036Z [2023-11-16 02:57:26 +0000] [73] [INFO] Listening at: http://0.0.0.0:8000 (73)
2023-11-16T02:57:26.519898639Z [2023-11-16 02:57:26 +0000] [73] [INFO] Using worker: sync
2023-11-16T02:57:26.532724923Z [2023-11-16 02:57:26 +0000] [76] [INFO] Booting worker with pid: 76
2023-11-16T02:57:33.648397766Z Region configured (None) != region detected ('australiaeast')
2023-11-16T02:57:34.198996732Z [2023-11-16 02:57:34 +0000] [76] [ERROR] Exception in worker process
2023-11-16T02:57:34.202803056Z Traceback (most recent call last):
2023-11-16T02:57:34.205056670Z File "/opt/python/3.10.12/lib/python3.10/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker

2023-11-16T02:57:34.205065170Z worker.init_process()
2023-11-16T02:57:34.205068070Z File "/opt/python/3.10.12/lib/python3.10/site-packages/gunicorn/workers/base.py", line 134, in init_process

2023-11-16T02:57:34.205070870Z self.load_wsgi()
2023-11-16T02:57:34.205073470Z File "/opt/python/3.10.12/lib/python3.10/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi

2023-11-16T02:57:34.205076270Z self.wsgi = self.app.wsgi()
2023-11-16T02:57:34.205079070Z File "/opt/python/3.10.12/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi

2023-11-16T02:57:34.205081770Z self.callable = self.load()
2023-11-16T02:57:34.205084370Z File "/opt/python/3.10.12/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 58, in load

2023-11-16T02:57:34.205086970Z return self.load_wsgiapp()
2023-11-16T02:57:34.205089670Z File "/opt/python/3.10.12/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp

2023-11-16T02:57:34.205092470Z return util.import_app(self.app_uri)
2023-11-16T02:57:34.205095070Z File "/opt/python/3.10.12/lib/python3.10/site-packages/gunicorn/util.py", line 359, in import_app

2023-11-16T02:57:34.205107070Z mod = importlib.import_module(module)
2023-11-16T02:57:34.205110170Z File "/opt/python/3.10.12/lib/python3.10/importlib/init.py", line 126, in import_module

2023-11-16T02:57:34.205112870Z return _bootstrap._gcd_import(name[level:], package, level)
2023-11-16T02:57:34.205115570Z File "", line 1050, in _gcd_import
2023-11-16T02:57:34.205118570Z File "", line 1027, in _find_and_load
2023-11-16T02:57:34.205121470Z File "", line 1006, in _find_and_load_unlocked
2023-11-16T02:57:34.205124170Z File "", line 688, in _load_unlocked
2023-11-16T02:57:34.205126870Z File "", line 883, in exec_module
2023-11-16T02:57:34.205129570Z File "", line 241, in _call_with_frames_removed
2023-11-16T02:57:34.205132470Z File "/tmp/8dbe641779eb75a/app.py", line 130, in
2023-11-16T02:57:34.205135270Z deployment = openai_mgmt_client.deployments.get(
2023-11-16T02:57:34.205138070Z File "/tmp/8dbe641779eb75a/antenv/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
2023-11-16T02:57:34.205140870Z return func(*args, **kwargs)
**2023-11-16T02:57:34.205145270Z File "/tmp/8dbe641779eb75a/antenv/lib/python3.10/site-packages/azure/mgmt/cognitiveservices/operations/_deployments_operations.py", line 349, in get
2023-11-16T02:57:34.205148270Z map_error(status_code=response.status_code, response=response, error_map=error_map)

2023-11-16T02:57:34.205155170Z File "/tmp/8dbe641779eb75a/antenv/lib/python3.10/site-packages/azure/core/exceptions.py", line 165, in map_error

2023-11-16T02:57:34.205158170Z raise error**

2023-11-16T02:57:34.205160870Z azure.core.exceptions.ResourceNotFoundError: (NotFound) Specified resource '/subscriptions/3d6eaea2-84d4-4caa-ad14-3af7da82a431/resourceGroups/infoasst-icazcolde04/providers/Microsoft.CognitiveServices/accounts/infoasst-aoai-xmlub/deployments/chat' cannot be found.

2023-11-16T02:57:34.205164770Z Code: NotFound
2023-11-16T02:57:34.205167370Z Message: Specified resource '/subscriptions/3d6eaea2-84d4-4caa-ad14-3af7da82a431/resourceGroups/infoasst-icazcolde04/providers/Microsoft.CognitiveServices/accounts/infoasst-aoai-xmlub/deployments/chat' cannot be found.

2023-11-16T02:57:34.205180371Z [2023-11-16 02:57:34 +0000] [76] [INFO] Worker exiting (pid: 76)
2023-11-16T02:57:34.908068005Z [2023-11-16 02:57:34 +0000] [73] [INFO] Shutting down: Master
2023-11-16T02:57:34.909452414Z [2023-11-16 02:57:34 +0000] [73] [INFO] Reason: Worker failed to boot.

[MetteNyland]

I'm receiving the same error.

[andresravinet]

My initial deployment were to existing Azure resources and resource groups. I deployed from scratch to create new resources and I no longer ran into the issues. Hope it helps.

[mikeimmanuel]

Thanks Andres, I've tried re-deploying but had the same issue.

I think the error below was showing it couldn't find a model named 'chat', I may have missed a variable or placeholder in the config/script files, but can't see where.

2023-11-16T02:57:34.205160870Z azure.core.exceptions.ResourceNotFoundError: (NotFound) Specified resource '/subscriptions/3d6eaea2-84d4-4caa-ad14-3af7da82a431/resourceGroups/infoasst-icazcolde04/providers/Microsoft.CognitiveServices/accounts/infoasst-aoai-xmlub/deployments/chat

I got it working by going into OpenAi Studio and creating a new deployment called 'chat' for the 'gpt-35-turbo' model. Then restarted the Web service.

Hope this helps if anyone else has the same issue.

[dayland]

Duplicate of #344