Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python language worker times out at startup #3551

Closed
balag0 opened this issue Oct 2, 2018 · 4 comments
Closed

Python language worker times out at startup #3551

balag0 opened this issue Oct 2, 2018 · 4 comments
Assignees

Comments

@balag0
Copy link
Contributor

balag0 commented Oct 2, 2018

FunctionsLogs
| where RoleInstance contains "171AE528-636740959597695685"
| where TIMESTAMP >= datetime(2018-10-02 16:55:00.0000000)
| where TIMESTAMP <= datetime(2018-10-02 16:55:00.0000000) + 30m
| where Tenant == "fe133483000c4137a26aabfc6365308c"
| where SourceNamespace == "LWAWS"
| extend t = todatetime(EventTimestamp)
| order by t asc
| where Summary !contains "host availability"

             
  t Source Details Summary ExceptionType ExceptionMessage
  2018-10-02 17:21:40.1780000 Worker.python.6261ebf9-cf33-4c4d-9fd2-7954e104d96c System.TimeoutException : The operation has timed out. Language Worker Process exited. System.TimeoutException The operation has timed out.
  2018-10-02 17:21:41.6600000 Worker.python.6261ebf9-cf33-4c4d-9fd2-7954e104d96c Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerProcessExitException : bash exited with code 137 Language Worker Process exited. Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerProcessExitException bash exited with code 137
  2018-10-02 17:22:21.6860000 Worker.python.b6d0060c-2e99-4746-a05b-1862e323f36a System.TimeoutException : The operation has timed out. Language Worker Process exited. System.TimeoutException The operation has timed out.
  2018-10-02 17:22:23.6880000 Worker.python.b6d0060c-2e99-4746-a05b-1862e323f36a Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerProcessExitException : bash exited with code 137 Language Worker Process exited. Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerProcessExitException bash exited with code 137
  2018-10-02 17:23:04.2140000 Worker.python.f1e47f9e-4198-4cbf-ab79-203c952c6a89 System.TimeoutException : The operation has timed out. Language Worker Process exited. System.TimeoutException The operation has timed out.
  2018-10-02 17:23:06.2160000 Worker.python.f1e47f9e-4198-4cbf-ab79-203c952c6a89 Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerProcessExitException : bash exited with code 137 Language Worker Process exited. Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerProcessExitException bash exited with code 137
  2018-10-02 17:23:12.4250000 Worker.python.f1e47f9e-4198-4cbf-ab79-203c952c6a89 Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerProcessExitException : Worker process is not attached.  ---\u003e System.InvalidOperationException : No process is associated with this object.    at System.Diagnostics.Process.EnsureState(State state)    at System.Diagnostics.Process.get_HasExited()    at System.Diagnostics.Process.EnsureState(State state)    at System.Diagnostics.Process.get_ExitCode()    at Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerChannel.OnProcessExited(Object sender,EventArgs e) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Rpc/LanguageWorkerChannel.cs : 188     End of inner exception Language Worker Process exited. System.InvalidOperationException No process is associated with this object.
  2018-10-02 17:23:12.4250000 Worker.python.b6d0060c-2e99-4746-a05b-1862e323f36a Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerProcessExitException : Worker process is not attached.  ---\u003e System.InvalidOperationException : No process is associated with this object.    at System.Diagnostics.Process.EnsureState(State state)    at System.Diagnostics.Process.get_HasExited()    at System.Diagnostics.Process.EnsureState(State state)    at System.Diagnostics.Process.get_ExitCode()    at Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerChannel.OnProcessExited(Object sender,EventArgs e) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Rpc/LanguageWorkerChannel.cs : 188     End of inner exception Language Worker Process exited. System.InvalidOperationException No process is associated with this object.
@asavaritayal
Copy link

@balag0 can you share steps for repro?

@balag0
Copy link
Contributor Author

balag0 commented Nov 7, 2018

This issue applies to both python and node.

There might be more than 1 way to repro this. Here is what i used-

Create a python / node linux consumption function app with a hello-world like function.
Hit the function endpoint using atleast 500 virtual clients for 5 minutes.
Some of the workers will hit this issue and return 500s.

@anirudhgarg
Copy link
Member

anirudhgarg commented May 30, 2019

@balag0 Can you please repro and if its not working find out root cause.

@balag0
Copy link
Contributor Author

balag0 commented Aug 12, 2019

No repro anymore.

@balag0 balag0 closed this as completed Aug 12, 2019
@ghost ghost locked as resolved and limited conversation to collaborators Jan 1, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants