Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix issue "returned non-zero exit status 52: b" on MCW data site #31

Open
tschaffter opened this issue Aug 12, 2021 · 4 comments
Open
Assignees
Labels
Bug Something isn't working Priority: High This issue should be prioritized

Comments

@tschaffter
Copy link
Member

Submissions:

  • 9713417
  • 9713423

The logs generated by the controller show:

"type": "PROGRESS NOTES", "patientId": "1192", "note_name": "dataset/mcw-phi-20210608/fhirStores/evaluation/fhir/Note/1192"}}']' in image 'curlimages/curl:7.73.0' returned     non-zero exit status 52: b'

Interestingly, I encountered this post that says that 52 is an error code if Apache Spark, which I use, associated to Out Of Memory.

However, the above logs is associated to a curl command. I found this comment online:

After some investigation, we found when
curl returns error code 52, it is because Apache server gets seg fault.
It seems not to be a problem with curl 7.9.5 either because I installed
another curl 7.9.5 on my own PC (linux) and I run it to call the same
web page and so far it works fine.

Did the tool crashed?

Other posts mention that this error is associated to "curl: (52) Empty reply from server".

Also:

When you get error 52 back, it literally means "Empty reply from server". When a client (like curl) speaks HTTP to a server, that server must send something back for it to be legitimate HTTP. Protocol-wise, there is no way to be compliant and not respond anything.

Error 52 means the server did just that. It didn't respond with a single byte of data (and then closed the connection). No server in good health should ever do this.

@gkowalski I'm wondering whether the tool crashed, which may have led to curl receiving an empty response. This is way having a look at the MCW logs for the tool could help to identify if the tool crashed.

@tschaffter tschaffter added the Bug Something isn't working label Aug 12, 2021
@tschaffter tschaffter self-assigned this Aug 12, 2021
@tschaffter
Copy link
Member Author

tschaffter commented Aug 12, 2021

The tool annotated 193 notes then died during the submission 9713417.

From 9713419_fail.csv.zip:

"
[Stage 191:==========================================>              (3 + 1) / 4]

                                                                                
[pid: 76|app: 0|req: 186/186] X.X.X.X () {32 vars in 432 bytes} [Fri Aug  6 17:51:50 2021] POST /api/v1/textPersonNameAnnotations => generated 149 bytes in 1162 msecs (HTTP/1.1 200) 2 headers in 72 bytes (1 switches on core 0)"
"
[Stage 192:==========================================>              (3 + 1) / 4]

                                                                                
[pid: 76|app: 0|req: 187/187] X.X.X.X () {32 vars in 432 bytes} [Fri Aug  6 17:51:52 2021] POST /api/v1/textPersonNameAnnotations => generated 562 bytes in 2165 msecs (HTTP/1.1 200) 2 headers in 72 bytes (1 switches on core 0)"
"
[Stage 193:==========================================>              (3 + 1) / 4]

                                                                                
[pid: 76|app: 0|req: 188/188] X.X.X.X () {32 vars in 432 bytes} [Fri Aug  6 17:51:54 2021] POST /api/v1/textPersonNameAnnotations => generated 258 bytes in 1270 msecs (HTTP/1.1 200) 2 headers in 72 bytes (1 switches on core 0)"
"
[Stage 194:==========================================>              (3 + 1) / 4]

                                                                                
[pid: 76|app: 0|req: 189/189] X.X.X.X () {32 vars in 432 bytes} [Fri Aug  6 17:51:56 2021] POST /api/v1/textPersonNameAnnotations => generated 470 bytes in 7631 msecs (HTTP/1.1 200) 2 headers in 72 bytes (1 switches on core 0)"
"
[Stage 195:==========================================>              (3 + 1) / 4]

                                                                                
[pid: 76|app: 0|req: 190/190] X.X.X.X () {32 vars in 432 bytes} [Fri Aug  6 17:52:04 2021] POST /api/v1/textPersonNameAnnotations => generated 351 bytes in 6201 msecs (HTTP/1.1 200) 2 headers in 72 bytes (1 switches on core 0)"
"
[Stage 196:==========================================>              (3 + 1) / 4]

                                                                                
[pid: 76|app: 0|req: 191/191] X.X.X.X () {32 vars in 432 bytes} [Fri Aug  6 17:52:11 2021] POST /api/v1/textPersonNameAnnotations => generated 663 bytes in 1790 msecs (HTTP/1.1 200) 2 headers in 72 bytes (1 switches on core 0)"
"
[Stage 197:==========================================>              (3 + 1) / 4]

                                                                                
[pid: 76|app: 0|req: 192/192] X.X.X.X () {32 vars in 431 bytes} [Fri Aug  6 17:52:13 2021] POST /api/v1/textPersonNameAnnotations => generated 38 bytes in 1383 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)"
"
[Stage 198:==========================================>              (3 + 1) / 4]

[Stage 198:==========================================>              (3 + 1) / 4]

                                                                                
[pid: 76|app: 0|req: 193/193] X.X.X.X () {32 vars in 433 bytes} [Fri Aug  6 17:52:15 2021] POST /api/v1/textPersonNameAnnotations => generated 38 bytes in 65091 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)"
SIGINT/SIGQUIT received...killing workers...
gateway "uWSGI http 1" has been buried (pid: 77)
Fri Aug  6 18:05:11 2021 - uWSGI worker 1 screams: UAAAAAAH my master disconnected: i will kill myself !!!

Second submission that failed:

From 9713424_fail.csv.zip:

"
[Stage 198:==========================================>              (3 + 1) / 4]

[Stage 198:==========================================>              (3 + 1) / 4]

                                                                                
[pid: 76|app: 0|req: 193/193] X.X.X.X () {32 vars in 433 bytes} [Fri Aug  6 19:47:33 2021] POST /api/v1/textPersonNameAnnotations => generated 38 bytes in 63493 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)"
SIGINT/SIGQUIT received...killing workers...
gateway "uWSGI http 1" has been buried (pid: 77)
Fri Aug  6 19:50:12 2021 - uWSGI worker 1 screams: UAAAAAAH my master disconnected: i will kill myself !!!

@tschaffter
Copy link
Member Author

Interestingly both submissions failed at/after the same request (193), which is before the end of the MCW dataset (424 notes). The tool logs don't show an error, just SIGINT/SIGQUIT received...killing workers....

@tschaffter
Copy link
Member Author

@thomasyu888 Given the above information, are there cases where event outside of the tool - for example from the controller - that could kill the tool?

@tschaffter tschaffter added the Priority: High This issue should be prioritized label Aug 12, 2021
@thomasyu888
Copy link
Member

@tschaffter Not that I know of, unless the tool reached max memory allotted to it by the docker run command.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working Priority: High This issue should be prioritized
Projects
None yet
Development

No branches or pull requests

2 participants