API method does not process incoming requests from time to time

[hunterbr72]

Describe the bug

I have a temporary bug which only occurs from time to time. My setup is Python 3.8, uvicorn 0.11.3, fastapi 0.52.0. Client side is an IDA (Interactive Disassembler) python plugin which sends out API requests using Pythons requests module version 2.23.0 and due to an IDA bug Pyhton 3.7. The latter shouldn't be important as far as I see a valid HTTP request coming in on the server side. Client and Server both running on Windows 10. The error occures totally randomly, sometimes three times in one minute, sometime only every two days or so. When the error occurs the server does not process any commands anymore, it just stuck at the point described below until I hit Ctrl-C in the servers cmd window, then it is proceeding with processing the request.

To Reproduce

Due to the described setup above it might be difficult to reproduce it on your side, but let me describe the details:

When the client sends out the JSON requests which looks like this:

{
"apikey": "RB05FY969WBAQEPT16E6K7P3",
"arch_bits": 64,
"buf_addr_addr": {},
"buf_addr_op": {},
"buf_dump_addr": {},
"buf_size_addr": {},
"buf_size_op": {},
"call_addr": {
"0": "1400016A0"
},
"call_ret": {
"0": "D0D0"
},
"dl_file": [],
"eflag_addr": {
"0": "140001830",
"1": "1400018B1"
},
"eflag_name": {
"0": "SF",
"1": "SF"
},
"filelist2del": [],
"id": 12,
"nop_end_addr": {
"0": "14000187C"
},
"nop_start_addr": {
"0": "14000184C"
},
"other": {
"breakaddr": "0x140001b05",
"imagebase": "0x140000000",
"loop_addr": 5368715789,
"offset_disk": 3597
},
"run_opt": null,
"sample_file": "buffertest.exe",
"sample_sha256": "2CB3293CBCA3E847A36D48B87D07BE1D593239F3D8555996E39545724BB3CB52",
"trace_breakaddress": {},
"trace_end": {
"0": "140003000"
},
"trace_light": {
"0": "TRUE"
},
"trace_max_instr": {
"0": "20000"
},
"trace_start": {
"0": "140001000"
}
}

I can see that the requests comes in:

[DDRSERVER][INFO] Successfully deleted temp. dump files. (Note: last successfull requests before bug is hit)
INFO 2020-03-31 13:32:40,191 -- 192.168.60.200:50060 - "POST /api/v1/json HTTP/1.1" 200 [h11_impl 2800 2024] (Note: it stucks here, no other output)

the first lines of the method which should process the last request are looking like this:
@app.post("/api/v1/json")
async def json_api_v1(ddr_api_req: ddr_api_v1_parameter):
print("[DDRSERVER][INFO] Incoming JSON API request.")
json_content = json.loads(ddr_api_req.json())
print("[DDRSERVER][DEBUG] Received JSON data:\n{}".format(json.dumps(json_content,indent=4, sort_keys=True)))

BTW I am starting the app programatically like this:
uvicorn.run(
app,
host=IPADDR,
port=int(PORT),
ssl_cert_reqs=ssl.CERT_OPTIONAL,
ssl_keyfile="ddr_server.key",
ssl_certfile="ddr_server.crt",
)

so, you can see in the log above that even the first 'print("[DDRSERVER][INFO] Incoming JSON API request.")' cmd is not executed when the request comes in. Also, the "[DDRSERVER][INFO] Successfully deleted temp. dump files." is the output of the last instruction in the method which was executed before, so in theory this method was successfully executed. Then a bit later the JSON request comes in which times out and is not processed at all. I have also verified with a sniffer trace that the JSON request is valid. When this error condition occurs, the server doesn't process any further requests anymore until I hit Ctrl-C and kill what ever it blocks.

I took a VM snapshot after the error occured, so I can locally reproduce it, if you want me to check anything.

Expected behavior

The 'json_api_v1' method should be processed when the JSON request comes in.

Environment

• OS: Windows 10
• FastAPI Version: 0.52.0
• Uvicorn Version 0.11.3
• Python version: 3.8.2
• Client Requests version: 2.23.0

Pls tell me if you need any further informations and if you have any idea how I can debug this.

thx,
Holger

[tiangolo]

Hmm, it's very possible that it's a Windows issue.

To begin with, if you start the process with CMD or PowerShell, and if you select text in the terminal, that will freeze everything in that terminal until you unselect it.

And like that, a bunch of other possible issues...

If this is for production, if possible, I would suggest you run it on a Linux machine. For performance, security, etc.

You might also want to try WSL2 or similar. If you are running a VM, maybe your Hypervisor can run a Linux VM too.

On the other side, you might want to try https://sentry.io/ to report errors. You might also try installing and using better_exceptions.

And for debugging, you can try devtools with the debug() function.

[hunterbr72]

thx for your answer tiangelo, I ll look into the debugging options. Unfortunately, I can not move to Linux, because the server is part of a reverse engineering tool for windows, so it has to run on windows.

[tiangolo]

Get it, then maybe those tools can help you debug it 🤞

[Zheaoli]

It seems like a CPython bug on Windows. Would you like to share a simple code that can reproduce your problem by using Gist? I'd like to debug it.

[hunterbr72]

Hi Zheaoli,

sorry for the late reply, was sick and then super busy, I cannot share code itm, because the client side is an IDA plugin (reveng tool), would be difficult to create a similar code snipplet as far as I can not say which part of the code triggers the issue. If you are open for doing a webex/skype or other video conference, I'd try to reproduce the bug and freeze it in a VM and we can have a look together. That might help to identify how we/I could built a code snipplet which we can use for testing/debugging ? ITM I can even not say if it is the issue described by tiangolo, that just somethign was marked in the cmd window, which let the process freeze. I'll ping you asap once i can reproduce the bug and can make sure that it is not the highlighting issue mentioned by tianglo.

thx,
holger

It seems like a CPython bug on Windows. Would you like to share a simple code that can reproduce your problem by using Gist? I'd like to debug it.

[tiangolo]

Thanks for the help here @Zheaoli !

Great, let's hope that works @hunterbr72

[github-actions]

Assuming the original issue was solved, it will be automatically closed now. But feel free to add more comments or create new issues.

[hunterbr72]

yup, I am 99% sure it was the issue tiangolo mentioned (freeze of app when marking something in cmd window). Thx a lot for your support !