Debug api "502 Bad Gateway" issue

[fec-jli]

What we’re after

Recently, some api users report get the following error frequently :
"502 Bad Gateway: Registered endpoint failed to handle the request." when per_page > some number (like 100, 80.. on a couple of endpoints)

From user 1 's email:

Unsure as to what may be causing the error, but I continue to receive a 502 error when running the below request:
https://api.open.fec.gov/v1/committee/C00075820/reports/?sort_hide_null=false&per_page=100&api_key=DEMO_KEY&page=1&sort=-coverage_end_date&sort_null_only=false&sort_nulls_last=false
Result: 502 Bad Gateway: Registered endpoint failed to handle the request.

However, if I change the results per page to anything 50 and below, the request is successful.
 
https://api.open.fec.gov/v1/committee/C00075820/reports/?sort_hide_null=false&per_page=50&api_key=DEMO_KEY&page=1&sort=-coverage_end_date&sort_null_only=false&sort_nulls_last=false
 
This is the case for only this committee. I have run five other party committee reports and have been able to use 100 results per page. 

From user 2 's email:

here's a URL that gave the "502 Bad Gateway" issue while I was scraping Disbursement data for the 2002 election cycle.

https://api.open.fec.gov/v1/schedules/schedule_b/?sort_null_only=false&api_key=DEMO_KEY&sort_hide_null=false&two_year_transaction_period=2002&per_page=100&sort=-disbursement_date&last_index=3062020110010618030&last_disbursement_date=2001-02-27T00:00:00

There is no issue on local api env. (connect dev db)

• Ref Test more gevent workers per app instance #4489

Action item(s)

• Identify the root cause of this issue
• Fix the "502 Bad Gateway" issue

Completion criteria

• We know what caused the 502s
  and
• the queries in this ticket no longer throw a 502, or
• we have created ticket(s) to fix it

[fec-jli]

cms gunicorn config: https://github.com/fecgov/fec-cms/blob/develop/bin/run.sh

api gunicorn config: https://github.com/fecgov/openFEC/blob/develop/bin/run.sh

[pennywise420]

https://api.open.fec.gov/v1/committee/C00075820/reports/?sort_hide_null=false&per_page=50&api_key=DEMO_KEY&page=1&sort=-coverage_en

[fec-jli]

https://api.open.fec.gov/v1/committee/C00075820/reports/?sort_hide_null=false&per_page=50&api_key=DEMO_KEY&page=1&sort=-coverage_en

Thanks, @pennywise420 the whole link is:
https://api.open.fec.gov/v1/committee/C00075820/reports/?api_key=DEMO_KEY&sort_null_only=false&per_page=100&page=1&sort_hide_null=false&sort=-coverage_end_date&sort_nulls_last=false

[pkfec]

Quick summary of the things observed on Stage space:

1. Tried increasing the API app instances to 4
2. Tested committee/<committee_id>/reports and /schedules/schedule_b/ on Stage space by changing the per_page ranges from 80-100. The endpoints returned results until we EXCEEDED the DEMO_KEY rate limits.
3. Will monitor and do more testing tomorrow
4. We have to revisit the logic to scale the gevent workers.
5. Or scale up the api app instances

[pkfec]

An API user recently reported getting a 502 Bad Gateway error on SA endpoint

https://api.open.fec.gov/v1/schedules/schedule_a/?api_key=DEMO_KEY&per_page=100&two_year_transaction_period=2022&min_amount=50000

https://api.open.fec.gov/v1/schedules/schedule_a/?api_key=DEMO_KEY&per_page=100&two_year_transaction_period=2020&min_amount=50000

[fec-jli]

ref ticket:
[Tenable] API missing/inadequate response headers
Add security-related headers

[dzhang-fec]

Summary: The above 502 error issue are on multiple links. There were reported by multiple clients. We need to find a way for the coming elections. We can duplicate it
Root Reason Analysis: The browser has the limit of a web page which is around 250KB if it is not HTML content (similar on all of browsers). It is not a high loading issue, not a database query performance issue, not app python issue (unless we change the one line JSON data format code to display it with html. or multiple JSON lines)

workaround: to use curl or wget to get the large >250-350KB content. Actually, it is better for the client to automate it with a script/command line to parse the JSON format automatically. (Our API is not for users to view data from browser :) but use scripts)

Please review analysis details, workaround code and demo here; #5184 (comment)
Problem:

Analysis
I use curl/wget can get the result successfully and quickly
Here is the page_size
-rw-r--r--. 1 oracle oinstall 507559 Oct 20 11:28 100.txt (no way from the browser)
-rw-r--r--. 1 oracle oinstall 456644 Oct 20 11:29 90.txt (no way from the browser)
-rw-r--r--. 1 oracle oinstall 405903 Oct 20 11:30 80.txt (no way from the browser)
-rw-r--r--. 1 oracle oinstall 354925 Oct 20 11:30 70.txt (no way from the browser or sometimes)
-rw-r--r--. 1 oracle oinstall 304284 Oct 20 11:30 60.txt (no way from the browser or sometimes)
-rw-r--r--. 1 oracle oinstall 254006 Oct 20 11:30 50.txt ---browser ok
-rw-r--r--. 1 oracle oinstall 507559 Oct 20 11:36 100curl.txt (no way from the browser)

--the best is to use curl sample code
curl "https://api.open.fec.gov/v1/committee/C00075820/reports/?sort_hide_null=false&per_page=100&api_key=DEMO_KEY&page=1&sort=-coverage_end_date&sort_null_only=false&sort_nulls_last=false" >100curl.txt

or wget , sample code
wget "https://api.open.fec.gov/v1/committee/C00075820/reports/?sort_hide_null=false&per_page=100&api_key=DEMO_KEY&page=1&sort=-coverage_end_date&sort_null_only=false&sort_nulls_last=false" >100wget.txt

This works for Linux, MacOS or window users. samples


--one line result

Summary: use curl workaround.
(If we still want to use browser to do it, please consider to use html, zip, file download or multiple lines)

[cnlucas]

Closing in favor of #5267