Website returning 504 error

[rjayasekera]

On 5/19/19 around 4:45 am accessing www.fec.gov giving the following error:

504 ERROR

The request could not be satisfied.

CloudFront attempted to establish a connection with the origin, but either the attempt failed or the origin closed the connection. 
If you received this error while trying to use an app or access a website, please contact the provider or website owner for assistance. 
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by following steps in the CloudFront documentation (https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/http-504-gateway-timeout.html). 
Generated by cloudfront (CloudFront)
Request ID: 2sILIATH-HC4KiJ6fgogFMuv-1lcOgdG6REbIcEJN4JI6oxXh8j0gA==

The reason I check www.fec.gov because I am getting lot of newrelic alerts:

** Directly calling api.open.fec.gov returning results **
** Kibana CMS and API log files did not show any obvious errors **

API umbrella is showing following error codes

** Not quite sure this is API or CMS specific issue. Website start to function normally after 5/19/19 6:00 am.

[jason-upchurch]

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/http-504-gateway-timeout.html

[jason-upchurch]

Submitted email inquiry to cloud.gov

[jason-upchurch]

CloudFront does not currently maintain logs. Need to check kibana to see if the logs offer any clues.

[jason-upchurch]

We are investigating the possibility that this issue was caused by a Tenable scan.

[jason-upchurch]

Moving to Sprint 9.2 as discussion is ongoing.

[jason-upchurch]

Investigating app scheduling relative to actual scan.

[jason-upchurch]

Next scan 6/19/2019, moving to Sprint 9.3 to gather information resulting from the scan.

[jason-upchurch]

Closing Summary

Overnight tenable scans over the evenings of April 18/19 and May 18/19 are correlated with website outages. Scans over these two evens appeared to target the proxy and cms cf.app in prod cf.space.

Over the 24-hour period of May 19, (a Sunday) 91.5 % of the traffic was generated by the Tenable scan.

It was noted during this investigation that all Tenable scan settings were default. During the course of the inquiry, additional debugging logs were enabled on the Tenable scan.

After following up on the June 18/19 scan, it was noted that the website did not experience outage. However, it was determined that the scan itself was aborted due to an unexpected problem.

This issue is being closed and the July 18/19 scan will be tracked under fecgov/openFEC#3833