Server crash: No indication in logs

[crspybits]

I just had to restart the server. I couldn't access it (e.g., /CheckCreds/ wasn't working), and neither could Dany. I tried two of my devices with SharedImages on it. The logs looked fine-- I could see no explicit reason for the server to stop receiving connections. (see output6.log on the server).

Once I killed and restarted the Main server process, all was working again. I put quotes around "crash" in the title because when I did a ps -A | grep Main, the Main process was indeed there. It wasn't that kind of crash.

The SyncServer instance had been running for about 8 days (see output6.log).

[crspybits]

I've just had another one of these "crash"es. /CheckCreds/ isn't working. I get no response on the server, when doing a tail on the log. With only 120 requests in the logs. Again, Main is present. See log output7.log.

[crspybits]

I have now updated to the latest Swift 4 build on AWS (since I'm using Swift 4 when testing on my local Linux/Ubuntu system), and have rebuilt the server and restarted.
It could also be useful to update all packages on AWS Ubuntu (and restart) and to attempt to deal with the build warnings I'm getting:

warning: you may be able to install openssl using your system-packager:
warning:     apt-get install openssl libssl-dev

warning: you may be able to install mysqlclient using your system-packager:
warning:     apt-get install libmysqlclient-dev

I believe those packages were installed before, but perhaps they need updating?

[crspybits]

One more instance of this today: Dany just saw it and I'm seeing it too:
Server is "crashing"-- becoming unresponsive to HTTP requests.
The AWS EC2 instance is still up.
Server ("Main") is still running.
It's just not responding.
See output8.log.

[crspybits]

I just updated all outstanding (out of date) packages on the EC2 Ubuntu instance.

[crspybits]

Same issue again. It is 9/7/16. The last successfully processed request is listed in the log on 9/4/17, which is the day after the server was started on 9/3/17. Log: output9.log.
A request to https://syncserver.cprince.com/HealthCheck/ times out.

[crspybits]

Question: If I make a request to the server from the same machine as the server is running on, does it give the same response? E.g., using curl.

I tried: curl https://127.0.0.1/HealthCheck/ from the AWS EC2 instance itself, and that too times out, with: curl: (35) gnutls_handshake() failed: Error in the pull function.

I did check, and the Main process is still running.
A report from top gives:

The process state S of S indicates sleeping.

Note that while the above curl request does fail, with a working server:
curl -k https://127.0.0.1/HealthCheck/ is needed when the server is actually running because of SSL signing.

[crspybits]

I'm now trying a different approach. This issue seemed to start just after I refreshed my https://letsencrypt.org SSL certificate. So, I wondered if this has something to do with SSL. One way around this is to proxy requests into the Swift server using NGINX (or Apache). For example, NGINX could listen on port 443 (for SSL/HTTPS), and forward those requests to port 8080.

Other pluses of doing this proxying are:

1. I'd no longer have to run the SyncServer as root-- since ports such as 8080 don't need root permissions for access.
2. This could make it easier to run a dev or staging instance of the server.

I now have proxying going on using NGINX. I'll document this in https://crspybits.github.io/SyncServerII/

[crspybits]

I've not seen this issue for over a month. Issue seems solved!