502 when trying to fetch updates #42
Comments
ghost
assigned 
|
Interesting. I can't even hit the application at all anymore. Once I'm home I'll try to check logs and see what's up. Last time I saw something similar it turned out to be the MongoDB being ripped out from under the app. |
|
Also +1 on chucked encoding. |
|
I'm seeing the same error in the logs where the MongoDB connection seems to get cut while getting data. Still looking. |
|
I just tried the URL above and I'm getting back: [] |
|
@abn can you see if you are getting the same error? |
|
Right now we are using mod_wsgi in Apache: http://flask.pocoo.org/docs/deploying/mod_wsgi/. I'm not opposed to using something different though. Also, looks like http://flask.pocoo.org/docs/patterns/streaming/ is a good pattern for returning the larger json items. |
|
@abn if you are still getting the error please reopen this issue. AFAICT it's no longer happening. The logs seemed to indicate that the Mongo database cut connections earlier today which may have been the cause. After restarted the container (and it reestablished connections) everything was back to normal. |
|
The service seems to be quite intermittent. When it is working, it still doesn't give me complete JSON responses. I think the issue is the server side imposing some kind of constraint on process execution time, or max download size. For example with victims-enforcer I get: +=============================+ fingerprint : fatal [info] Victims database last entry was created on Thu Jan 01 10:00:00 EST 1970. Trying to do the sync manually via wget: $ wget https://victims-websec.rhcloud.com/service/v2/update/2013-01-01T00:00:00/ 2% [> ] 1,071,688 --.-K/s in 19s 2013-05-13 17:22:46 (56.5 KB/s) - Connection closed at byte 1071688. Retrying. --2013-05-13 17:22:47-- (try: 2) https://victims-websec.rhcloud.com/service/v2/update/2013-01-01T00:00:00/ So it looks like the server side is dropping the connection 1mb into the 50mb response :( |
|
@ashcrow now I am getting a 500 for https://victims-websec.rhcloud.com/service/v2/update/1900-01-01T00:10:20/ I think this should hopefully be fixed with #44 As @dfj 's comment mentions, the service is dying off before serving the entire response. |
|
I think you guys are right and there is a max set as to how much |
|
I'm hoping streaming will fix this. If not we may have to look at some other options for responding with such a "large" data set. |
|
@ashcrow is streaming live? |
|
👍 |
|
@abn, latest code was pushed out. The service so far is working for me -- though it's not the fastest thing in the world. Let me know if it stays stable for you guys or if the connections get killed off. |
[abn@whippersnapper ~]$ curl "https://victims-websec.rhcloud.com/service/v2/update/2010-01-01T01:01:01" | tee vic.json
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 547 100 547 0 0 2 0 0:04:33 0:04:02 0:00:31 132
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request <em><a href="/service/v2/update/2010-01-01T01:01:01">GET /service/v2/update/2010-01-01T01:01:01</a></em>.<p>
Reason: <strong>Error reading from remote server</strong></p></p>
<hr>
<address>Apache/2.2.15 (Red Hat) Server at victims-websec.rhcloud.com Port 443</address>
</body></html>
[abn@whippersnapper ~]$ And I doubt if the streaming is working :( |
|
Using victims-enforcer: [info] Synchronizing CVE definitions with local database.. $ wget https://victims-websec.rhcloud.com/service/v2/update/2013-01-01T00:00:00/ :( |
|
@ashcrow seems to be working now since the restart. |
|
waiting for it to complete so i can verify the data |
|
Works for me (hopefully reproducible) [abn@whippersnapper ~]$ cat vic.json | grep -o "cves" | wc -l
352 |
|
Looks like it is very temperamental (wonder why though). Tried https://victims-websec.rhcloud.com/service/v2/update/2014-01-01T01:01:01 (even thought there should be no results, it 502s). @ashcrow would this be something caused by OpenShift? |
|
I think the solution is move off openshift. I am happy to provide a VM on the rackspace cloud to host it, billed to my own account. If we prove that this solves the problems, then I can request funding for the ongoing operation of the VM. @ashcrow what do you think of that idea? If you're OK with it, just let me know and I'll create the VM and give you credentials to login. |
|
Let's give a full, controlled vm a shot. I have room for another vm we can use for the test bed. I'm setting it up now and will try to deploy to it tomorrow mid day EDT for testing morning Wed morning your time. |
|
I've setup a test instance on a VM. Today please try the same tests against this VM instance and see if we continue to get 5xx's or if things work. |
|
https://victims-websec.rhcloud.com/service/v2/update/2014-01-01T01:01:01 returns [] correctly. Testing bigger requests. |
|
@abn note that the rhcloud is going to be the original instance. I've sent an email with test instance location. Booting it back up now in case you'd like to look at it. |
|
is the base uri the same?? |
|
Also, @abn, do you ever sleep? :-) |
|
sure will test now. @ashcrow can you hit me @ gmail please? |
|
@abn will do. Done. |
|
2 simultaneous requests for service/v2/update/2010-01-01T01:01:01/ |
|
rhcloud works too, however is unreliable the dedicated vm seems to be more reliable. service/v2/update/2014-01-01T01:01:01/ Interesting bit here is that the cookie was set correctly here but not the previous request. |
|
Service seems a tiny bit slow though, do we want to investigate that and fine tune in another issue? (not bad bad) |
|
Dedicated VM 👍 Openshift gives 500 on second request. |
|
@ashcrow also sleep? What be this thing you speak off? |
|
Raised #52 as a longer term issue to look at perf. |
|
@abn yes, it is slow but I think that is to be expected. 50MB transfer and the client can not really load the json struct until all the data is been downloaded due to it being contained in a list (... I think). |
|
standalone client works with vm host @gcmurphy enforce still fails, but this seems to be a different issue |
|
@ashcrow might be yes, but that is peculiar for a request that returns "[]" |
|
@abn oh, yes, very much. We can look at less controller based transforms and better caching as well. |
|
@abn any objection to closing this since we now know what the fix is and have open tickets for them? |
|
I have raised #52 for a general perf review. Would be interesting to see how much we can squeeze out of it. |
|
@ashcrow no objections closing now. The fix for this issue will be migrating the service to a dedicated VM. #51 @dfj fix for enforcer will be at victims/victims-enforcer-legacy#3 |
Not sure if this OpenShift's doing or the flask web-server. If it is the latter, should we consider alternatives? eg: CherryPy? (http://flask.pocoo.org/snippets/24/)
We also need to be able chunk the json response.
The text was updated successfully, but these errors were encountered: