-
Notifications
You must be signed in to change notification settings - Fork 512
[Heads up] Cloud currently down... #1211
Comments
UPDATE: cloud values for a project do arrive... eventually... |
Ah, fixed... great stuff! :) |
I have restarted the varserver for http. |
Cloud has been going through some rough patches today, on and off since ~1500UT. It's currently really iffy – particularly new change getting reported back by the varserver URL. |
Looking good now... |
Cloud is down again – for the last few hours (since ~1500UT)... |
OK, looks like it's come back again just now. :) |
Cloud down again for last few hours. Taking a minute or so again for projects to retrieve cloud values. |
Cloud was back ~1647UT, but was looking pretty dodgy from ~1730UT, basically unusable from ~1800UT, and failed soon after for a bit, until ~1840UT. It was iffy again from ~1900UT, failing completely at ~1930UT, and is still down now... |
Cloud came back at 0223UT, and it's been basically OK since. However, I'm currently seeing frequent latency spikes for the latest value getting reported back by the varserver URL (up to 1.5s delay about every 10-20 seconds). I've seen this sort of behaviour in the past when cloud has been on the verge of failing, so it may be that it will be down again within the next hour or two... |
I think it's getting worse, so perhaps it'll be even less than hour... Yup, hardly ever seeing a varserver delay less than a second now, and sometimes even >3s. :( |
Looking at the servers (the software), they appear to be making normal progress and we may be running up against limits of the hardware in terms of where they are running. Further analysis is being tracked in another repo. |
FYI, last month we turned on a global throttling mechanism for certain classes of requests that arrive via http. By far, the type of query caught by the throttler are too many requests being sent too quickly to the cloud vars server. The response is a 429 to those requests. |
It looks like cloud is responding somewhat more sensibly now, after going down for about a minute at 1624UT. |
BTW, what I'm looking at is not in a project, but in a cloud client. Every two seconds it updates a cloudvar value, and then checks with the project's varserver URL to see if the new value is 'visible' there. It checks the varserver URL about every 0.15s (up to 5s max) after setting the new value, until it finds the new value has appeared. (Is that too often? –It fairly often finds the value is updated by the first test, which comes back at ~0.2s. If not, it's nearly always there by the second test, which comes back ~0.35s. Occasionally it sees a 'lag spike' of a bit longer [maybe 0.8ish]. If cloud is getting dodgy, as above, then it sees these spikes more regularly.) |
Every 0.15s for 5 seconds is not too often |
OK, that's good to know – though it's very rare that it goes up to 5s at that rate. Looking at the stats, if cloud is behaving well then the cloud client is typically doing three (occasionally a few more) cloudvar sets every 2 secs, one of which is followed 0.15s later by a single varserver URL check, sometimes followed by another 0.15s later if the first didn't show the update. (And every few minutes or so there's a 'lag spike' and it takes a few more checks, separated by 0.15s.) I guess the downside of the way this works is that if cloud is already not behaving well for some reason then it'll end up sending more of those varserver requests, which is probably not ideal for cloud. :/ Given above, I've made my client now increase the delay between each new varserver request by 0.05s after the first two checks. BTW: I'm currently getting long delays of one to four secs again for the varserver URL to notice updates. |
Oops, now it look like cloud has (just about) gone... :( |
Cloud is currently looking iffy, as far as varserver URLs are concerned – getting delays of one to four seconds before updates are seen through varserver URL. However, the behaviour of the rest of cloud appears not too bad (projects are getting their values within a second or two after first loading, and the round-trip speed is at about a second, which is higher than the usual ~250ms for me, but still basically workable for most projects). |
Another user reported running into something that seems like it was caused by this issue in freshdesk no. 59768:
|
Looks like cloud is totally down at the moment – cloudvars not arriving at projects even several minutes after project loads. Also unable to create a viable cloud session via API. |
It does appear that the tcp version might be stuck, while the http server is processing ~475 successful requests per second at the moment. Restarting tcp. |
Any updates on this? |
Bump. |
Any idea on what's happening with the TCP interface? |
Nobody knows. Im sick of fallback to be honest, its too slow |
They said they were porting cloud variables to websockets. I'm not sure
if they're actually doing that though.
…On Sat, Jun 3, 2017 at 1:19 PM, Enclo ***@***.***> wrote:
Nobody knows. Im sick of fallback to be honest, its too slow
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
|
dude, that'd be sick bro |
Last time they said anything was in like October and they said the new system would come in January/Febuary this year |
Also I'm hoping if they even finish this, they'll keep the format the same so that I don't have to rewrite the cloud part of scratchapi yet again 📦 |
@liam4 Very Soon™ 👍 |
Also that ninja was insane |
3 months later |
The new cloud data work has been deployed as a soft launch (accessible by specific URLs) on Production. If anyone would like to help test (which would be greatly appreciated!), here's what you need to know:
Please report any bugs you find here! Thank you. |
Made a small test for it... works great! https://scratch.mit.edu/projects/171867511/?newcloud |
Could be just me, but this is my console output. It continuously reconnects before giving an error of "WebSocket is already in CLOSING or CLOSED state.", then reconnects again, then gives the same error message, over and over again. However, the cloud log correctly updates the variable. Meaning that despite the errors everything is working well. So, just error messages -- no actual problem. I'm using a Windows 10 computer, with my browser being Google Chrome, version 60.0.3112.101 |
That's exciting! Do you have any information about this new cloud system...
Is it still throttled to prevent real time gaming or have you been able to
release more bandwidth? Scratch is a wonderful community and I always felt
the we nothing more compelling than releasing a project where you got to
interact with other scratchers...
…On 22 Aug 2017 17:03, "jwzimmer" ***@***.***> wrote:
Could be just me, but this is my console output.
[image: image]
<https://user-images.githubusercontent.com/18113170/29588937-0073055c-879c-11e7-8f68-a2a42517d116.png>
It continuously reconnects before giving an error of "WebSocket is already
in CLOSING or CLOSED state.", then reconnects again, then gives the same
error message, over and over again.
However, *the cloud log correctly updates the variable*. Meaning that
despite the errors everything is working well.
So, just error messages -- no actual problem.
I'm using a Windows 10 computer, with my browser being Google Chrome,
version 60.0.3112.101
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1211 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AGbNvrp-LkmLofqmBjqGM2FjCgpdzn9nks5sa0y8gaJpZM4KQeOR>
.
|
@griffpatch The system is still throttled (otherwise we simply would not be able to scale and maintain it) but much more liberally than what is in place right now. The goal is to get everyone moved over to this new websocket-based system in the short term and then we'll continue to evaluate and add new features on top of this new infrastructure after Scratch 3.0 is released. |
(Bug persists when all extensions are disabled) |
On Firefox (latest stable) I get Every 5 seconds or so |
@thisandagain You say that cloud is still throttled very well below the rates required for realtime multiplayer. I have a project here: https://scratch.mit.edu/projects/171930267/?newcloud that works quite well with some very basic multiplayer functionality. (Movement and costume changes.) Has this changed since you posted last? |
@CosmicWebServices Thanks for reporting that. Did you notice any problems in the functionality of your project when you saw the error? If you have a link to the project that occurred in as well as steps that cause it to happen, that would be helpful, too. 👍 |
@jwzimmer no not really the link is already above (the one I posted) Suggestion: https://scratch.mit.edu/discuss/topic/274278/?page=2#post-2799178 |
Clouddata has now been migrated to the new websockets based platform and |
I am seeing quite a few problems with the cloud data over the last week
since I started looking at it again.
I have been seeing issues with a high score table script where it appears
to lose previous players scores.
What appears to happen is that the scratch project must fail to load the
current value from the cloud server, but succeeds in writing the value back
(such that only the scores in the cloud variable at the time I last saved
the project are persisted long term). Also, complete failure to connect
appear to be frequent enough.
Right now I am unable to even get a cloud variable to update in the cloud
log. I update it in the project, but it is not seen by other connected
players and no record is logged.
Chrome's console is logging this:
11:18:09 AM | inf | Connecting to cloud data server {server: "
clouddata.scratch.mit.edu"}server: "clouddata.scratch.mit.edu"__proto__:
constructor: ƒ Object()hasOwnProperty: ƒ hasOwnProperty()isPrototypeOf: ƒ
isPrototypeOf()propertyIsEnumerable: ƒ
propertyIsEnumerable()toLocaleString: ƒ toLocaleString()toString: ƒ
toString()valueOf: ƒ valueOf()__defineGetter__: ƒ
__defineGetter__()__defineSetter__: ƒ __defineSetter__()__lookupGetter__: ƒ
__lookupGetter__()__lookupSetter__: ƒ __lookupSetter__()get __proto__: ƒ
__proto__()set __proto__: ƒ __proto__()
VM949:1 11:18:10 AM | inf | Successfully connected to cloud data server
VM950:1 11:18:36 AM | inf | Play recorded
VM951:1 11:19:11 AM | inf | Attempt reconnect to cloud server.
VM953:1 11:19:14 AM | inf | Successfully connected to cloud data server
11:18:09 AM | inf | Connecting to cloud data server {server: "
clouddata.scratch.mit.edu"}server: "clouddata.scratch.mit.edu"__proto__:
constructor: ƒ Object()hasOwnProperty: ƒ hasOwnProperty()isPrototypeOf: ƒ
isPrototypeOf()propertyIsEnumerable: ƒ
propertyIsEnumerable()toLocaleString: ƒ toLocaleString()toString: ƒ
toString()valueOf: ƒ valueOf()__defineGetter__: ƒ
__defineGetter__()__defineSetter__: ƒ __defineSetter__()__lookupGetter__: ƒ
__lookupGetter__()__lookupSetter__: ƒ __lookupSetter__()get __proto__: ƒ
__proto__()set __proto__: ƒ __proto__()
VM949:1 11:18:10 AM | inf | Successfully connected to cloud data server
VM950:1 11:18:36 AM | inf | Play recorded
VM951:1 11:19:11 AM | inf | Attempt reconnect to cloud server.
VM953:1 11:19:14 AM | inf | Successfully connected to cloud data server
VM954:1 11:20:16 AM | inf | Attempt reconnect to cloud server.
VM956:1 11:20:17 AM | inf | Successfully connected to cloud data server
VM957:1 11:21:19 AM | inf | Attempt reconnect to cloud server.
VM959:1 11:21:20 AM | inf | Successfully connected to cloud data server
VM960:1 11:22:28 AM | inf | Attempt reconnect to cloud server.
VM962:1 11:22:33 AM | inf | Successfully connected to cloud data server
VM963:1 11:23:45 AM | inf | Attempt reconnect to cloud server.
VM965:1 11:23:46 AM | inf | Successfully connected to cloud data server
VM966:1 11:24:47 AM | inf | Attempt reconnect to cloud server.
over and over! but it doesn't appear tat it has actually succeeded...
I have been aware of quite a number of times that connectivity appeared to
be down, and I haven't actually been on that many times... so it feels like
it might be quite an issue?
Thanks,
Griffpatch
…On 11 September 2017 at 01:24, Colby Gutierrez-Kraybill < ***@***.***> wrote:
Clouddata has now been migrated to the new websockets based platform and
?newcloud is no longer required to have a project use it.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1211 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AGbNvh6EsiMQMAQS_B6nr68-UMzTTZpsks5shH28gaJpZM4KQeOR>
.
|
Please ignore me... Right now it IS working... I made a schoolboy error ;)
- However, the original issue stands with the score table and periodic
cloud down?
…On 22 September 2017 at 11:28, Andrew Griffin ***@***.***> wrote:
I am seeing quite a few problems with the cloud data over the last week
since I started looking at it again.
I have been seeing issues with a high score table script where it appears
to lose previous players scores.
What appears to happen is that the scratch project must fail to load the
current value from the cloud server, but succeeds in writing the value back
(such that only the scores in the cloud variable at the time I last saved
the project are persisted long term). Also, complete failure to connect
appear to be frequent enough.
Right now I am unable to even get a cloud variable to update in the cloud
log. I update it in the project, but it is not seen by other connected
players and no record is logged.
Chrome's console is logging this:
11:18:09 AM | inf | Connecting to cloud data server {server: "
> clouddata.scratch.mit.edu"}server: "clouddata.scratch.mit.edu"__proto__:
> constructor: ƒ Object()hasOwnProperty: ƒ hasOwnProperty()isPrototypeOf: ƒ
> isPrototypeOf()propertyIsEnumerable: ƒ propertyIsEnumerable()toLocaleString:
> ƒ toLocaleString()toString: ƒ toString()valueOf: ƒ
> valueOf()__defineGetter__: ƒ __defineGetter__()__defineSetter__: ƒ
> __defineSetter__()__lookupGetter__: ƒ __lookupGetter__()__lookupSetter__:
> ƒ __lookupSetter__()get __proto__: ƒ __proto__()set __proto__: ƒ __proto__()
> VM949:1 11:18:10 AM | inf | Successfully connected to cloud data server
> VM950:1 11:18:36 AM | inf | Play recorded
> VM951:1 11:19:11 AM | inf | Attempt reconnect to cloud server.
> VM953:1 11:19:14 AM | inf | Successfully connected to cloud data server
> 11:18:09 AM | inf | Connecting to cloud data server {server: "
> clouddata.scratch.mit.edu"}server: "clouddata.scratch.mit.edu"__proto__:
> constructor: ƒ Object()hasOwnProperty: ƒ hasOwnProperty()isPrototypeOf: ƒ
> isPrototypeOf()propertyIsEnumerable: ƒ propertyIsEnumerable()toLocaleString:
> ƒ toLocaleString()toString: ƒ toString()valueOf: ƒ
> valueOf()__defineGetter__: ƒ __defineGetter__()__defineSetter__: ƒ
> __defineSetter__()__lookupGetter__: ƒ __lookupGetter__()__lookupSetter__:
> ƒ __lookupSetter__()get __proto__: ƒ __proto__()set __proto__: ƒ __proto__()
> VM949:1 11:18:10 AM | inf | Successfully connected to cloud data server
> VM950:1 11:18:36 AM | inf | Play recorded
> VM951:1 11:19:11 AM | inf | Attempt reconnect to cloud server.
> VM953:1 11:19:14 AM | inf | Successfully connected to cloud data server
> VM954:1 11:20:16 AM | inf | Attempt reconnect to cloud server.
> VM956:1 11:20:17 AM | inf | Successfully connected to cloud data server
> VM957:1 11:21:19 AM | inf | Attempt reconnect to cloud server.
> VM959:1 11:21:20 AM | inf | Successfully connected to cloud data server
> VM960:1 11:22:28 AM | inf | Attempt reconnect to cloud server.
> VM962:1 11:22:33 AM | inf | Successfully connected to cloud data server
> VM963:1 11:23:45 AM | inf | Attempt reconnect to cloud server.
> VM965:1 11:23:46 AM | inf | Successfully connected to cloud data server
> VM966:1 11:24:47 AM | inf | Attempt reconnect to cloud server.
over and over! but it doesn't appear tat it has actually succeeded...
I have been aware of quite a number of times that connectivity appeared to
be down, and I haven't actually been on that many times... so it feels like
it might be quite an issue?
Thanks,
Griffpatch
On 11 September 2017 at 01:24, Colby Gutierrez-Kraybill <
***@***.***> wrote:
> Clouddata has now been migrated to the new websockets based platform and
> ?newcloud is no longer required to have a project use it.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#1211 (comment)>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AGbNvh6EsiMQMAQS_B6nr68-UMzTTZpsks5shH28gaJpZM4KQeOR>
> .
>
|
Thanks @griffpatch. I'm going to move this over to another issue. |
@griffpatch Thanks for working on cloud data projects! Last night (21 Sep 2017), I deployed some changes to the new cloud data service that has improved connectivity issues. Could you point me at a project where you're seeing the high score and data integrity issues? |
It was on this project :)
https://scratch.mit.edu/projects/174671513/
…On 22 Sep 2017 20:56, "Colby Gutierrez-Kraybill" ***@***.***> wrote:
@griffpatch <https://github.com/griffpatch> Thanks for working on cloud
data projects!
Last night (21 Sep 2017), I deployed some changes to the new cloud data
service that has improved connectivity issues.
Could you point me at a project where you're seeing the high score and
data integrity issues?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1211 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AGbNvpid0CZH6451yI09FNaKNlr7sI0Gks5slBDggaJpZM4KQeOR>
.
|
No connection for https://scratch.mit.edu/projects/21501896/
Other projects are similarly not seeing the cloud values arrive after "connecting to server" box.
Also, cannot connect via API (so custom cloud clients are not working).
However, values are visible through varserver URL, and cloud logs look ok.
The text was updated successfully, but these errors were encountered: