Overall performance is slow

[eliezedeck]

I love the fact that AWS is tackling the GraphQL world. And being able to take advantage of the many other services that are accessible from AppSync is quite an attractive option.

But as the title says, the experience when using AppSync is very slow.

Compared to the likes of Firebase Realtime Database, Meteor, Graph.cool, etc... it is usually 3x slower. I am already using the closest region that I get a 250 ms RTT (Europe WEST I)), yet, queries, mutations, subscriptions all range in the 1100 - 1400 milliseconds.

Rarely do I see queries being resolved in less than 500 ms. Whereas using the alternatives mentioned above, it is rare to see any response above 500 ms. I am from Madagascar and am already on the fastest connection I can get my hands on over here. I agree that GraphQL itself needs a very complex implementation, but Graph.cool in their worst case scenario is able to respond in less than 500 ms. Note that I'm talking about the simplest query possible here; I have not even tried joins.

While AppSync is usable right now, the performance leaves little to be desired. I have no idea how the implementations are, but AWS, above all the company examples I could think of, should be the first to know the negative impact that slow apps have on users and clients.

Looking forward to having this performance issue solved.

[appwiz]

Hi @eliezedeck, I'm the lead for AWS AppSync. I'd like to work with you to understand why you are experiencing these latencies. Could you DM me on Twitter (I'm @appwiz) with your API endpoint please?

[burmisov]

Having a similar experience here; any news popped up the last month?
@eliezedeck @appwiz

[undefobj]

@burmisov it's hard to answer without knowing the details of your specific configuration. Many things can impact latency including the region you're running from, the data sources you're using, and the queries that you are running. The benchmarking methodology must also be looked at - for example the AppSync console is not a reflective test of how clients would operate, instead you're better off running tests using the language specific controls (like console.time() in JavaScript and looking at round trips). You should also run multiple tests as individual HTTP requests will have some less consistent results. If your client is on a lossy network or geographically far away from the AppSync region that can impact latencies, as can Lambda data sources where the runtime is more variable depending on the code you use. Finally the queries or operations you run on a data source can impact latencies as well as the resource provisioning, for example a full table scan on a DynamoDB table with low RCUs will return results slower than a tuned operation.

AppSync provides monitoring, logging, and debugging resources for looking at request times which you can find here:
https://docs.aws.amazon.com/appsync/latest/devguide/monitoring.html
https://docs.aws.amazon.com/appsync/latest/devguide/test-debug-resolvers.html#debugging-a-live-query

If you'd like assistance benchmarking and isolating any latency issues I'd also recommend opening up a ticket using the AWS console or visiting the AppSync forums: https://forums.aws.amazon.com/forum.jspa?forumID=280&start=0

[tebeco]

@appwiz is the following good enough for a scenario ?

If we have a web page

• we divide the screen page on ~20 block/group
• each block as a div with a field to update
• We push values on each group (between 1 every 3 sec and 4 per seconds)
• We need to update each block's div with the value
• The order of the update MUST be in the exact same order as the push
• If there's no data in a group for 0.5s we push an empty frame to the group to that we know the connection is not dead
• AppSync does the fanout and conflation for us ?

• FanOut : we don't have to do the foreach(client in clients) {push to one client}
• Conflation : if i push too much data over appsync, AppSync handle the pressure and discard the oldest value and will only focus on the top most recent one

[undefobj]

@tebeco this question would be best suited for a different issue or posting on the AppSync forum as this is the JS SDK repo. But some notes on your inquiry before closing the issue:

• AppSync does handle the fanout for you to clients and connection management
• The SDK handles reconnections of subscriptions
• Message ordering is not guaranteed. You will need to implement that yourself.
• Missed messages - this would need investigation. Can you post the setup to the AppSync forum or open a ticket with AWS support to troubleshoot?

[tebeco]

i am not in contact with the creator of the issue so may want to re-open it for him

[tebeco]

Also, the question is about slowness,
we also tried AppSync since it is a “real time solution” we observe that it was quite slow too

so i posted a simple scenario :

• multiple groups service side (AppSync cluster)
• we get a websocket and add it in some (~20 group)
• we push 4 value per second in multiple group (not the same values)

4 per sec is a good and simple scenario for “real time” also since it is the same websocket used i guess the payload of each frame will be handled by the SDK client side so we would only to map the push(group, value) to whatever we want

[undefobj]

@tebeco as requested please post this in the AppSync forum. The information above is not enough to diagnose. We would need information like what your GraphQL schema looks like, region you're using, resolvers, etc.

[tebeco]

hummm ... so what is this github all about ? when do we come here create issue ?
what about all the github goodness ?

[undefobj]

@tebeco that's a valid question - this is a repo for the JavaScript SDK and you're asking a question related to the actual service itself. The questions in this repo should be related to the JavaScript SDK so we can track, triage, and address them as we have engineers here from that team. Similarly the AWS AppSync forum has engineers supporting the service that can assist you best there. Hopefully that clarifies things.

[ngocketit]

I also have the same issue. Is this still possible to discuss things here or should I create a new ticket? I'm willing to provide all the needed detail if that helps to pinpoint the source of the problem. For my case, it never takes less than 500ms for a response payload of around 4K.

[hisham]

Hi - I recently posted on the AppSync forum on this: https://forums.aws.amazon.com/thread.jspa?threadID=296504&tstart=0

Was told to send more info on my API requests and they did investigate and gave me some suggestions.

[ngocketit]

@hisham As Appsync is getting more popular, it looks like more people are experiencing the similar issues. This is my first project built with Appsync and it'd be very disappointing going live with that performance.

[appwiz]

@ngocketit As requested earlier in the thread, please post your issue to the AppSync forum and the team can assist you.

[ngocketit]

@appwiz Thanks for your comment! I tried to post but got error Your account is not ready for posting messages yet. I'll try again after few hours. Like I said earlier, I'm willing to provide all the necessary detail if that help to look into what's causing the high latency. So in case you need something from me, please let me know.

[hisham]

@ngocketit generally these add to latency:

• Lambda methods you are calling that might take some time to resolve
• Other data sources AppSync is calling (dynamo, etc..) that are in a different region than your AppSync setup
• Nested graphql queries. I find the default queries auto-generated by amplify codegen for relational data can tend to be too heavy for some purposes.

I'm wondering do you have any of that?

[ngocketit]

@hisham I'm not having any of those you mentioned. The resolver is talking to Elasticsearch directly which managed to get the result for the same query in almost no time when I tried in Kibana. Even when I tried to run another query (GetItem) for a DynamoDb resolver, I still observed the similar latency.

[hisham]

Ah then post to the AWS forum and let me know what they find out, as I'm very curious and need to know for my own development purposes.

[tebeco]

can someone explain why why a github was created to redirect users that have issue to a forum ?

is it to hide info somewhere else so no psy see that there are issues ?
or is it because employee don’t understand github ?
or is it an interresting choice to pay for you own website and maintains it when github is designed to do that for you ?

i mean ... AWS right ? let’s reinvent a tool

[hisham]

Hi @tebeco - I believe @undefobj already answered your question above. This github repo is for aws-mobile-appsync-sdk-js package not the AppSync backend managed service. Separation of concerns! Github is for code not support issues for a backend service.

[ngocketit]

I agree that this isn't the best place to discuss about Appsync so I posted the question here on Aws forum https://forums.aws.amazon.com/thread.jspa?messageID=886300#886300 but got no reply yet. I really hope someone from AWS could have a look. In my opinion, there seems to be something very wrong with Appsync. I've been using it for a while now and really love it. I didn't notice the latency issue until recently when I tried to benchmark my app. It's rather disappointing.

[undefobj]

@ngocketit could you open a new issue with your API details on the AppSync forum? You just commented on an old issue that is 6 months old. The team monitors the forums but it's not possible to monitor all old threads that are since resolved and get new comments months/years later.

[ngocketit]

@undefobj Sorry that I didn't notice your comment until after I posted the message on this thread (which is pretty recent) https://forums.aws.amazon.com/thread.jspa?threadID=296504&tstart=0. Lets see if someone at AWS will get back to me about that. If not, I'll open a new one. Thank you for your support!

[flyandi]

2020 - still same issues persists. I just have very simple pubsub with no data source and just sent a timestamp over and than compare local to message time...

[Audioworker] started
Controller.js:44 1601582933916 722
Controller.js:44 1601582934001 695
Controller.js:44 1601582953820 628
Controller.js:44 1601582953932 659
Controller.js:44 1601582956505 571
Controller.js:44 1601582956619 2162
Controller.js:46 Message OOD
Controller.js:44 1601582963377 624
Controller.js:44 1601582963466 712
Controller.js:44 1601582965310 659
Controller.js:44 1601582965403 667

Not getting anything better than 700ms .. I filter out a message if its pass 1s as it might be overruled already by another message but those latency times are crazy high. Tested the same thing with a simple raw websocket server running as lambda with latencies < 80ms.

[tebeco]

I suppose that as it's named "mobile-appsync" low latency was never a consideration but maybe reliability or other.
So you won't have it working for you

We was comparing overall offers between AWS and Azure at that time and never found a managed service on AWS side for that use case (well the only only was way way more expensive)
We ended up using this: https://docs.microsoft.com/en-us/azure/azure-signalr/
It does not have the same idea/feature in mind, not ACK for example so if a client drop / reconnect you can loose "signals" (not "message" because it's not "messaging")

Also the fact that having to use github and then other forum / site etc .... did not helped for that it's really discouraging to be redirected or split in multiple places and then you're just sharing link that you hope will never break

[Jaftem]

2022 - seeing 200-350 ms. This still isn't acceptable for real-time apps that need <50ms.

[JKKholmatov]

Appsync GraphQL query sometimes takes 3 minutes to be executed from client app, X-ray shows ~200mls performance, I assume that the problem is the client app, it sends request after 3 minutes, because after function call network doesn't send or accept any kb for a long time, is it connection lost between client and server or what?