graphql-subscriptions is not working consistently.

[ShaharDadon]

Hi all,
We are using PubSub in our application in this manner:

When deletion event is arriving into server:
pubsub.publish('onDeleteItem', {onDeleteItem: [msg.payload.ItemId]});

Resolvers:
Subscription: { onDeleteItem: { subscribe: () => pubsub.asyncIterator('onDeleteItem'), }, },

Subscribing to it from our client side:
const onDeleteItemSubscription = gql

          subscription onDeleteItem {
                onDeleteItem
  }

getItemsList () {
  queryRef.subscribeToMore({
      document: onDeleteItemSubscription ,
      updateQuery(prev: { itemList: AssetList }, {subscriptionData: {data}}) {
        if (!data) {
          return prev;
        } else {
          const deletedItems = (data as any).onDeleteItem;
          return {
                   itemList {...    ### //return filtered list
                       }
            } 
          } 
       }
    }

;

The problem is that sometimes the subscription is working fine and the deleted item is filtered as expected, but very often it just not working at all. There are no error logs.
When I debug it, I can see that the problem is that the client is not notified by pubsub.publish()

The problem is occurring with all our subscriptions and this is only one example of them.

What am I missing here?
Thanks.

[grantwwu]

Would it be possible for you to provide a runnable example of the issue? It's kind of vague right now...

[ShaharDadon]

Hi @grantwwu , thank you for your fast answer.
Unfortunately, it is not really possible for me to provide a running example.
The subscriptions are done through WS. for some reason, sometimes some of the subscriptions not registered. It is pretty much random.
Maybe this is kind of network problem?

Connection to WS

`export function apolloLinkFactory(httpLink: HttpLink, injector: Injector) {
  const http = httpLink.create({uri: '/graphql'});
  const type = isDevMode() ? 'ws' : 'wss' ;
  const ws = new WebSocketLink({
    uri: `${type}://${location.host}/subscriptions`,
    options: {
      reconnect: true,
      lazy: true,
      connectionParams() {
        return {
          authToken: injector.get(IamService).wsToken,
        };
      }
    }
  });`

[svrpeakers]

Having this same issue, sometimes the subscription just doesn't trigger on the app end

[ShaharDadon]

Hi @svrpeakers, thanks for your reply.
Did you manage to figure out why is that?

[JonathanCallewaert]

We had something similar. Everything worked local, but not on production.
The problem was that we had 2 scaled server instances running. So it only worked 50% of the time because the Pubsub has to be on the same server instance

[svrpeakers]

So yes, my issue ended up being we weren't' closing subscriptions when we closed the component. So we would keep opening new subscriptions and the server couldn't manage all the subscriptions. (all the subscriptions came in the same socket)

[ShaharDadon]

@svrpeakers It sounds very reasonable and I think this is the same problem in our project.
Thank you!

@JonathanCA97 Good point, we've experienced it too.

[TimSusa]

We still have strange issues here in production causing from memory-leaks (using Websockets) and gentleman, it is really not a pleasure to fix such a problem. We ended up by making use of polling as a first hot fix, furthermore we have noticed same strange behaviour in our connector implementation axelspringer/graphql-google-pubsub#16. Contributors are scratching behind their heads, developers are catching after ghosts. From our side, we do not trust the whole async iterator stuff at the moment.

[svrpeakers]

so yeah, we also still have this issue in production, closing subscriptions helped but didn't resolve the problem completely. we also had to add a polling option to our production app.

[taozhi8833998]

having the same issue....

[vladoro]

We have same issue in production with the following stack:

• redis pubsub graphql-redis-subscriptions
• ioredis
• apollo-server
• apollo-client
• react

Redis & apollo server are on separate VMs.

[benxtsai]

I am having the same issue.

Are there any fallback mechanisms that I would be able to implement to refrain this from happening or this is not the correct use case for reliable messaging?

[taozhi8833998]

We had something similar. Everything worked local, but not on production.
The problem was that we had 2 scaled server instances running. So it only worked 50% of the time because the Pubsub has to be on the same server instance

maybe we have the same root cause, how you fixed this???

[andrew-ironforge]

I am finding that when running locally behind an ngrok tunnel and using the PubSub from 'graphql-subscriptions', my server will notify all connected clients when events are published.

When I push my code to an Elastic Beanstalk, queries and mutations will work fine, but clients are not notified of events. On the server, I use the RedisPubSub from 'graphql-redis-subscriptions' library.

Would welcome any advice on how to fix this.

[mogarick]

hi @andrew-ironforge @vladoro did you solved your problem?
I'm having

"Could not connect to websocket endpoint ws://:/graphql. Please check if the endpoint url is correct.".

Same occurs whe using wss. +
Apollo GraphQL Server runs in a container and Redis is an external service.

[andrew-ironforge]

@mogarick no I never fixed my problem. I think we are having different issues though because your URL does not look valid.

[mogarick]

@mogarick no I never fixed my problem. I think we are having different issues though because your URL does not look valid.

Sorry @andrew-ironforge , it looks the url got lost in formatting. the url is for example ws://myhost.com:4000/graphql

So what did you do? changed from RedisPubsub lib to other one or something?
Thank you for yor attention. :)

[andrew-ironforge]

@mogarick I never actually figured it out. When I ran my API locally, subscriptions would work fine, but ceased to work when running in a docker container on the elastic beanstalk.

It might also be important to note that the client is an iOS application; however I don't believe this matters.

[BerenLuth]

Same problem here, subscriptions are not working properly.

I just added a new collection that manages comments on my project, but I can't find a way to make it work properly.
I have several collections already, all working fine with many subscriptions, but this new one I just added, with basically the same code, notifies only the last client connected.

I also tried to remove all the other subscriptions to see if there are too many of them, but nothing changes

[arnasledev]

having the same issue on AWS. using multiple instances and it works perfectly local but not in production. Using graphql-redis-subscriptions and ioredis for multi instances.

import { RedisPubSub } from 'graphql-redis-subscriptions';
import Redis from 'ioredis';

const options = {
  host: process.env.REDIS_HOST,
  port: process.env.REDIS_PORT,
  password: process.env.REDIS_PASSWORD,
  retryStrategy: times => {
    return Math.min(times * 50, 2000);
  }
};

const pubsub = new RedisPubSub({
  publisher: new Redis(options),
  subscriber: new Redis(options)
});

export default pubsub;

the error I see in the playground:

{
"error": "Could not connect to websocket endpoint wss://.../graphql. Please check if the endpoint url is correct."
}

[nickwalton]

Also having the same issue where subscriptions work fine locally, but don't update over wss...

[evenfrost]

Same here with PM2 cluster mode and graphql-redis-subscriptions over Redis. Subscriptions still work on single node instead of all.

[Dajust]

August 11, 2020: Did anyone ever figured this?

It used to work well (sometimes) but has currently stopped working for the chat section of my app, even on local.

[fadulalla]

It's very weird, it used to work well here, too, on both production and local. But a couple of weeks ago, it broke on one of my production servers, and on my dev environment.

My client will say "connected" but I receive no messages. And if I subscribe through apollo's Playground, I get the following message:

{
  "error": {
    "message": "Subscription field must return Async Iterable. 
     Received: {
        pubsub: {
            triggerTransform: [function],
            reviver: undefined,
            redisPublisher: [Redis],
            redisSubscriber: [Redis],
            subscriptionMap: [Object],
            subsRefsMap: [Object],
            currentSubscriptionId: 26
        },
        options: undefined,
        pullQueue: [],
        pushQueue: [],
        listening: true,
        eventsArray: ['TEST'],
        allSubscribed: {}
    }
}

but chrome stays connected, and inspecting the network request, I see that I continue to receive keep-alive messages:

@Dajust can you please check if you get the same error message?

I don't understand how something can break on its own. My resolver, just in case:

  resolvers: {
        Subscription: {
            ...,
            Test: {
                subscribe: () => redis.asyncIterator("TEST"),
                resolve: (testString) => testString
            },
        },
    }

redis is just import { RedisPubSub } from 'graphql-redis-subscriptions';

---

edit: weirdly enough, it works on local. Same code, same dependencies (I didn't mock redis, just ran a local redis db, I didn't mock anything).

[skjorrface]

I stumbled upon the same problem yesterday. I'm using apollo-server locally so, to provide https with certbot to the "outside" I am using dmz + nginx as a proxy server. When using this configuration subscriptions didn't work at all, while, instead, everything worked fine with ngrok! After some research it turns out one has to configure nginx (or whatever proxy server/http server) to handle the websocket protocol (it can be handled via http/s either, but it has to be configured anyways). In my case, adding these lines to nginx.conf solved the problem:

`

      proxy_set_header Upgrade $http_upgrade;

      proxy_set_header Connection "Upgrade";

      proxy_set_header Host $host;

`

I hope this helps.

[fadulalla]

Okay so I sort of found out what my issue was, and it wasn't headers. Strangely, node version seems to have been the culprit.

Locally I was working with 12.16.1, or so node --version says. My dev ec2 instance was on 14. I compared the object returned by redis.asyncIterator("TEST") locally and on my instance, and they were almost the same except for a few stuff, including a property that was missing on the server: [Symbol(kCapture)]: false.

It's slightly confusing though because sudo node --version returns 14, and without sudo it returns 8.10 (ec2 instance). So to be honest, I'm not sure which one my app was using using. I found this: nodejs/node#31787 so I just decided to test it. I installed the same node version on the server as my local, 12.16.1 and forced it to use it for both root and non-root, and voila! Subscriptions started working again.

I found where the function that throws the error above (https://github.com/graphql/graphql-js/blob/master/src/subscription/subscribe.js#L287) and how they determine whether an object is an async iterator or not (https://github.com/graphql/graphql-js/blob/35f6df8693eaf9f484df8566f752a515aee4893b/src/jsutils/isAsyncIterable.js#L11).

I'm not a js developer so I don't know what any of this means, but it's working now after explicitly setting node to 12.16.1.

Very scientific analysis, I know.

[ahmad-punch]

Same Issue here.
Subscriptions work sometime and other times they just refuse to connect and show the following error:
{
"error": "Could not connect to websocket endpoint ws://****/graphql. Please check if the endpoint url is correct."
}

[ahmad-punch]

Any Progress here?
Subscriptions work fine on local but on production (Google Compute engine- they fail.)

[xino1010]

Any Progress here?

[ricardoribas]

Any progress? I wonder if there is any workaround while we wait for the problem to be solved?

[stanyq4]

Bumping this up as we are experiencing similar behaviour with some of the clients missing subscription messages without a reproducible pattern.

[BeauGieskens]

Our team gave up on waiting for a fix here and made the switch to graphql-ws for our subscriptions, but still using the PubSub from this package. We have not this issue since, and it only took a little bit of effort to adapt the client code, since they have an example for just about every client you can imagine. The main drawback is that the GraphQL playground/sandbox doesn't support it natively yet.

[guru-pi-xcels]

@mogarick I never actually figured it out. When I ran my API locally, subscriptions would work fine, but ceased to work when running in a docker container on the elastic beanstalk.

It might also be important to note that the client is an iOS application; however I don't believe this matters.

Did you solved your problem?

I'm facing the same issue, only in local subscriptions working fine for me.

[fierysolid]

If Person A has a socket connection to Container A, Person B has a socket connection to Container B, and Person A sends Person B a message, how does that message get routed to Container B if Container A only knows about its own connections?

The answer is using a Pub/Sub service that lives outside of the docker containers like Cloud Pub/Sub or Redis Pub/Sub that allows the Containers to send messages to each other so they can notify any clients that may need that message.

The Pub/Sub from this package will only work locally with 1 running instance of your graphql server since it's just using node event emitter.

[stronglify13]

Any Progress here?
Subscriptions work fine on local but on production AWS, fail

[mhdazl]

Did anyone find any solution?. I am having the same issue.Every thing fine in local, But not in production (AWS)

[guru-pi-xcels]

Did anyone find any solution?. I am having the same issue.Every thing fine in local, But not in production (AWS)

Note that the default PubSub implementation is intended for demo purposes. It only works if you have a single instance of your server and doesn't scale beyond a couple of connections. For production usage you'll want to use one of the PubSub implementations backed by an external store. (e.g. Redis) - from graphql-subscriptions doc

[ngdthinh97]

I stumbled upon the same problem yesterday. I'm using apollo-server locally so, to provide https with certbot to the "outside" I am using dmz + nginx as a proxy server. When using this configuration subscriptions didn't work at all, while, instead, everything worked fine with ngrok! After some research it turns out one has to configure nginx (or whatever proxy server/http server) to handle the websocket protocol (it can be handled via http/s either, but it has to be configured anyways). In my case, adding these lines to nginx.conf solved the problem:

`

      proxy_set_header Upgrade $http_upgrade;

      proxy_set_header Connection "Upgrade";

      proxy_set_header Host $host;

`

I hope this helps.

Hi @skjorrface , it is work well for me thank you

[eric-ho-github]

Any Progress here?
I'm having the same problem.