Ably Not Subscribing to Events in Production

[coreyarch4321]

I shipped an application to production using Vercel and Ably. In short, on a button click, I publish an event to a channel from my react client. On my node backend, I am subscribed to that channel for that specific event. In my local setup, the event listener triggers perfectly, but in production, the event never runs. I've used the Ably dev console to publish the same event on that channel and that still doesn't work.

I've used log: { level: 4 } to output verbose logs using Axiom and it seems that ably connects to the proper channel on the backend.

Here is the problem in more detail, with files attached for the logs, ably dev console and code snippets.

FRONT END:

• On the frontend, I click a button called "Find a match". This opens up the "FindingMatchDialog.js" (see code snippet). Within the dialog, 3 major steps happen
  • We attach to the channel, "all_users_matching_channel". All attaching on the frontend is done through the "useChannel.js" hook (see code snippet).
  • We attach to the channel, "match-finding-user-${user.user_id}". In this case, the user_id is 1, so the channel becomes, "match-finding-user-1".
    • Notice that it is on this channel that we listen to the event, "match_not_found".
  • A timer is set, so that after 3 seconds, we publish an event to the "all_users_matching_channel". This event is called, "match_request".

Note: The expected outcome is that ably will respond to the "match_not_found" event that is published on the "match-finding-user-1" channel from the backend. You can see that in "FindingMatchDialog.js", if ably were to respond to the "match_not_found" event, we would get a console.log of "listening to Match not found". The issue is that this never gets logged.

BACK END:

• On the backend, we define the "/token-request" API route that returns the ably instance for the frontend to use (see "routes/ably/index.js" code snippet).

• We have an "ably-service" file that runs an "initialize" function. This connects to ably.

• The initialize function also runs the function, "setUpAllUsersChannelSubscriptions". This function connects to the "all_users_matching_channel" channel (recall from the frontend). We set up the subscription on that channel for the event, "match_request". In that subscription handler, we run the function, "mockOnMatchRequest" function.

  • In the "mockOnMatchRequest", we retrieve the "match-finding-user-1" channel and publish the "match_not_found" event.

It is this "match_not_found" event that should then be listened to on the front end. So here is the issue:

THE ISSUE:

The "match_not_found" is never listened to on the frontend. It seems like the "mockOnMatchRequest" on the backend is not run at all. None of the logs are outputted.

It turns out that after the button press, no new backend logs appear. It's as if the "match_request" event is not being listened to on the backend. I've attached screenshots of the Ably dev console, where I've attached to the "all_users_matching_channel" and the "match-finding-user-1" channel. You'll notice that the "match_request" event does get published (as seen in the Ably dev console). However, the "match_not_found" never shows up on the Ably dev console. This makes me believe there is something wrong with the subscription to the "all_users_matching_channel" for the event, "match_request".

You have a copy of the JS code where I subscribe to the "all_users_matching_channel" on the backend, so please let me know if something is being done incorrectly there.

MY HUNCHES:

• I don't think that the backend is subscribing to the "all_users_matching_channel" properly for the event, "match_request". Because if it were, it would run the "mockOnMatchRequest" function and the frontend would listen to the "match_not_found" event that gets published.
• Sometimes, when I refresh my screen, then the "match_not_found" event gets triggered. It's as if these chain of events were stored somewhere, waiting for a refresh. It's confusing. I'm not so sure why this happens.

I hope this abundance of information is not overwhelming. Please take a look at how I am connection to ably, publishing/subscribing to the events (especially on the backend). Hopefully, you can help me figure out what the issue is. Also, this is all working fine on my local environment, so that makes it a little tougher.

Thank you! And let me know if you need any more information or clarification.

Here are the code snippets:

FindingMatchDialog.js

**

...
import { useChannel } from '../../hooks/useChannel';
//
----------------------------------------------------------------

export default function FindingMatchDialog({ isOpen, onClose,
imgSrc, user, question }) {

const [allUsersMatchChannel, allUsersMatchChannelAbly] = useChannel({
   channelName: ALL_USERS_MATCHING_CHANNEL,
});

const [userChannel, userChannelAbly] = useChannel({ 
   channelName: `match-finding-user-${user.user_id}`,
   eventNames: [MATCH_FOUND, MATCH_NOT_FOUND],
   callbackOnMessageMap: {
      [MATCH_FOUND]: (message, _channel) => {
          const data = message.data;
          navigate(PATH_APP.conversations.view(data.match_id));
    },
     [MATCH_NOT_FOUND]: (message, _channel) => {
        console.log("listening to Match not found")
       openNoMatchDialog();
    }
},
});

useEffect(() => {
   const matchRequestTimer = setTimeout(() => {
      console.log("Sending out match request!!")
      allUsersMatchChannel.publish(MATCH_REQUEST, {
          question_id: question.question_id,
          user_id: user.user_id,
 });
}, 3000);

return () => {
   clearTimeout(matchRequestTimer);
}
}, [])

...

}

useChannel.js

**

import Ably from "ably/promises";

export const ably = new Ably.Realtime.Promise({
   authUrl: `${API_URL}/ably/token-request`,
   log: { level: 4 }
});

export function useChannel({ channelName, eventNames, callbackOnMessageMap, handleUnmount, effectListenerProps = [] })
{
   const channel = ably.channels.get(channelName);
   
   const onMount = async () => {
      if (eventNames) {
         console.log("subscribing channelName: ", channelName)
         
        for (const eventName of eventNames) {
            channel.subscribe(eventName, (msg) => {
                callbackOnMessageMap[eventName](msg, channel);
        });
}
}
}

const onUnmount = () => {
   console.log("unsubscribing channelName: ", channelName)
   
   channel.unsubscribe();
   
   if (handleUnmount) {
       handleUnmount();
   }
}

const useEffectHook = () => {
    onMount();
    
    return () => { onUnmount(); };
};

useEffect(useEffectHook, [...effectListenerProps]);

return [channel, ably];
}

ably-service/index.js

**

...

class AblyService {

ably;

initialize = async (passedTrx = null) => {
   let trx;

    try {
        log.debug('ABLY INITIALIZE', { testData: 32423 })
        
        const realtime = new Ably.Realtime({
            key: process.env.VERCEL_ENV === "production" ? process.env.ABLY_API_KEY_PRODUCTION : 
            process.env.ABLY_API_KEY_DEVELOPMENT,
            log: { level: 4 },
      });

       this.ably = realtime;
       this.ably.connection.once("connected");
       console.log("NOW Connected to Ably!!");

      trx = await startTrx(Match, passedTrx);
      await this.setUpAllUsersChannelSubscriptions({});
 
      await commitTrx(trx, passedTrx);
} catch (err) {
    console.error({
       filename,
       function: "initialize",
       message: `Failed to initialize ably service: ${err}`,
    });

    await rollbackTrx(trx, err);
    throw err;
}
}

setUpAllUsersChannelSubscriptions = async ({ passedTrx = null }) => {
   let trx;

    try {
        trx = await startTrx(Match, passedTrx);

     // get the finding match channel
    const allUsersMatchChannel = this.ably.channels.get(ALL_USERS_MATCHING_CHANNEL);

    await allUsersMatchChannel.subscribe(MATCH_REQUEST, async (message) => {
       console.log("GOT MATCH REQUEST!!! message: ", message)

      const currentUserChannel = this.ably.channels.get(`user-${message.data.user_id}`);

      await this.mockOnMatchRequest({ 
          user_id: +message.data.user_id,
          question_id: +message.data.question_id, 
          currentUserChannel: currentUserChannel,
      });
})

    await commitTrx(trx, passedTrx);
} catch (err) {
    console.error({
       filename,
       function: "setUpAllUsersChannelSubscriptions",
       message: `Failed to set up finding match channel subscriptions: ${err}`
,
});
await rollbackTrx(trx, err);
throw err;
}
}

mockOnMatchRequest = async ({ user_id, question_id, currentUserChannel, passedTrx = null }) => {
   console.log('mockOnMatchRequest! SENDING MATCH NOT FOUND')

   const leftUserChannel = this.ably.channels.get(`match-finding-user-${user_id}`);

   await leftUserChannel.publish(MATCH_NOT_FOUND, {});
   }
}

const ablyService = new AblyService();
export default ablyService;

routes/ably/index.js

**

...

router.get("/token-request", async (_req, res) => {
   try {
      const client = new Ably.Realtime({
          key: process.env.VERCEL_ENV === "production" ?
          process.env.ABLY_API_KEY_PRODUCTION :
          process.env.ABLY_API_KEY_DEVELOPMENT,
          log: { level: 4 },
});

const tokenRequestData = await client.auth.createTokenRequest({ 
    clientId: process.env.VERCEL_ENV === "production" 
    ? process.env.ABLY_CLIENT_ID_PRODUCTION 
    : process.env.ABLY_CLIENT_ID_DEVELOPMENT });

return res.status(StatusCodes.OK).json(tokenRequestData);
} catch (err) {
   return res.status(StatusCodes.INTERNAL_SERVER_ERROR).send({ err: (err.message || err) });
 }
});

export default router;

FindingMatchDialog.js.pdf
useChannel.js.pdf
ably-service-index.js.pdf
routes-ably-index.js.pdf
backend-end-logs.pdf
front-end-logs.pdf
ably-dev-console.pdf

[sync-by-unito]

➤ Automation for Jira commented:

The link to the corresponding Jira issue is https://ably.atlassian.net/browse/SDK-4041

[mattheworiordan]

Hey @coreyarch4321. I saw your note on Twitter so having a quick look before our team gets back to you. One thing that I think would be useful for us is for you to have a think what is different between your local and production environments. Ably does not differentiate production/dev/local development in any way whatsoever. Some customers use separate apps for production and dev, but on our end they're just apps, and we treat all traffic and client connections the same. So whilst we will certainly try and help resolve this for you, knowing more about what's different between these environments may help to more quickly resolve this issue. Can you have a think about what may be different between the two that is relevant?

[coreyarch4321]

Hi @mattheworiordan . Thanks for looking into this. Here are some differences that may be worth mentioning:

• Since the data stored in production vs local is different, the channel names point to totally different datasets. For instance, one channel name is, "match-finding-user-<user_id>". For user with user_id = 1, this would be, "match-finding-user-1" for both local and production. Even though the channel names are the same, we are pointing to two completely different users.

• On the frontend, I connect to ably using the token method (by making a request to ${API_URL}/ably/token-request). I also connect to ably on the node backend. The domain names are different in production than they are in local. My backend domain name is actually a subdomain of my frontend domain name. I needed to set it up this way because when they were different domain names, configuring cookies became much more difficult. I'm not so sure if the domain names would have any impact with the ably setup, but it's worth mentioning.

I can't really think of anything else worth mentioning. Please let me know if you have any other questions.

[ttypic]

Hi @coreyarch4321,

Here are the outcomes of the internal investigation into the issue:

• Vercel's backend runs as a serverless function.
• The backend subscribes to the Ably Realtime channel.
• However, Ably Realtime is not suitable for use in a serverless environment. In such cases, it is recommended to use Ably Rest instead.

The issue arises from the fact that Ably Realtime's connections are persistent and maintain state over time, while serverless functions are designed to be stateless and short-lived.

If you have any further questions or require additional clarification, feel free to ask.

I will leave this issue open and suggest that we work on improving the documentation to provide more specific guidance on using Ably in serverless environments.

[ttypic]

@coreyarch4321 it's also worth mentioning that you can utilize Ably's Webhook functionality in serverless environments.

[coreyarch4321]

Thank you all. The fix works. For anyone else running into the problem, here is a TLDR of the issue and the solution I'm using:

The issue was that Vercel has a default execution timeout for their backend functions (serverless functions). That means that within 10 seconds or so, the ably connection would die out. This is why the backend subscriptions were not triggering.

My solution is to send a regular HTTP request to the backend instead of publishing. Then, on the backend, instead of publishing via ably realtime, I'll simply use the REST API (which does not require a persistent connection to Ably). This has worked fine for me. Hopefully this helps someone else.

The important takeaway is - if your backend subscriptions aren't working in a Vercel production environment, this is probably the issue^.