Streaming not working on Heroku install #1119

Closed
ashfurrow opened this Issue Apr 7, 2017 · 16 comments

Comments

Projects
None yet
9 participants
@ashfurrow
Collaborator

ashfurrow commented Apr 7, 2017

Hi there, running into some issues with streaming not working on my instance. I'm hosting on Heroku, and I'm seeing the following 404 errors:

Apr 06 17:25:44 mastodon-technology heroku/router:  at=info method=GET path="//api/v1/streaming/?access_token=...&stream=public" host=mastodon.technology request_id=... fwd="..." dyno=web.1 connect=1ms service=20ms status=404 bytes=1566 protocol=https 
...
Apr 06 17:25:44 mastodon-technology app/web.1:  [...] method=GET path=/api/v1/streaming format=html controller=ApplicationController action=raise_not_found status=404 duration=12.25 view=0.38 db=3.92 

I took a look at the code but can't figure it out. I did notice the // at the beginning of the api call. Could that be it? It looks similar to #809 and #1016 but I'm hosting on Heroku and not Apache. Thanks for any tips 🙇


  • I searched or browsed the repo’s other issues to ensure this is not a duplicate.
@Gargron

This comment has been minimized.

Show comment
Hide comment
@Gargron

Gargron Apr 7, 2017

Member

I am not sure. @ineffyble please help. How do we add a streaming API process (node.js) to the Heroku deployments?

Member

Gargron commented Apr 7, 2017

I am not sure. @ineffyble please help. How do we add a streaming API process (node.js) to the Heroku deployments?

@Spunkie Spunkie referenced this issue Apr 7, 2017

Closed

External user avatars not working. #1126

1 of 1 task complete
@ineffyble

This comment has been minimized.

Show comment
Hide comment
@ineffyble

ineffyble Apr 8, 2017

Collaborator

I'll have a look now. That said, from my experiences, Heroku deployments utilising only free dynos aren't viable for a production instance.

Collaborator

ineffyble commented Apr 8, 2017

I'll have a look now. That said, from my experiences, Heroku deployments utilising only free dynos aren't viable for a production instance.

@ashfurrow

This comment has been minimized.

Show comment
Hide comment
@ashfurrow

ashfurrow Apr 8, 2017

Collaborator

I appreciate it 🙇 If it helps, I'm using a 2x Dyno and have paid for upgrades to redis abd postgres addons. Let me know if no can provide any other details.

Collaborator

ashfurrow commented Apr 8, 2017

I appreciate it 🙇 If it helps, I'm using a 2x Dyno and have paid for upgrades to redis abd postgres addons. Let me know if no can provide any other details.

@ecmendenhall

This comment has been minimized.

Show comment
Hide comment
@ecmendenhall

ecmendenhall Apr 9, 2017

Hi all! Based on the advice from @pixeldesu over here, I was able to get the streaming API up and running on Heroku by deploying two separate apps.

As others have noted, Rails will catch requests to /api/v1/streaming/ that are meant for Node unless there's a proxy like nginx in front. Deploying the streaming API to a separate domain with shared Redis/DB creds and setting STREAMING_API_BASE_URL to the second app's config works for me. FWIW, I'm using Heroku SSL, not Cloudflare. Here's a breakdown of how things are configured between the two:

Main app:

  • Custom domain and (Heroku) SSL setup.
  • Postgres/Redis addons.
  • Procfile: repo defaults (web runs puma, worker runs sidekiq)
  • Buildpacks: repo defaults (Ruby and Node)
  • Set STREAMING_API_BASE_URL to point to the second app's herokuapp.com URL.

Streaming API app:

  • No custom domain, just https://nature-words-11234.herokuapp.com
  • No addons.
  • Procfile: just web: npm start
  • Buildpacks: just heroku/nodejs. Set it explicitly with $ heroku buildpacks:set heroku/nodejs
  • Set the following env vars. Heroku addons usually use full connection strings like redis://username:password@redis-hostname.example.com:1234/dbname, so you'll have to deconstruct some of them from REDIS_URL and DB_URL:
    • DB_USER
    • DB_PASS
    • DB_NAME
    • DB_HOST
    • DB_PORT
    • REDIS_HOST
    • REDIS_PORT
    • REDIS_PASSWORD

ecmendenhall commented Apr 9, 2017

Hi all! Based on the advice from @pixeldesu over here, I was able to get the streaming API up and running on Heroku by deploying two separate apps.

As others have noted, Rails will catch requests to /api/v1/streaming/ that are meant for Node unless there's a proxy like nginx in front. Deploying the streaming API to a separate domain with shared Redis/DB creds and setting STREAMING_API_BASE_URL to the second app's config works for me. FWIW, I'm using Heroku SSL, not Cloudflare. Here's a breakdown of how things are configured between the two:

Main app:

  • Custom domain and (Heroku) SSL setup.
  • Postgres/Redis addons.
  • Procfile: repo defaults (web runs puma, worker runs sidekiq)
  • Buildpacks: repo defaults (Ruby and Node)
  • Set STREAMING_API_BASE_URL to point to the second app's herokuapp.com URL.

Streaming API app:

  • No custom domain, just https://nature-words-11234.herokuapp.com
  • No addons.
  • Procfile: just web: npm start
  • Buildpacks: just heroku/nodejs. Set it explicitly with $ heroku buildpacks:set heroku/nodejs
  • Set the following env vars. Heroku addons usually use full connection strings like redis://username:password@redis-hostname.example.com:1234/dbname, so you'll have to deconstruct some of them from REDIS_URL and DB_URL:
    • DB_USER
    • DB_PASS
    • DB_NAME
    • DB_HOST
    • DB_PORT
    • REDIS_HOST
    • REDIS_PORT
    • REDIS_PASSWORD
@ashfurrow

This comment has been minimized.

Show comment
Hide comment
@ashfurrow

ashfurrow Apr 9, 2017

Collaborator

@ecmendenhall @pixeldesu thanks a lot for looking into this! I've followed the instructions above and can confirm it's working for me. One note: I had to specify STREAMING_API_BASE_URL to include the protocol https://.

Follow-up: now that the streaming is handled on another Dyno, do I still need NodeJS in my main app's build pack? I ask because I'd like to add ffmpeg to the buildpack but it puts me over the 300MB limit, and removing NodeJS would help. Is Node required for anything except the streaming server?

(Edit: I've tried removing NodeJS from the main app's buildpack and things seem okay. Will monitor.)

I'm running the streaming server on a hobby Dyno, not sure how well it will scale but I'll report back in a few days 👍

Thanks again 🙇

Collaborator

ashfurrow commented Apr 9, 2017

@ecmendenhall @pixeldesu thanks a lot for looking into this! I've followed the instructions above and can confirm it's working for me. One note: I had to specify STREAMING_API_BASE_URL to include the protocol https://.

Follow-up: now that the streaming is handled on another Dyno, do I still need NodeJS in my main app's build pack? I ask because I'd like to add ffmpeg to the buildpack but it puts me over the 300MB limit, and removing NodeJS would help. Is Node required for anything except the streaming server?

(Edit: I've tried removing NodeJS from the main app's buildpack and things seem okay. Will monitor.)

I'm running the streaming server on a hobby Dyno, not sure how well it will scale but I'll report back in a few days 👍

Thanks again 🙇

@ashfurrow

This comment has been minimized.

Show comment
Hide comment
@ashfurrow

ashfurrow Apr 10, 2017

Collaborator

Oh, one update: I did need to set PGSSLMODE to require to force SSL for the streaming Dyno, to force Heroku to connect using SSL. Hat tip to this SO question.

Collaborator

ashfurrow commented Apr 10, 2017

Oh, one update: I did need to set PGSSLMODE to require to force SSL for the streaming Dyno, to force Heroku to connect using SSL. Hat tip to this SO question.

@ashfurrow ashfurrow referenced this issue Apr 10, 2017

Closed

GIF upload fails on Heroku instances #1007

1 of 1 task complete

@Floppy Floppy referenced this issue Apr 10, 2017

Closed

Streaming not working on Heroku #1431

1 of 1 task complete
@Floppy

This comment has been minimized.

Show comment
Hide comment
@Floppy

Floppy Apr 10, 2017

Contributor

I had a short conversation with Heroku support on this: https://twitter.com/xavriley/status/851448436473958400. I'm not sure what the solution is, but this is the documentation referred to: https://devcenter.heroku.com/articles/node-websockets#option-2-socket-io

Contributor

Floppy commented Apr 10, 2017

I had a short conversation with Heroku support on this: https://twitter.com/xavriley/status/851448436473958400. I'm not sure what the solution is, but this is the documentation referred to: https://devcenter.heroku.com/articles/node-websockets#option-2-socket-io

@Floppy

This comment has been minimized.

Show comment
Hide comment
@Floppy

Floppy Apr 10, 2017

Contributor

I think it means we basically need one app to proxy to the other if it gets traffic not intended for it.

Contributor

Floppy commented Apr 10, 2017

I think it means we basically need one app to proxy to the other if it gets traffic not intended for it.

@ashfurrow

This comment has been minimized.

Show comment
Hide comment
@ashfurrow

ashfurrow Apr 10, 2017

Collaborator

Thanks for contacting Heroku @Floppy, that's a great idea. I'm not sure how to do proxy traffic to a specific Dyno, hopefully they'll follow-up with more details. Keep us in the loop!

Collaborator

ashfurrow commented Apr 10, 2017

Thanks for contacting Heroku @Floppy, that's a great idea. I'm not sure how to do proxy traffic to a specific Dyno, hopefully they'll follow-up with more details. Keep us in the loop!

@Floppy

This comment has been minimized.

Show comment
Hide comment
@Floppy

Floppy Apr 10, 2017

Contributor

One more tip to follow #1119 (comment) - make the Procfile change on a separate branch (like streaming) in your mastodon fork, then you can set up heroku to autodeploy both master and streaming in your two apps.

Contributor

Floppy commented Apr 10, 2017

One more tip to follow #1119 (comment) - make the Procfile change on a separate branch (like streaming) in your mastodon fork, then you can set up heroku to autodeploy both master and streaming in your two apps.

@Floppy

This comment has been minimized.

Show comment
Hide comment
@Floppy

Floppy Apr 10, 2017

Contributor

And one more. NODE_ENV=production is required in the ENV.

Contributor

Floppy commented Apr 10, 2017

And one more. NODE_ENV=production is required in the ENV.

@Spunkie

This comment has been minimized.

Show comment
Hide comment
@Spunkie

Spunkie Apr 10, 2017

So as far as I understand when people say streaming they are just talking about websockets and the only thing these websockets are used for is live updates to my timelines and possibly on profiles? Is that right?

Maybe I'm totally off base here but for a single user instance on a free heroku dyno wouldn't it be much cheaper and simpler for me to fallback to an ajax solution? On profiles every minute or so ajax to the current page and then replace the body of the page with the body from them the ajax request? The same thing could be done for my timelines with some additional code to ensure it does not refresh the page if I've started writing a reply/toot.

Spunkie commented Apr 10, 2017

So as far as I understand when people say streaming they are just talking about websockets and the only thing these websockets are used for is live updates to my timelines and possibly on profiles? Is that right?

Maybe I'm totally off base here but for a single user instance on a free heroku dyno wouldn't it be much cheaper and simpler for me to fallback to an ajax solution? On profiles every minute or so ajax to the current page and then replace the body of the page with the body from them the ajax request? The same thing could be done for my timelines with some additional code to ensure it does not refresh the page if I've started writing a reply/toot.

@Floppy

This comment has been minimized.

Show comment
Hide comment
@Floppy

Floppy Apr 10, 2017

Contributor

Yep, I think that's right. Having it all handled by one app would certainly be simpler for Heroku deployments, but that's not the way the app is set up at the moment.

Mastodon could either (1) have an AJAX fallback, or (2) merge the websocket server into the rails server; both would mean you wouldn't need the extra server, but I'll let @Gargon pass judgement on whether either would make sense or be desirable from his POV :)

Contributor

Floppy commented Apr 10, 2017

Yep, I think that's right. Having it all handled by one app would certainly be simpler for Heroku deployments, but that's not the way the app is set up at the moment.

Mastodon could either (1) have an AJAX fallback, or (2) merge the websocket server into the rails server; both would mean you wouldn't need the extra server, but I'll let @Gargon pass judgement on whether either would make sense or be desirable from his POV :)

@ashfurrow

This comment has been minimized.

Show comment
Hide comment
@ashfurrow

ashfurrow Apr 22, 2017

Collaborator

I'm going to close this issue since we've figured out the issue. I don't think falling back to AJAX requests is probably not the highest priority, but it's worth considering. @Floppy can you open a new issue?

Collaborator

ashfurrow commented Apr 22, 2017

I'm going to close this issue since we've figured out the issue. I don't think falling back to AJAX requests is probably not the highest priority, but it's worth considering. @Floppy can you open a new issue?

@SerenadeX

This comment has been minimized.

Show comment
Hide comment
@SerenadeX

SerenadeX May 15, 2018

Looking for a way to set this up on my heroku server, following the steps doesn't make it work though. I assume a lot has changed over a year though. Are any of you still able to set up the streaming api with heroku?

Looking for a way to set this up on my heroku server, following the steps doesn't make it work though. I assume a lot has changed over a year though. Are any of you still able to set up the streaming api with heroku?

@Kjwon15

This comment has been minimized.

Show comment
Hide comment
@Kjwon15

Kjwon15 May 15, 2018

Contributor

@SerenadeX Maybe uws is the problem.
I use heroku(heroku alternative) and node 10.x cannot install uws.

uws offers pre-compiled binary, if there are no compatible binary, it tries to compile. And the problem is that compilation is always failing.

If use fix node version to 8.x you might be run without problem.

Contributor

Kjwon15 commented May 15, 2018

@SerenadeX Maybe uws is the problem.
I use heroku(heroku alternative) and node 10.x cannot install uws.

uws offers pre-compiled binary, if there are no compatible binary, it tries to compile. And the problem is that compilation is always failing.

If use fix node version to 8.x you might be run without problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment