New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can I actually deploy a cgi script to something like Heroku? #1

Open
varjmes opened this Issue Jan 6, 2015 · 13 comments

Comments

Projects
None yet
4 participants
@varjmes
Owner

varjmes commented Jan 6, 2015

~

@varjmes varjmes added the help wanted label Jan 6, 2015

@tomwhoiscontrary

This comment has been minimized.

Show comment
Hide comment
@tomwhoiscontrary

tomwhoiscontrary Jan 13, 2015

It's definitely possible, because you can use custom buildpacks on Heroku, which basically lets you do anything. But writing a custom buildpack could be quite an undertaking.

The answer here should work - run a persistent app that is a CGI server:

http://stackoverflow.com/questions/13520175/run-python-cgi-application-on-heroku

EDIT: Possibly your server.py could be run this way? Don't know enough about Python webserving to say.

And some loonbag has already written a custom buildpack which only does CGI - in shell script (turns out quite simple, but it's not exactly production-strength):

https://github.com/charliesome/heroku-buildpack-cgi

Actually using a custom buildpack is not too hard:

https://devcenter.heroku.com/articles/buildpacks

You can specify the buildpack in the app.json:

https://devcenter.heroku.com/articles/app-json-schema

Which means you can deploy it using a button on the GitHub page:

https://devcenter.heroku.com/articles/heroku-button

EDIT: But you should probably ignore buildpacks and try and get server.py running under the normal Python buildpack.

tomwhoiscontrary commented Jan 13, 2015

It's definitely possible, because you can use custom buildpacks on Heroku, which basically lets you do anything. But writing a custom buildpack could be quite an undertaking.

The answer here should work - run a persistent app that is a CGI server:

http://stackoverflow.com/questions/13520175/run-python-cgi-application-on-heroku

EDIT: Possibly your server.py could be run this way? Don't know enough about Python webserving to say.

And some loonbag has already written a custom buildpack which only does CGI - in shell script (turns out quite simple, but it's not exactly production-strength):

https://github.com/charliesome/heroku-buildpack-cgi

Actually using a custom buildpack is not too hard:

https://devcenter.heroku.com/articles/buildpacks

You can specify the buildpack in the app.json:

https://devcenter.heroku.com/articles/app-json-schema

Which means you can deploy it using a button on the GitHub page:

https://devcenter.heroku.com/articles/heroku-button

EDIT: But you should probably ignore buildpacks and try and get server.py running under the normal Python buildpack.

@varjmes

This comment has been minimized.

Show comment
Hide comment
@varjmes

varjmes Jan 13, 2015

Owner

@tomwhoiscontrary thank you Tom! I will try and investigate this tomorrow. The buildpack is tempting purely because I don't know how capable I'm going to be without it. We will see :)

Owner

varjmes commented Jan 13, 2015

@tomwhoiscontrary thank you Tom! I will try and investigate this tomorrow. The buildpack is tempting purely because I don't know how capable I'm going to be without it. We will see :)

@varjmes

This comment has been minimized.

Show comment
Hide comment
@varjmes

varjmes Jan 13, 2015

Owner

I spent the last few hours attempting to deploy this to Heroku and it was a complete failure.
Project will be local only until I become some kind of Heroku magician, so, never!

As for charliesome's cgi build-pack, it could well be what I need but without a README or any guidance to how that specific project works, I daren't use it.

Owner

varjmes commented Jan 13, 2015

I spent the last few hours attempting to deploy this to Heroku and it was a complete failure.
Project will be local only until I become some kind of Heroku magician, so, never!

As for charliesome's cgi build-pack, it could well be what I need but without a README or any guidance to how that specific project works, I daren't use it.

@tomwhoiscontrary

This comment has been minimized.

Show comment
Hide comment
@tomwhoiscontrary

tomwhoiscontrary Jan 13, 2015

:rage4:

Sounds like something to ignore for release 1, at least!

I'm not very familiar with Heroku (i've used Cloud Foundry, which is similar but different) and yeah, there are a lot of little nuances you need to somehow know about. Even more so if depoying CGI possibly.

I strongly suspect you can deploy your app without having to do anything special with buildpacks. You might have to change the server.py from the built-in web server to Flask. Would be great to find someone who's actually done this with Python and Heroku, ie not me (sorry).

tomwhoiscontrary commented Jan 13, 2015

:rage4:

Sounds like something to ignore for release 1, at least!

I'm not very familiar with Heroku (i've used Cloud Foundry, which is similar but different) and yeah, there are a lot of little nuances you need to somehow know about. Even more so if depoying CGI possibly.

I strongly suspect you can deploy your app without having to do anything special with buildpacks. You might have to change the server.py from the built-in web server to Flask. Would be great to find someone who's actually done this with Python and Heroku, ie not me (sorry).

@varjmes

This comment has been minimized.

Show comment
Hide comment
@varjmes

varjmes Jan 13, 2015

Owner

No need to apologise, you definitely gave me some things to look at, I'm just not knowledgeable enough yet to get it working. 👍

Owner

varjmes commented Jan 13, 2015

No need to apologise, you definitely gave me some things to look at, I'm just not knowledgeable enough yet to get it working. 👍

@pkqk

This comment has been minimized.

Show comment
Hide comment
@pkqk

pkqk Jan 13, 2015

You shouldn't have to play around with buildpacks, you have a requirements.txt file so Heroku should work out that it needs python.

I've not tested all this but hopefully it should give you enough of a start and clue for what to google

Get the server running

First though since you're not using a framework which has a default way to run it you need to tell Heroku to do the python server.py, this is done with a file called Procfile, this is what Foreman uses but you don't need to care, it just needs to be a file with the line:

web: python server.py

in it, which tells Heroku for the web task you want to run this command.

By this point you should be able to deploy and it might not crash but it probably won't work.

To get the web server bit to work you will need to make it listen on the right port and to not just listen on 127.0.0.1

So first change line 4 in server.py to

HOST = "0.0.0.0"

The difference here is that 127.0.0.1 is a special address that is only available from the computer the program is running on, so when you're running it on your machine you can get to it by localhost but someone else even on the same network couldn't. 0.0.0.0 means that port will be available on all the networks the computer is connected to (see security issue below).

Next we need to get the the server to listen on a port that Heroku has exposed to the internet. Before running the command in Procfile Heroku will set up some environment variables so they are available to your program. One of them is PORT which is the port exposed to the internet via Heroku's routing layer.

So we need to get access to that variable from inside the python process. In the os module there is a function called getenv that does this for us.

So in server.py add

import os

at the top and change line 3 to

PORT = int(os.getenv("PORT", 8000))

It will then set PORT to the value Heroku gives it or default to 8000 if you're running it locally.

Security issue

There is a slight security issue here, in that if you are developing on your machine someone else on the network can now get to your app, you might want this but if you don't you could change the code too:

PORT = os.getenv("PORT")
# We are not running on Heroku
if PORT is None:
  PORT = 8000
  HOST = "127.0.0.1"
else:
  HOST = "0.0.0.0"
  PORT = int(PORT)

Get the database working

The image of the code that Heroku uses is readonly so line 7 in database.py needs to use a different location. If we are lucky you might be able to change it to:

DATABASE = "/tmp/guestbook.db"

To get it working temporarily, the only issue here is that the computer running your code in Heroku is temporary, so after you haven't requested it in a while Heroku will stop the code and throw away the virtual machine which means next time it starts up /tmp/guestbook.db won't exist anymore. This is why you will need to make it use postgres.

If you don't already have a database attached to the project you can provision one following this guide and you'll then have an environment variable called DATABASE_URL with the connection string in it.

So to access the database when running on Heroku you can change the database setup lines to:

import os
import urlparse

DATABASE_URL = os.getenv('DATABASE_URL')
# use sqlite
if DATABASE_URL is None:
  database = SqliteDatabase("guestbook.db")
else:
  # extract all the database connection parameters from DATABASE_URL
  database_url = urlparse.urlparse(DATABASE_URL)
  # get rid of the "/" from the path to get the database name
  database_name = database_url.path[1:]
  # connection arguments are documented here:
  # http://initd.org/psycopg/docs/module.html#psycopg2.connect
  database = PostgresqlDatabase(
    database_name,
    user=database_url.username,
    password=database_url.password,
    host=database_url.hostname,
    port=database_url.port
  )

We're really close now

It should all work now, fingers crossed, except that the postgres database won't have the tables created in it. This is what frameworks usually call migrations but for the moment you can just cheat and get a python shell on a Heroku node and run database.create_tables() by doing this on your machine

heroku run python

Once you get the python shell

import database
database.create_tables()
quit()

Then if everything has worked you should be able to access it in the web browser. 😬

pkqk commented Jan 13, 2015

You shouldn't have to play around with buildpacks, you have a requirements.txt file so Heroku should work out that it needs python.

I've not tested all this but hopefully it should give you enough of a start and clue for what to google

Get the server running

First though since you're not using a framework which has a default way to run it you need to tell Heroku to do the python server.py, this is done with a file called Procfile, this is what Foreman uses but you don't need to care, it just needs to be a file with the line:

web: python server.py

in it, which tells Heroku for the web task you want to run this command.

By this point you should be able to deploy and it might not crash but it probably won't work.

To get the web server bit to work you will need to make it listen on the right port and to not just listen on 127.0.0.1

So first change line 4 in server.py to

HOST = "0.0.0.0"

The difference here is that 127.0.0.1 is a special address that is only available from the computer the program is running on, so when you're running it on your machine you can get to it by localhost but someone else even on the same network couldn't. 0.0.0.0 means that port will be available on all the networks the computer is connected to (see security issue below).

Next we need to get the the server to listen on a port that Heroku has exposed to the internet. Before running the command in Procfile Heroku will set up some environment variables so they are available to your program. One of them is PORT which is the port exposed to the internet via Heroku's routing layer.

So we need to get access to that variable from inside the python process. In the os module there is a function called getenv that does this for us.

So in server.py add

import os

at the top and change line 3 to

PORT = int(os.getenv("PORT", 8000))

It will then set PORT to the value Heroku gives it or default to 8000 if you're running it locally.

Security issue

There is a slight security issue here, in that if you are developing on your machine someone else on the network can now get to your app, you might want this but if you don't you could change the code too:

PORT = os.getenv("PORT")
# We are not running on Heroku
if PORT is None:
  PORT = 8000
  HOST = "127.0.0.1"
else:
  HOST = "0.0.0.0"
  PORT = int(PORT)

Get the database working

The image of the code that Heroku uses is readonly so line 7 in database.py needs to use a different location. If we are lucky you might be able to change it to:

DATABASE = "/tmp/guestbook.db"

To get it working temporarily, the only issue here is that the computer running your code in Heroku is temporary, so after you haven't requested it in a while Heroku will stop the code and throw away the virtual machine which means next time it starts up /tmp/guestbook.db won't exist anymore. This is why you will need to make it use postgres.

If you don't already have a database attached to the project you can provision one following this guide and you'll then have an environment variable called DATABASE_URL with the connection string in it.

So to access the database when running on Heroku you can change the database setup lines to:

import os
import urlparse

DATABASE_URL = os.getenv('DATABASE_URL')
# use sqlite
if DATABASE_URL is None:
  database = SqliteDatabase("guestbook.db")
else:
  # extract all the database connection parameters from DATABASE_URL
  database_url = urlparse.urlparse(DATABASE_URL)
  # get rid of the "/" from the path to get the database name
  database_name = database_url.path[1:]
  # connection arguments are documented here:
  # http://initd.org/psycopg/docs/module.html#psycopg2.connect
  database = PostgresqlDatabase(
    database_name,
    user=database_url.username,
    password=database_url.password,
    host=database_url.hostname,
    port=database_url.port
  )

We're really close now

It should all work now, fingers crossed, except that the postgres database won't have the tables created in it. This is what frameworks usually call migrations but for the moment you can just cheat and get a python shell on a Heroku node and run database.create_tables() by doing this on your machine

heroku run python

Once you get the python shell

import database
database.create_tables()
quit()

Then if everything has worked you should be able to access it in the web browser. 😬

@varjmes

This comment has been minimized.

Show comment
Hide comment
@varjmes

varjmes Jan 13, 2015

Owner

So! I have made progress thanks to @pkqk. I followed your instructions but make a couple of notes:

to work with the Postgres peewee database, psycogpg2 needs to be installed in the environment with pip. Also, the PORT config var needs to be cast to an int to work.

If you go to: https://afternoon-everglades-3926.herokuapp.com/ you will see the assets load.
https://afternoon-everglades-3926.herokuapp.com/form.html works too. However the homepage (https://afternoon-everglades-3926.herokuapp.com/cgi-bin/script.py) does not and I am not sure why.

I receive the following error message:

sock=backend at=error code=H18 desc="Request Interrupted" method=GET path="/cgi-bin/script.py" host=afternoon-everglades-3926.herokuapp.com with some other bits and a status of 503.

Heroku tells me that:
the ... backend (your app’s web process) socket was closed before the backend returned an HTTP response and I am unsure what to do with this information to make it work.

But, still, progress. And thank you so much @pkqk.

Owner

varjmes commented Jan 13, 2015

So! I have made progress thanks to @pkqk. I followed your instructions but make a couple of notes:

to work with the Postgres peewee database, psycogpg2 needs to be installed in the environment with pip. Also, the PORT config var needs to be cast to an int to work.

If you go to: https://afternoon-everglades-3926.herokuapp.com/ you will see the assets load.
https://afternoon-everglades-3926.herokuapp.com/form.html works too. However the homepage (https://afternoon-everglades-3926.herokuapp.com/cgi-bin/script.py) does not and I am not sure why.

I receive the following error message:

sock=backend at=error code=H18 desc="Request Interrupted" method=GET path="/cgi-bin/script.py" host=afternoon-everglades-3926.herokuapp.com with some other bits and a status of 503.

Heroku tells me that:
the ... backend (your app’s web process) socket was closed before the backend returned an HTTP response and I am unsure what to do with this information to make it work.

But, still, progress. And thank you so much @pkqk.

@pkqk

This comment has been minimized.

Show comment
Hide comment
@pkqk

pkqk Jan 13, 2015

Ah script might be crashing at some point. You might have to resort to print line debugging by adding

import sys

# and then add this everywhere or more informative strings
sys.stderr.write("I got here before crashing\n")

If you're not already getting a backtrace then normal stdout is probably being hidden because that output normally goes to the browser as that's how CGI works so hopefully using stderr for logging will show up in the heroku logs too

pkqk commented Jan 13, 2015

Ah script might be crashing at some point. You might have to resort to print line debugging by adding

import sys

# and then add this everywhere or more informative strings
sys.stderr.write("I got here before crashing\n")

If you're not already getting a backtrace then normal stdout is probably being hidden because that output normally goes to the browser as that's how CGI works so hopefully using stderr for logging will show up in the heroku logs too

@varjmes

This comment has been minimized.

Show comment
Hide comment
@varjmes

varjmes Jan 13, 2015

Owner

So I spent the last hour or two working with someone familiar with the Heroku infrastructure. I learnt a bunch of neat tricks but unfortunately it seems that Heroku just will not support even the simplest of cgi-scripts.

So this project will be local only until some kind of magic happens.
Thank you very much for all your input, @tomwhoiscontrary & @pkqk 👍

Owner

varjmes commented Jan 13, 2015

So I spent the last hour or two working with someone familiar with the Heroku infrastructure. I learnt a bunch of neat tricks but unfortunately it seems that Heroku just will not support even the simplest of cgi-scripts.

So this project will be local only until some kind of magic happens.
Thank you very much for all your input, @tomwhoiscontrary & @pkqk 👍

@tef

This comment has been minimized.

Show comment
Hide comment
@tef

tef Jan 16, 2015

One of the things that prevents it from working on heroku is the use of the filesystem: The filesystem is reset to the original state every 24 hours. You need to use an external database like postgres.

tef commented Jan 16, 2015

One of the things that prevents it from working on heroku is the use of the filesystem: The filesystem is reset to the original state every 24 hours. You need to use an external database like postgres.

@tomwhoiscontrary

This comment has been minimized.

Show comment
Hide comment
@tomwhoiscontrary

tomwhoiscontrary Jan 16, 2015

This may be another really unhelpful rabbit-holing comment, but you could try Cloud Foundry. There, the filesystem is only reset when you restart or redeploy the app. So you still want an external database really, but it might be a better short-term solution than Heroku.

tomwhoiscontrary commented Jan 16, 2015

This may be another really unhelpful rabbit-holing comment, but you could try Cloud Foundry. There, the filesystem is only reset when you restart or redeploy the app. So you still want an external database really, but it might be a better short-term solution than Heroku.

@tef

This comment has been minimized.

Show comment
Hide comment
@tef

tef Jan 16, 2015

I was wrong and forgot, dynos have a read only file system except for one-off dynos. You have to remove all code that mutates the filesystem, which includes updating index.html, counter.txt, and guestbook.db. I'm working on getting it on heroku for fun

tef commented Jan 16, 2015

I was wrong and forgot, dynos have a read only file system except for one-off dynos. You have to remove all code that mutates the filesystem, which includes updating index.html, counter.txt, and guestbook.db. I'm working on getting it on heroku for fun

@tef tef referenced a pull request that will close this issue Jan 16, 2015

Open

Heroku Support #15

@varjmes

This comment has been minimized.

Show comment
Hide comment
@varjmes

varjmes Jan 16, 2015

Owner

@tef thank you so so much for all of your help, I can't wait to go through your PR and take a look at the changes. 🎈

Owner

varjmes commented Jan 16, 2015

@tef thank you so so much for all of your help, I can't wait to go through your PR and take a look at the changes. 🎈

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment