Issue with Heroku deployment #136

Closed
peter opened this Issue Jul 11, 2015 · 29 comments

Projects

None yet

3 participants

@peter
peter commented Jul 11, 2015

This may not be a problem with the Luminus framework as such but there seems to be issues with Heroku deployment. I keep getting "SQLException No suitable driver found" (DriverManager.java:689). I created my luminus app with +postgres and I have tried a variety of different postgresql driver versions to no avail. Any idea what the issue could be? The apps work great for me locally on my Macbook.

@yogthos
Member
yogthos commented Jul 11, 2015

I don't believe this is Luminus specific and if it works locally, then It could be an issue with the URL for Heroku according to this. Check to make sure that the DATABASE_URL variable is available in the environment on the server with the correct jdbc URL.

@peter
peter commented Jul 11, 2015

The database URL (DATABASE_URL) is set automatically by Heroku when the PostgreSQL addon is created. Also, I am able to connect to PostgreSQL directly like this:

(require '[clojure.java.jdbc :as db])
(db/insert! (System/getenv "DATABASE_URL") :posts {:subject "clojure works on heroku"})

If it matters, my project.clj dependencies are currently these:

             [org.clojure/java.jdbc "0.3.2"]
             [postgresql "9.1-901.jdbc4"]

(from https://devcenter.heroku.com/articles/clojure-web-application)

So is the issue maybe with yesql or some other library since direct JDBC access works?

@peter
peter commented Jul 11, 2015

Another issue I have with Heroku deployments is that "heroku run lein repl" takes forever to start up and seems to time out about every second time.

@yogthos
Member
yogthos commented Jul 11, 2015

It sounds like the issue might be somewhere else, I doubt it's yesql since it just piggybacks on clojure.java.jdbc. Using lein repl isn't the recommended way to run the app. The default Procfile sets it up to use the main function from the compiled project, e.g:

web: java $JVM_OPTS -cp target/app.jar clojure.main -m app.core
``
@yogthos
Member
yogthos commented Jul 11, 2015

It might be an issue with environ that's used within Luminus. The library will try to resolve environment variables and should generate :database-url key for you if there's a DATABASE_URL available as a shell variable.

@peter
peter commented Jul 12, 2015

I found the issue. It turns out db-spec needs to be a string for the username:password type URL that Heroku uses to work. It would be great if you could update your Heroku deployment documentation to reflect this.

For details, check out the get-connection function in clojure.java.jdbc:

https://github.com/clojure/java.jdbc/blob/ffea0332a87ec4100d459ec9533e9619d46fb2c8/src/main/clojure/clojure/java/jdbc.clj#L176

This works:

(DriverManager/getConnection "jdbc:postgresql://ec2-54-217-202-108.eu-west-1.compute.amazonaws.com:5432/dcql6a1c91v5up" (as-properties {:user "uorytkslaggjoj", :password ""})

But this doesn't:

(DriverManager/getConnection (System/getenv "DATABASE_URL"))

@peter
peter commented Jul 12, 2015

NOTE: the db-spec needs to be changed to a string in db/migrations.clj as well as it's duplicated there.

@peter
peter commented Jul 12, 2015

And now for the next stumbling block on Heroku... Migrations run fine locally on postgresql (lein run migrate) but not on Heroku (heroku run lein run migrate). I get this exception:

INFO: Starting migrations
Jul 12, 2015 10:00:07 AM clojure.tools.logging$eval106$fn__110 invoke
INFO: Ending migrations
Exception in thread "main" java.lang.NullPointerException, compiling:(/tmp/form-init1050664351735383545.clj:1:73)
at clojure.lang.Compiler.load(Compiler.java:7239)
at clojure.lang.Compiler.loadFile(Compiler.java:7165)
at clojure.main$load_script.invoke(main.clj:275)
at clojure.main$init_opt.invoke(main.clj:280)
at clojure.main$initialize.invoke(main.clj:308)
at clojure.main$null_opt.invoke(main.clj:343)
at clojure.main$main.doInvoke(main.clj:421)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:383)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.Var.applyTo(Var.java:700)
at clojure.main.main(main.java:37)
Caused by: java.lang.NullPointerException
at org.postgresql.jdbc2.AbstractJdbc2Statement.replaceProcessing(AbstractJdbc2Statement.java:829)
at org.postgresql.jdbc2.AbstractJdbc2Statement.(AbstractJdbc2Statement.java:149)
at org.postgresql.jdbc3.AbstractJdbc3Statement.(AbstractJdbc3Statement.java:42)
at org.postgresql.jdbc3g.AbstractJdbc3gStatement.(AbstractJdbc3gStatement.java:28)
at org.postgresql.jdbc4.AbstractJdbc4Statement.(AbstractJdbc4Statement.java:32)
at org.postgresql.jdbc4.Jdbc4Statement.(Jdbc4Statement.java:30)
at org.postgresql.jdbc4.Jdbc4PreparedStatement.(Jdbc4PreparedStatement.java:23)
at org.postgresql.jdbc4.Jdbc4PreparedStatement.(Jdbc4PreparedStatement.java:18)
at org.postgresql.jdbc4.Jdbc4Connection.prepareStatement(Jdbc4Connection.java:39)
at org.postgresql.jdbc3.AbstractJdbc3Connection.prepareStatement(AbstractJdbc3Connection.java:275)
at org.postgresql.jdbc2.AbstractJdbc2Connection.prepareStatement(AbstractJdbc2Connection.java:198)
at clojure.java.jdbc$prepare_statement.doInvoke(jdbc.clj:419)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.core$apply.invoke(core.clj:634)
at clojure.java.jdbc$db_query_with_resultset.invoke(jdbc.clj:736)
at clojure.java.jdbc$query.doInvoke(jdbc.clj:769)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at migratus.database$completed_ids_STAR_$fn__2930.invoke(database.clj:177)
at clojure.java.jdbc$db_transaction_STAR_.doInvoke(jdbc.clj:541)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at migratus.database$completed_ids_STAR_.invoke(database.clj:175)
at migratus.database.Database.completed_ids(database.clj:223)
at migratus.core$uncompleted_migrations.invoke(core.clj:39)
at migratus.core$migrate_STAR_.invoke(core.clj:57)
at migratus.core$run.invoke(core.clj:25)
at migratus.core$migrate.invoke(core.clj:63)
at marklunds.db.migrations$migrate.invoke(migrations.clj:17)
at marklunds.core$_main.doInvoke(core.clj:37)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.lang.Var.invoke(Var.java:379)
at user$eval58.invoke(form-init1050664351735383545.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6782)
at clojure.lang.Compiler.eval(Compiler.java:6772)
at clojure.lang.Compiler.load(Compiler.java:7227)
... 11 more

@yogthos
Member
yogthos commented Jul 12, 2015

Ah, I'll definitely update the docs for Heroku connection url. You should be able to run migrations from the compiled jar by running java -jar target/app.jar migrate.

@yogthos
Member
yogthos commented Jul 12, 2015

Out of curiosity, how are you populating the db-url currently. Locally, you would populate it in profiles.clj where you have

{:provided {:env {:database-url "jdbc:postgresql://localhost/app_dev?user=gallery&password=pictures"}}}

The string in the DATABASE_URL on Heroku should match this, e.g:

"jdbc:postgresql://ec2-54-217-202-108.eu-west-1.compute.amazonaws.com:5432/dcql6a1c91v5up?user=uorytkslaggjoj&password=yourpass"

The migrations.clj namespace creates a map that should work with get-connection:

{:store :database
 :db {:connection-uri (:database-url env)}}

The clojure.java.jdbc accepts the :connection-uri key with the raw connection string.

@peter
peter commented Jul 12, 2015

I am passing in the Heroku postgresql URL string directly as the db-spec (or db) instead of a map. That should work with jdbc/get-connection. However, when I run the migrations, I get this:

peter@Peters-MacBook-Air ~/src/marklunds]$ heroku run java -jar target/marklunds.jar migrate
Running java -jar target/marklunds.jar migrate attached to terminal... up, run.6593
2015-07-12 13:22:17.254:INFO::main: Logging initialized @14975ms
Jul 12, 2015 1:22:19 PM clojure.tools.logging$eval36$fn__40 invoke
INFO: Starting migrations
Jul 12, 2015 1:22:19 PM clojure.tools.logging$eval36$fn__40 invoke
INFO: creating migration table 'schema_migrations'
Jul 12, 2015 1:22:19 PM clojure.tools.logging$eval36$fn__40 invoke
INFO: Ending migrations
Exception in thread "main" java.lang.NullPointerException
at org.postgresql.jdbc2.AbstractJdbc2Statement.replaceProcessing(AbstractJdbc2Statement.java:829)
at org.postgresql.jdbc2.AbstractJdbc2Statement.(AbstractJdbc2Statement.java:149)
at org.postgresql.jdbc3.AbstractJdbc3Statement.(AbstractJdbc3Statement.java:42)
at org.postgresql.jdbc3g.AbstractJdbc3gStatement.(AbstractJdbc3gStatement.java:28)
at org.postgresql.jdbc4.AbstractJdbc4Statement.(AbstractJdbc4Statement.java:32)
at org.postgresql.jdbc4.Jdbc4Statement.(Jdbc4Statement.java:30)
at org.postgresql.jdbc4.Jdbc4PreparedStatement.(Jdbc4PreparedStatement.java:23)
at org.postgresql.jdbc4.Jdbc4PreparedStatement.(Jdbc4PreparedStatement.java:18)
at org.postgresql.jdbc4.Jdbc4Connection.prepareStatement(Jdbc4Connection.java:39)
at org.postgresql.jdbc3.AbstractJdbc3Connection.prepareStatement(AbstractJdbc3Connection.java:275)
at org.postgresql.jdbc2.AbstractJdbc2Connection.prepareStatement(AbstractJdbc2Connection.java:198)
at clojure.java.jdbc$prepare_statement.doInvoke(jdbc.clj:419)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.core$apply.invoke(core.clj:634)
at clojure.java.jdbc$db_query_with_resultset.invoke(jdbc.clj:736)
at clojure.java.jdbc$query.doInvoke(jdbc.clj:769)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at migratus.database$completed_ids_STAR_$fn__2930.invoke(database.clj:177)
at clojure.java.jdbc$db_transaction_STAR_.doInvoke(jdbc.clj:541)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at migratus.database$completed_ids_STAR_.invoke(database.clj:175)
at migratus.database.Database.completed_ids(database.clj:223)
at migratus.core$uncompleted_migrations.invoke(core.clj:39)
at migratus.core$migrate_STAR_.invoke(core.clj:57)
at migratus.core$run.invoke(core.clj:25)
at migratus.core$migrate.invoke(core.clj:63)
at marklunds.db.migrations$migrate.invoke(migrations.clj:17)

As you can see it successfully creates the schema_migrations table but then it falls over.

@yogthos
Member
yogthos commented Jul 12, 2015

The documentation for get-connection states that you should be passing a map pointing to the uri though:

Raw:
    :connection-uri (required) a String
                 Passed directly to DriverManager/getConnection
@peter
peter commented Jul 12, 2015

You can also pass a string to jdbc/get-connection though according to the same documentation:

" String:
subprotocol://user:password@host:post/subname
An optional prefix of jdbc: is allowed."

In fact, as I wrote above, that's what you need to do on Heroku for the Heroku DATABASE_URL to be parsed.

https://github.com/clojure/java.jdbc/blob/ffea0332a87ec4100d459ec9533e9619d46fb2c8/src/main/clojure/clojure/java/jdbc.clj#L176

Here is my corresponding REPL session where migrations fail:

marklunds.core=> (require '[clojure.java.jdbc :as sql])
nil
marklunds.core=> (require '[marklunds.db.core :as db])
nil
marklunds.core=> (sql/get-connection db/db-spec)
#object[org.postgresql.jdbc4.Jdbc4Connection 0x57f41477 "org.postgresql.jdbc4.Jdbc4Connection@57f41477"]
marklunds.core=> (require '[migratus.core :as migratus])
nil
marklunds.core=> (migratus/up {:store :database :db db/db-spec} [])
Jul 12, 2015 1:30:10 PM clojure.tools.logging$eval47$fn__51 invoke
INFO: Starting migrations
Jul 12, 2015 1:30:10 PM clojure.tools.logging$eval47$fn__51 invoke
INFO: Ending migrations
NullPointerException org.postgresql.jdbc2.AbstractJdbc2Statement.replaceProcessing (AbstractJdbc2Statement.java:829)

@yogthos
Member
yogthos commented Jul 12, 2015

I tested with both approaches locally and you're right either a string or a map will work. I'm not sure why the map isn't working for you on heroku as it's the same URL string in both cases though.

@yogthos
Member
yogthos commented Jul 14, 2015

@pupeno out of curiosity have you run into anything similar with your heroku setup? @peter and have you had any luck? :)

@pupeno
Contributor
pupeno commented Jul 14, 2015

Yes, I believe I have run into this problem. I had some workarounds which I intended to integrate back into mainstream but I never got around to it. This is in my to-do list as I plan on using Heroku.

@pupeno
Contributor
pupeno commented Jul 14, 2015

For example, my patch for Korma accepts both, jdbc as well as Heroku-like URIs: korma/Korma#316

@yogthos
Member
yogthos commented Jul 14, 2015

looks like we might need a function in db.core to handle different kinds of urls then

@pupeno
Contributor
pupeno commented Jul 15, 2015

@yogthos actually, many libraries should handle those URLs, so I created this little library: https://github.com/carouselapps/to-jdbc-uri

@yogthos
Member
yogthos commented Jul 15, 2015

Good idea, if using the library to parse different style URIs would be the best approach. If it works with both Heroku style and jdbc URIs then I think it would make sense to update Luminus to use it.

@pupeno
Contributor
pupeno commented Jul 15, 2015

Yes, that's the goal of the library. I wonder whether it should be used by Luminus or by the other libraries, like migratus, yesql, korma, etc. Having it in more than one place is not a problem as it's a no-op for proper jdbc URIs.

It works for Heroku and JDBC at the moment. I'm open to add more.

@yogthos
Member
yogthos commented Jul 15, 2015

I think it would probably be easier to start using it in Luminus first, as it doesn't require buy in from the library maintainers.

@pupeno
Contributor
pupeno commented Jul 16, 2015

Agreed.

J. Pablo Fernández pupeno@pupeno.com
http://pupeno.com
On Jul 15, 2015 15:25, "Dmitri Sotnikov" notifications@github.com wrote:

I think it would probably be easier to start using it in Luminus first, as
it doesn't require buy in from the library maintainers.


Reply to this email directly or view it on GitHub
#136 (comment)
.

@pupeno
Contributor
pupeno commented Jul 17, 2015

There you go: #140

@yogthos
Member
yogthos commented Jul 17, 2015

@pupeno oh by the way does this lib do the same thing by any chance? :) https://github.com/weavejester/hanami

@pupeno
Contributor
pupeno commented Jul 18, 2015

Yeah, it kinda does. I intend to-jdbc-uri to be more generic than just Heroku though.

@yogthos
Member
yogthos commented Jul 18, 2015

I think that's a good plan. :)

@yogthos
Member
yogthos commented Jul 18, 2015

@peter the problem should be solved in the latest template thanks to the library from @pupeno that parses both the Heroku and JDBC style URLs. The following steps should resolve the issue:

add the dependency:

[to-jdbc-uri "0.1.0"]

require:

[to-jdbc-uri.core :refer [to-jdbc-uri]]

then set the :connection-uri as follows:

:connection-uri (to-jdbc-uri (:database-url env))
@yogthos
Member
yogthos commented Jul 29, 2015

I'm going to close this one as it should be working now.

@yogthos yogthos closed this Jul 29, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment