Add deploy on Heroku (via badge) #231

wilzbach · 2017-07-13T12:21:07Z

So I am publicly experimenting with a "Deploy on Heroku badge" to simplify deployment for new mirrors (and do a bit of dog-fooding on the way).
I managed to get the app to start on Heroku, but currently it still fails with:

2017-07-13T12:15:06.389476+00:00 heroku[web.1]: Starting process with command `./dub-registry --port 5199 --mirror=https://code.dlang.org`
2017-07-13T12:15:10.288494+00:00 app[web.1]: Enforcing certificate trust.
2017-07-13T12:15:10.370178+00:00 app[web.1]: MongoDB reply was longer than expected, skipping the rest: 105 vs. 36
2017-07-13T12:15:10.370495+00:00 app[web.1]: object.Exception@../../app/.dub/packages/vibe-d-0.8.0-beta.8/vibe-d/mongodb/vibe/db/mongo/cursor.d(318): Query failed.
2017-07-13T12:15:10.370805+00:00 app[web.1]: ----------------
2017-07-13T12:15:10.370826+00:00 app[web.1]: ??:? [0x6d19ee]
2017-07-13T12:15:10.370851+00:00 app[web.1]: ??:? [0x6d7a52]
2017-07-13T12:15:10.370918+00:00 app[web.1]: ??:? [0x4e144b]
2017-07-13T12:15:10.370931+00:00 app[web.1]: ??:? [0x466f95]
2017-07-13T12:15:10.370933+00:00 app[web.1]: ??:? [0x4700ed]
2017-07-13T12:15:10.370951+00:00 app[web.1]: ??:? [0x46722f]
2017-07-13T12:15:10.370954+00:00 app[web.1]: ??:? [0x466eec]
2017-07-13T12:15:10.370968+00:00 app[web.1]: ??:? [0x46698a]
2017-07-13T12:15:10.370971+00:00 app[web.1]: ??:? [0x421779]
2017-07-13T12:15:10.370985+00:00 app[web.1]: ??:? [0x421433]
2017-07-13T12:15:10.370988+00:00 app[web.1]: ??:? [0x4cb892]
2017-07-13T12:15:10.371003+00:00 app[web.1]: ??:? [0x6ddefe]
2017-07-13T12:15:10.371006+00:00 app[web.1]: ??:? [0x6ddebd]
2017-07-13T12:15:10.371268+00:00 app[web.1]: ??:? __libc_start_main [0x20ff482f]
2017-07-13T12:15:10.371022+00:00 app[web.1]: ??:? [0x6dddb8]
2017-07-13T12:15:10.371286+00:00 app[web.1]: ??:? [0x4074a8]
2017-07-13T12:15:10.477784+00:00 heroku[web.1]: Process exited with status 1
2017-07-13T12:15:10.489833+00:00 heroku[web.1]: State changed from starting to crashed

I will comment on the invidividual fixes needed so far.
This can be reproduced locally, by checking out this branch and then pushing it to Heroku:

heroku create
heroku buildpacks:add http://github.com/MartinNowak/heroku-buildpack-d.git 
git push heroku

(huge thanks to Martin's awesome buildpack)

wilzbach · 2017-07-13T12:22:49Z

source/dubregistry/mongodb.d

+	import std.process : environment;
+	string mongodbURI = environment.get("MONGODB_URI", "mongodb://127.0.0.1");
+	parseMongoDBUrl(mongoSettings, mongodbURI);
+	mongoSettings.authMechanism = MongoAuthMechanism.scramSHA1;


Default for Mongo3 (released since two years) - many cloud instances don't support none anymore:

vibe-d/vibe.d#1785

wilzbach · 2017-07-13T12:25:38Z

app.json

+  "repository": "https://github.com/heroku/node-js-sample",
+  "logo": "https://dlang.org/images/dlogo_opengraph.png",
+  "keywords": ["d", "dlang", "dub", "mirror"]
+}


Setting up a mirror will be only two clicks away - see e.g. https://devcenter.heroku.com/articles/heroku-button

A mongoDB addon is needed though, but there are many free ones (e.g. mLab).

MartinNowak · 2017-07-14T12:34:05Z

Think you need to get the github secrets via env to make this work for Heroku.

MartinNowak · 2017-07-14T12:35:31Z

README.md

@@ -4,3 +4,5 @@ DUB registry
 ![vibe.d logo](public/images/logo-small.png) Online registry for [dub](https://github.com/dlang/dub/) packages, see <http://code.dlang.org/>.

 [![Build Status](https://travis-ci.org/dlang/dub-registry.svg)](https://travis-ci.org/dlang/dub-registry)
+
+[![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://heroku.com/deploy?template=https://github.com/dlang/dub-registry)


Should come with a bit more context, e.g. "You can host your own registry instance for private repos.".

wilzbach · 2017-07-14T12:49:51Z

Think you need to get the github secrets via env to make this work for Heroku.

The mirror-mode doesn't need any credentials and this is only intended for the mirror mode.

wilzbach · 2017-07-14T12:51:23Z

. "You can host your own registry instance for private repos.".

That's actually a good use case and once we have figured out the Mongo failure, I will definitely make this as an option.

wilzbach · 2017-07-19T02:49:05Z

Good news 🎉
I finally got this to work, see e.g. https://dub-registry.herokuapp.com

I managed to get the app to start on Heroku, but currently it still fails with:

This was due to "vpmreg" being the default database (and thus getting the segfaults).
FWIW MongoSettings.safe would have shown a nice error message like "not authorized for insert on vpmreg.packages" instead of the rather unspecific Exception from above -> vibe-d/vibe.d#1858

Also for the future it might make sense to show a warning about a non-existent database directly at the handleQuery Exception: vibe-d/vibe.d#1859

. "You can host your own registry instance for private repos.".
That's actually a good use case and once we have figured out the Mongo failure, I will definitely make this as an option.

This could be a follow-up to this PR - this one is big enough already.

MartinNowak

Looks OKish

MartinNowak · 2017-07-25T11:55:43Z

Procfile

@@ -0,0 +1,3 @@
+default_process_types:
+  web: ./dub-registry --port $PORT --mirror=https://code.dlang.org --hostname=dub-registry.heroku.com --separate-cron --vv --bind 0.0.0.0
+  cron: ./dub-registry --mirror=https://code.dlang.org --run-cron --vv


Not particularly nice as a cron task in a separate process cannot update in-process caches.
Might be problematic during future development, just support a minor Heroku suspend case.
Any other ideas?

Not really. Heroku instances fall asleep very soon, so chances are good that the main dyno gets suspended / terminated a lot anyways.

MartinNowak · 2017-07-25T11:56:27Z

source/dubregistry/mongodb.d

+MongoClientSettings mongoSettings;
+string databaseName = "vpmreg";
+
+shared static this()


To avoid another shared static this(), maybe lazily initialize settings or just parse them everytime connectMongoDB is called (once usually).

MartinNowak · 2017-07-25T11:58:17Z

README.md

+Select the "Scheduler" addon and add the following task:
+
+```
+./dub-registry --mirror=https://code.dlang.org --run-cron --vv


I'd say the private registry use-case is more interesting than encouraging dozens of mirrors, in particular with the current inefficient mirror protocol.

dlang-bot · 2017-09-14T23:22:56Z

Thanks for your pull request, @wilzbach!

wilzbach · 2017-09-14T23:33:01Z

Not particularly nice as a cron task in a separate process cannot update in-process caches.
Might be problematic during future development, just support a minor Heroku suspend case.
Any other ideas?

Dynos are stopped and restarted on the free version of Heroku quite often anyways and this doesn't affect the default in-process cron job (it's an opt-in CLI flag).

I'd say the private registry use-case is more interesting than encouraging dozens of mirrors, in particular with the current inefficient mirror protocol.

Well, for now, having one free mirror node in a totally different network is quite nice and of course I agree that being able to run a private mirror directly in Heroku would be quite nice, but it's a lot more complicated, e.g.

setting GitHub / GitLab authorization credentials
Allowing clients to securely & privately access the registry

As a serious company wouldn't host their code with a third-party provider anyways, this is more "fun" use-case, but in any case, this PR should lay the groundworks if this becomes interesting to anyone.

Addressed other comments and rebased.

wilzbach · 2017-09-14T23:36:01Z

source/app.d


 	readOption("mirror", &s_mirror, "URL of a package registry that this instance should mirror (WARNING: will overwrite local database!)");
 	readOption("hostname", &hostname, "Domain name of this instance (default: code.dlang.org)");
+	readOption("separate-cron", &separateCron, "Use a separate cron job to query for packages.");


We could also remove this flag as the dyno must likely gets suspended and one or two additional updates won't be problematic.

MartinNowak · 2017-09-18T17:27:33Z

The readme still says Deploy your own mirror in a header.
There is no point in having 10 people run a dub-registry mirror on Heroku just because they can.
It will cause unnecessary traffic and load on our main dub-registry instance.
The only interesting use-case is running a separate registry with unofficial/private repos (and optionally mirror-mode fou your Heroku mirror).

wilzbach · 2017-12-18T05:37:26Z

I just found a new use case for this: setting up automatic deployment to Heroku is really easy, so we could do the following setup:

master -> automatic deployment to dub-registry.herokuapp.com
testing -> automatic deployment to dub-registry-testing.herokuapp.com

testing could be an unprotected branch to which people can force-push their changes to let everyone preview their PR (like alpha.dub.pm).

So can we move forward with this?

s-ludwig · 2017-12-18T11:29:37Z

Mainly Martin's comment about the mirror vs. private registry still stands. I'd agree that we shouldn't unnecessarily advertise the mirror mode at this point, especially because we don't have means to make this a trusted process (e.g. signing the database snapshot), so encouraging the use of code.dlang.org mirrors that are not controlled by trusted people would not be a good idea.

wilzbach · 2017-12-18T11:37:22Z

Mainly Martin's comment about the mirror vs. private registry still stands. I'd agree that we shouldn't unnecessarily advertise the mirror mode at this point,

OK. Is removing the "Deploy to Heroku" badge from the README enough? I would leave the app.json config for Heroku in as at some point I can/will/might get back to the deploying your private registry use case.

MartinNowak · 2018-01-26T10:50:52Z

Yes, a staging preview instance for PRs on Heroku would be nice.

MartinNowak

Otherwise LGTM.

MartinNowak · 2018-01-26T10:56:34Z

source/app.d

@@ -80,12 +91,20 @@ shared static this()

 	auto router = new URLRouter;
 	if (s_mirror.length) router.any("*", (req, res) { req.params["mirror"] = s_mirror; });
-	router.get("*", (req, res) @trusted { if (!s_checkTask.running) startMonitoring(); });


The check task shouldn't die, so that looks more like a safety guard.

MartinNowak · 2018-01-26T10:57:00Z

source/app.d


 	readOption("mirror", &s_mirror, "URL of a package registry that this instance should mirror (WARNING: will overwrite local database!)");
 	readOption("hostname", &hostname, "Domain name of this instance (default: code.dlang.org)");
+	readOption("separate-cron", &separateCron, "Use a separate cron job to query for packages.");


Let's remove the separate cron, it seems superfluous, even on Heroku.

Heroku suspends the dynos if they aren't run. How else would we make sure that the DB gets updated at least daily?

Ok removed.

wilzbach · 2018-03-30T03:45:24Z

views/dubregistry.mail.package-version-errors.dt

@@ -5,6 +5,7 @@ block title

 block body
 	- import vibe.textfilter.urlencode;
+	- import std.algorithm.searching : startsWith;


if (!branch_or_version.startsWith("~")) (below)

wilzbach · 2018-06-29T18:51:40Z

So I'm gave this another shot as I want to setup Heroku Review Apps. In short. For each PR, a new temporary dyno will be created and be existent for five days (the limit can be customized).
I also removed the controversial bit about deploying a mirror on Heroku and running a separate cron.
However, it now fails with:

2018-06-29T18:03:58.732613+00:00 heroku[web.1]: Starting process with command `./dub-registry --port 43675 --hostname=dub-registry.heroku.com --vv --bind 0.0.0.0`
2018-06-29T18:04:01.339272+00:00 app[web.1]: [main(----) dbg] Initializing OpenSSL...
2018-06-29T18:04:01.339295+00:00 app[web.1]: [main(----) dbg] ... done.
2018-06-29T18:04:01.339296+00:00 app[web.1]: Enforcing certificate trust.
2018-06-29T18:04:01.339298+00:00 app[web.1]: [main(----) dbg] add route GET *
2018-06-29T18:04:01.339303+00:00 app[web.1]: [main(----) dbg] creating new MongoConnection connection, all 0 are in use

I tried disabling the userman part, but without luck so far.
Any ideas?

wilzbach · 2018-07-05T11:56:17Z

Finally figured it out and it's working -> https://dub-registry-staging.herokuapp.com
Travis will fail for now as Vibe.d doesn't work with 2.081 (vibe-d/vibe.d#2181)

wilzbach · 2018-07-05T11:53:51Z

.d-compiler

@@ -0,0 +1 @@
+dmd-2.080.1


Required as Vibe.d currently doesn't build with 2.081 (see e.g. vibe-d/vibe.d#2181)

wilzbach · 2018-07-06T17:46:05Z

Going to merge this now as this has been open for more than a year, no one seems to be actively objecting this and I want to get started on the Heroku PR preview.

wilzbach commented Jul 13, 2017

View reviewed changes

wilzbach force-pushed the deploy-on-heroku branch from c0b75d2 to 253c48f Compare July 13, 2017 12:23

wilzbach commented Jul 13, 2017

View reviewed changes

MartinNowak reviewed Jul 14, 2017

View reviewed changes

wilzbach mentioned this pull request Jul 19, 2017

MongoSettings.safe should be the default vibe-d/vibe.d#1858

Open

wilzbach force-pushed the deploy-on-heroku branch 3 times, most recently from e9125e3 to 62e969e Compare July 19, 2017 02:39

wilzbach force-pushed the deploy-on-heroku branch 2 times, most recently from 6b3bd02 to ec0da2c Compare July 19, 2017 02:56

This was referenced Jul 19, 2017

add code-mirror.dlang.io as fallback registry dlang/dub#1190

Merged

Use abort() for fatal exit and avoid catching Throwable unless really necessary. vibe-d/vibe.d#1856

Merged

Merge remote-tracking branch 'upstream/mangle' into merge_mangle dlang/dmd#6998

Merged

MartinNowak approved these changes Jul 25, 2017

View reviewed changes

wilzbach mentioned this pull request Sep 10, 2017

Use multiple registry fallbacks instead of one dlang/dub#1246

Merged

wilzbach force-pushed the deploy-on-heroku branch from ec0da2c to 976f861 Compare September 14, 2017 23:22

wilzbach commented Sep 14, 2017

View reviewed changes

wilzbach force-pushed the deploy-on-heroku branch from 041bc10 to 156d464 Compare January 10, 2018 13:57

MartinNowak approved these changes Jan 26, 2018

View reviewed changes

wilzbach force-pushed the deploy-on-heroku branch 4 times, most recently from 31334fa to afc618b Compare March 30, 2018 03:36

wilzbach commented Mar 30, 2018

View reviewed changes

wilzbach force-pushed the deploy-on-heroku branch from afc618b to ca6073c Compare March 31, 2018 20:23

Move mongoSettings to a separate module and lazily initialize them

6337e11

wilzbach force-pushed the deploy-on-heroku branch 6 times, most recently from 2b98f05 to 2e3bf97 Compare June 29, 2018 18:49

wilzbach temporarily deployed to dub-registry-staging June 29, 2018 18:57 Inactive

wilzbach force-pushed the deploy-on-heroku branch from 2e3bf97 to 6d4fd7e Compare July 5, 2018 11:51

Add deploy to Heroku

6d4fd7e

wilzbach changed the title ~~Add deploy on Heroku badge~~ Add deploy on Heroku (via badge) Jul 5, 2018

wilzbach commented Jul 5, 2018

View reviewed changes

.d-compiler

@@ -0,0 +1 @@

dmd-2.080.1

Copy link

Member Author

wilzbach Jul 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Required as Vibe.d currently doesn't build with 2.081 (see e.g. vibe-d/vibe.d#2181)

wilzbach merged commit a76cdec into dlang:master Jul 6, 2018

wilzbach deleted the deploy-on-heroku branch July 6, 2018 17:46

s-ludwig added a commit that referenced this pull request Jun 24, 2020

Fix certificate trust regression introduced by #231.

4f5ff94

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add deploy on Heroku (via badge) #231

Add deploy on Heroku (via badge) #231

wilzbach commented Jul 13, 2017 •

edited

Loading

wilzbach Jul 13, 2017

wilzbach Jul 13, 2017

MartinNowak commented Jul 14, 2017

MartinNowak Jul 14, 2017

wilzbach commented Jul 14, 2017

wilzbach commented Jul 14, 2017

wilzbach commented Jul 19, 2017

MartinNowak left a comment

MartinNowak Jul 25, 2017

wilzbach Mar 30, 2018

MartinNowak Jul 25, 2017

MartinNowak Jul 25, 2017 •

edited

Loading

dlang-bot commented Sep 14, 2017

wilzbach commented Sep 14, 2017

wilzbach Sep 14, 2017

MartinNowak commented Sep 18, 2017

wilzbach commented Dec 18, 2017

s-ludwig commented Dec 18, 2017

wilzbach commented Dec 18, 2017

MartinNowak commented Jan 26, 2018

MartinNowak left a comment

MartinNowak Jan 26, 2018

MartinNowak Jan 26, 2018

wilzbach Feb 9, 2018

wilzbach Mar 30, 2018

wilzbach Mar 30, 2018

wilzbach commented Jun 29, 2018

wilzbach commented Jul 5, 2018

wilzbach Jul 5, 2018

wilzbach commented Jul 6, 2018

Add deploy on Heroku (via badge) #231

Add deploy on Heroku (via badge) #231

Conversation

wilzbach commented Jul 13, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MartinNowak commented Jul 14, 2017

Choose a reason for hiding this comment

wilzbach commented Jul 14, 2017

wilzbach commented Jul 14, 2017

wilzbach commented Jul 19, 2017

MartinNowak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MartinNowak Jul 25, 2017 • edited Loading

Choose a reason for hiding this comment

dlang-bot commented Sep 14, 2017

wilzbach commented Sep 14, 2017

Choose a reason for hiding this comment

MartinNowak commented Sep 18, 2017

wilzbach commented Dec 18, 2017

s-ludwig commented Dec 18, 2017

wilzbach commented Dec 18, 2017

MartinNowak commented Jan 26, 2018

MartinNowak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wilzbach commented Jun 29, 2018

wilzbach commented Jul 5, 2018

Choose a reason for hiding this comment

wilzbach commented Jul 6, 2018

wilzbach commented Jul 13, 2017 •

edited

Loading

MartinNowak Jul 25, 2017 •

edited

Loading