Quick dev. environment using Vagrant & Salt

[thor]

Following straight from the commit:

Ubuntu 14.04.1 with osm2pgsql, postgis and kosmtik ready to use

• Uses Vagrant to quickly create a VM based on Ubuntu 14.04.1 LTS
  • Adjustable amounts of CPU cores and memory
• Easily have your own kosmtik plugins ready if you want to start from scratch
  • Sets up kosmtik as a service, so you can easily start/stop using $ service kosmtik <start/stop/status/restart>
• Added .vagrant/ and Vagrantfile to the ignore list to avoid unintentional commits with edits
• Does NOT import any .pbfs automatically, as it can quickly get messy
  doing such. However, I've played around with it and it works -- as
  long as you've got the memory and you're not importing overlapping
  areas.
• However, it does execute get-shapefiles for you -- just vagrant up or vagrant provision
• Edit as you please and reload it for kosmtik to pick it up
• All you need to do to get started is to import with osm2pgsql

I queried @pnorman about this very briefly in #osm-dev at some point, and so here is a PR to see if there is any interest in adding this as part of the repo.
The benefits are mainly listed above, but the recap would be the following:

• No messing around with relatively long lists of instructions
  • The configuration is combined from switch2osm and the notepad with instructions that I've seen linked around here
• Allows for a "shared" development environment
  • This is only to some degree; you all have vastly different HW requirements and desires for what and how to render, but it's an easy way to get an identical setup
• Makes it a whole lot easier for people on Windows to join the party
  • I started it while I was using Windows myself, but even on any other distribution it's nice to not have to worry about the development environment too much. Note the .gitattributes addition is to avoid gits conversion of line endings for the shell scripts, so that errors don't occur when running i.e. get-shapefiles.sh

For those of you not familiar with Vagrant, you won't be adding a whole VM to the repository -- just a configuration file for Vagrant and Salt roots for configuring the VM for osm2pgsql, kosmtik and postgis. (Though, not in that order.) You can also clone my branch and simply run vagrant up once you've installed Vagrant on your computer, and have a look.

Usage is currently as follows:

1. If not already present, install Vagrant and VirtualBox.
2. Clone the repository.
3. If you'd like kosmtik plugins, edit the Vagrantile and add the ones you want.
4. Run vagrant up from within the repository.
5. Run vagrant ssh to SSH in, so that you can run your own osm2pgsql import. (Place any files in data/ and they'll be available at /srv/openstreetmap-carto/data/ inside the VM.)
6. Once data is imported, the map will magically be visible at http://127.0.0.1:6789.
7. Ideally, edit stuff and see how simple it is to vagrant destroy && vagrant up if you want to recreate it.

Either way, if it's not something for you folks, that's completely understandable: the PR addes one file to the root, as well as an entire folder containing 4 files. (These can be put elsewhere if desired.) Should it be something that is a bit too meta, then I'll just keep it separated in its own repository. However, at least for me it's really practical when I can clone a repo, run vagrant up and be at the finish line for what I need to contribute to a project. ☀️

If there is an interest in it, but something is amiss, let me know and I'll try what I can to fix & improve it.

[pnorman]

You're checking in the Vagrantfile, should it really be ignored?

[thor]

@pnorman It's mentioned in the commit, but it's ignored to avoid accidentally pushing a modified Vagrantfile unless it really is your intention, as you'd use it to configure the virtual machines number of CPU cores, memory, on/off for the hardware extensions for virtualization -- as well as changing which kosmtik plugins get installed during/by the Salt highstate / vagrant provision.

[pnorman]

it sounds like switching between branches might cause problems then?

[thor]

Yeah, you'll have to commit or stash your changes, otherwise git will get vocal. Stashing would be the most convenient, then you can just pop wherever you checkout, and potentially git update-index --assume-unchanged Vagrantfile if you'd like to tell git to be quiet about it.

An alternative approach is to never commit the Vagrantfile, and instead have a Vagrantfile.default or similarly that a user would have to rename to Vagrantfile. That sort of makes a simple thing somewhat more complex, but also can avoid minor things down the road.

[pnorman]

14.04.2 now?

[thor]

@pnorman Not as of yet; the trusty64 box has yet to be updated to 14.04.2. This isn't much of a problem, as .2 mainly consists of kernel and hardware support updates (beyond other typical package updates).

[pnorman]

It looks useful and I've added some comments, but is this the right repo for this? I'm not the best judge, as I know I'd never use it.

[thor]

It looks useful and I've added some comments, but is this the right repo for this?

👍 Glad to hear.
Also, that's a good question. I alone definitely can't answer that. What I can say is that having a simple Vagrantfile in a project allows for people to quickly contribute, as well as see what the project "does" with a higher degree of ease - generally speaking. In this case I think it's extra valuable, at least if you haven't setup the whole of PostGIS before, or your Unix knowledge is limited, while your CartoCSS and SQL experience is more useful.

If it'd make a difference I could link to other projects doing so, but it shouldn't be included on the merit of "somebody else does it!", rather if it makes sense. Potentially as mentioned in the PR, I could split it out to its own repository, but then it'd be harder to find, which again would decrease the usefulness of it.

I'm not the best judge, as I know I'd never use it.

I'm not going to try to convince you to do so, but could you give me some pointers on what makes it less useful for you, or irrelevant, so that I could potentially improve it for other people alike?

[pnorman]

I'm not going to try to convince you to do so, but could you give me some pointers on what makes it less useful for you, or irrelevant, so that I could potentially improve it for other people alike?

I have multiple database servers running on bare hardware. I will be running a minutely updated osm2pgsql database on my dev box. I edit multiple stylesheets.

I could virtualize Kosmtik and point it at an existing database host with PGHOST, but I wouldn't gain much out of that.

[gravitystorm]

I think this is useful, but it should be managed in a separate repository. I'm happy to link to that repository in the documentation.

Like @pnorman I wouldn't use it myself so I'm reluctant to have it in the main repository - it would be misleading. I run multiple stylesheets and databases, and use multiple tools (kosmtik, tilemill) and multiple text editors - never mind compiling my own mapnik (for testing various things) and linking node-mapnik against that custom mapnik. All in all, I wouldn't want to run tilemill/kosmtik in an isolated environment.

I can see that it will be useful for other people, but it's best managed separately.

[thor]

I think this is useful, but it should be managed in a separate repository. I'm happy to link to that repository in the documentation.

Understandable. I'll close the pull request and see if I can put the files together in a way that makes it as convenient as possible to get started with it.

When it's ready, how should I get in touch re: linking to it, if desireable?

[matkoniecz]

@thor
Maybe it would be a good idea to setup repository with alternative ways to setup environment? Maybe for some mine "notepad with instructions that I've seen linked around here" would be better (at least for me it is, as on my machine carto inside VM is really slow).

[thor]

@mkoniecz Absolutely -- it's a complex combination of different components, and there's no reason why I shouldn't try to make it useful for power users in the same way you'd try to make it quick and easy for new users. 👍

A thought that did occur to me is that it is silly to have the PostGIS database on the same VM as mapnik/kosmtik/tilemill, simply because you'd never want to constantly be destroying and re-importing your PostGIS data -- it's time consuming and useless.

It seems the main target audience isn't you folks ( ;-) ), but even I grow tired of importing Great Britain among others! I'm setting it up so that either way, even if you're a new user, the PostGIS database and your choice of rendering/editing are on two different machines. (If you use a "cloud" provider or OpenStack provider, vSphere, KVM, etc, Vagrant can still take care of deploying the VM to your own stack. 💡 )

[matthijsmelissen]

I think this is useful, but it should be managed in a separate repository.

So I'm closing the PR. Feel free to continue discussing here about setting this up elsewhere.

[pnorman]

furry-sansa is also related, being a tool to manage importing osm2pgsql databases and set up config files. Slightly different use-case.

[kocio-pl]

Just a quick note: I'm interested in Docker and I know that Vagrant 1.6+ can use Docker images, so probably that could be a perfect match for all the OS-es.

BTW: Docker is said to be running only in 64-bit Linux, but that's not the full truth - we could run containers also on 32-bit Linux if we use special 32-bit base images (like https://github.com/docker-32bit/ubuntu or https://registry.hub.docker.com/repos/32bit/), so all the Linuxes would run at full speed, with no VM overhead.