Monorepo install script and Dockerfile #163

gordonbrander · 2017-04-19T22:41:49Z

Summary

This pull request is a followup for #158. It introduces a new installer script (scripts/install_dev) and an updated Dockerfile.

Before landing this PR, we should land #158.

Motivation

A consistent challenge during the build and deploy of the first 10-20 PFC kits has been the difficulty of:

Creating reproducible releases
Updating and releasing changes to openag_brain, openag_python and various firmware modules in an easy-to-manage and easy-to-test way.
Installing the result on a Pi in a reproducible way.

This PR is part of a series of changes aimed at moving most or all of openag_brain's openag dependencies into a single "monorepo".

This PR moves the install script into openag_brain, so they can be versioned together. It also modifies the install script to create and build a single catkin workspace (instead of the two previously created). The goal is to produce a single install directory that can be ADDed to the docker image or distributed as a folder full of binary files, so you don't need to compile from source.

End Goal

Eventually, we want to get to a place where you can install openag_brain via apt-get install openag_brain and all dependencies are distributed as binaries.

In the interim, we want at least to be able to offer a zip file with all compiled binaries for openag_brain, ROS and dependencies on the download page.

Details

Since Raspberry Pi does not have ROS binaries available, this PR bundles roscore and ros_comm with the openag_brain in the same workspace. This is different from the previous dev installer, which created 2 workspaces, one with ros + dependencies, the other containing openag_brain. There were several interconnected motivations for this change:

By bundling everything into one workspace, we can copy the resulting install artifact and avoid cloning git dependencies (for example, usb_cam) when building the Docker image. This is part of our general effort to avoid cloning git dependencies during build time (instead relying on locked-in releases)
The ROS indigo Docker image we were using is out-of-date and the author no longer builds against Debian (instead using Ubuntu). This is ok, I guess, but it is a bit spooky. The new Dockerfile is based on the minimal resin/rpi-raspbian, which is frequently updated. We install libboost and python with apt-get, since these are not part of the image.
ROS's dependency story is a mess. By keeping it to one workspace, we simplify the generation of .rosinstall files and so forth. I've documented all of the details on the wiki http://wiki.openag.media.mit.edu/ros/build_tooling, http://wiki.openag.media.mit.edu/ros/dependencies.

The new install script uses .rosinstall files that are checked in and versioned with the rest of openag_brain. This lets us ensure our source dependencies are always the same, and are versioned along with the rest of the software.

Alternatives

It would be nice to get releases for ros, ros_comm, usb_cam, etc published to the ROS APT package repository. This would make builds faster. But I'm not sure how to do that, so the barrier to entry for that solution is high.
Another alternative would be to use Ubuntu on the Pi, since there are binary releases for ROS on this platform.
Still another alternative would be to use the Beaglebone, for which there are Ubuntu releases. This might make a lot of sense long term, since the Beaglebone is supposed to be more reliable than the Pi when it comes to memory corruption.

All of these alternatives seem attractive, but this PR also seems like a good step on the way to any of them.

Working notes

http://wiki.openag.media.mit.edu/contributors/gordonb/update_install_self_contained

This rosinstall file can be used to set up a catkin ws that has ROS indigo, openag_brain and all catkin dependencies.

We hope to deprecate this soon.

gordonbrander · 2017-04-19T23:50:07Z

@LeonChambers requesting review from you since you did a deep dive on this once already.

gordonbrander · 2017-04-19T23:58:50Z

@CarrierDirectJoe it shouldn't really change much... just moves the same old firmware into the main repo. It does make flashing go faster though.

novemberalpha · 2017-04-20T00:00:31Z

Switched to my personal account. I'm knee deep in this on my PFC1 and struggling with the launch yaml fixtures vs the CouchDB entries and which is authoritative. It's exciting to see this moving in a positive direction. :-)

novemberalpha · 2017-04-20T00:02:01Z

I'd like there to be a day when you could hit a PFC1 launch file if you've got old HW and it'd pick up just fine. Of course the new sensors are super sexy so I'm fighting an overhaul until I get the PFC1 launch to work. So thank you for your hard work!

gordonbrander · 2017-04-20T00:03:38Z

@novemberalpha

I'd like there to be a day when you could hit a PFC1 launch file if you've got old HW and it'd pick up just fine.

I want to get to this point too. I don't have the V1 sensors to test with, but the firmware is still there in the firmware directory. You should be able to create a yaml and launch file with them configured and add it to launch. If you want to give it a go, lmk.

novemberalpha · 2017-04-20T00:15:39Z

I'm not sure if you saw I put together a tut on getting the DHT22 up on PFC1. It works on the docker container because the V2 launch file has the firmware_module_type specified. Unfortunately the GC0012 driver was coded to use a serial port which is not how it's implemented on the PFC1 microcontroller shield. I'm currently re-writing a GC0011 firmware_module that uses code directly from the gro-microcontroller repo and leverages the legacy install. That shield was super tight solder work and I'm sure many are not interested in resoldering it. :-)

I'll take a look at the new firmware folder and continue to contribute legacy firmware modules (we have two PFC1s we just completed) until the first HW is fully supported. I'm hoping this will be a gateway to experiment with new HW on both platforms.

Cheers!

J

gordonbrander · 2017-04-20T17:13:14Z

Just discovered the Docker Builder Pattern http://wiki.openag.media.mit.edu/docker/builder_pattern. It may make sense to ADD the whole dev environment into the image. The tradeoff is that it makes the docker image larger. The advantage is that we have snapshotted dev environments.

This way you can run the script from anywhere

LeonChambers · 2017-04-21T17:35:43Z

Installing ROS from scratch is such a pain. That's why I initially just used the ROS Indigo image. This solution is probably better than that. The best solution I can think of (which you also mentioned) is to get binaries for all the ROS dependencies we need on the ROS APT server, but I don't know how to do that either.

This lets the image double as a development environment with Docker volumes. See http://wiki.openag.media.mit.edu/docker/builder_pattern

novemberalpha · 2017-04-21T19:20:36Z

I just spent 4 hours yesterday, no exaggeration, installing ROS and Catkin so I could get the ros_lib libraries for Arduino. I never want to do that again. Haha.

Ever.

gordonbrander · 2017-04-21T19:22:32Z

@novemberalpha no joke. It takes soooo long on the Pi. The goal of this PR is to allow us to create dists so people who don't want to develop openag_brain don't have to deal with this.

sp4ghet

Everything looks good, waiting to try it.

sp4ghet · 2017-04-21T22:47:58Z

scripts/install_dev

+echo "source /opt/ros/indigo/setup.bash" >> ~/.bashrc
+
+# Install platformio
+rosrun openag_brain install_pio


This is something that's always been around, but I think this command installs pio into the user home directory, leaving ambiguous stuff like "lib" under ~. I'm still flashing the SD card before I go and test this out, but I think this can be fixed pretty easily if it still is a problem.

This is not the case, works fine :)

sp4ghet

Tried it out last night, works fine!

sp4ghet · 2017-04-22T18:00:34Z

scripts/install_dev

+echo "source /opt/ros/indigo/setup.bash" >> ~/.bashrc
+
+# Install platformio
+rosrun openag_brain install_pio


This is not the case, works fine :)

gordonbrander · 2017-04-22T18:03:38Z

@spaghet thank you!!!

gordonbrander added 24 commits March 31, 2017 01:58

Add rosinstall file

ff99fd1

WIP installer script

272d7fe

Add rosinstall file and generator for ALL THE THINGS

f1e809a

This rosinstall file can be used to set up a catkin ws that has ROS indigo, openag_brain and all catkin dependencies.

Remove indigo.rosinstall

c37e2f7

Progress on new Dockerfile

9f59240

Everything except entrypoint script working

2cdcac0

chown catkin dir

5c5cd5f

Remove unused generate_launch command

48eda6b

Update install to force ~/catkin_ws location

60acf6d

Update instructions for install-from-source

808f180

Use echo rather than printf for error case

61f8209

Bring back build deps that were necessary

b5c6164

Fix setup for build

355d237

Move scripts/install_dev to scripts/init_dev

5669880

Add install script that also builds

2e94058

Clean up both scripts

53481d2

Install openag_python too

9169ce6

We hope to deprecate this soon.

Add libpython-dev

06d3a8d

Install dependencies from root of intstall space

09f3c8d

Install openag_brain service during dev install

7587cef

Install platformio as part of dev install

a29fdfe

Add instructions for building using new docker image

3180be6

Add -q to apt-get install

1b84650

Merge remote-tracking branch 'upstream/master' into monorepo_docker

357713c

gordonbrander changed the title ~~HOLD: Monorepo install script and Dockerfile~~ Monorepo install script and Dockerfile Apr 19, 2017

gordonbrander requested a review from LeonChambers April 19, 2017 23:49

Invoke wstool with absolute path

471c7c8

This way you can run the script from anywhere

gordonbrander added 3 commits April 21, 2017 11:37

Merge remote-tracking branch 'upstream/master' into monorepo_docker

824b91f

Include whole catkin directory when building Docker image

4b4f90b

This lets the image double as a development environment with Docker volumes. See http://wiki.openag.media.mit.edu/docker/builder_pattern

Merge branch 'monorepo_docker_builder_pattern' into monorepo_docker

00e69d6

gordonbrander requested review from sp4ghet and rbaynes and removed request for LeonChambers April 21, 2017 21:04

sp4ghet reviewed Apr 21, 2017

View reviewed changes

sp4ghet approved these changes Apr 22, 2017

View reviewed changes

sp4ghet merged commit 0557e89 into master Apr 22, 2017

gordonbrander deleted the monorepo_docker branch April 22, 2017 18:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Monorepo install script and Dockerfile #163

Monorepo install script and Dockerfile #163

gordonbrander commented Apr 19, 2017 •

edited

gordonbrander commented Apr 19, 2017

gordonbrander commented Apr 19, 2017

novemberalpha commented Apr 20, 2017

novemberalpha commented Apr 20, 2017

gordonbrander commented Apr 20, 2017

novemberalpha commented Apr 20, 2017

gordonbrander commented Apr 20, 2017

LeonChambers commented Apr 21, 2017

novemberalpha commented Apr 21, 2017

gordonbrander commented Apr 21, 2017

sp4ghet left a comment

sp4ghet Apr 21, 2017

sp4ghet Apr 22, 2017

sp4ghet left a comment

sp4ghet Apr 22, 2017

gordonbrander commented Apr 22, 2017

Monorepo install script and Dockerfile #163

Monorepo install script and Dockerfile #163

Conversation

gordonbrander commented Apr 19, 2017 • edited

Summary

Motivation

End Goal

Details

Alternatives

Working notes

gordonbrander commented Apr 19, 2017

gordonbrander commented Apr 19, 2017

novemberalpha commented Apr 20, 2017

novemberalpha commented Apr 20, 2017

gordonbrander commented Apr 20, 2017

novemberalpha commented Apr 20, 2017

gordonbrander commented Apr 20, 2017

LeonChambers commented Apr 21, 2017

novemberalpha commented Apr 21, 2017

gordonbrander commented Apr 21, 2017

sp4ghet left a comment

Choose a reason for hiding this comment

sp4ghet Apr 21, 2017

Choose a reason for hiding this comment

sp4ghet Apr 22, 2017

Choose a reason for hiding this comment

sp4ghet left a comment

Choose a reason for hiding this comment

sp4ghet Apr 22, 2017

Choose a reason for hiding this comment

gordonbrander commented Apr 22, 2017

gordonbrander commented Apr 19, 2017 •

edited