Only do PCL workarounds for x86

[dhood]

Looks like #71 broke the new turtlebot CI job for aarch64:

It's a docker mounting error (error creating aufs mount to /var/lib/docker/aufs/mnt/25641437ee2e6f787c6877d8df5ca9441c85594145074c71f3d0ca27082f484d: invalid argument) but it persists even with a clean cache. From testing it seems to only have issues mounting if these lines are run (e.g. if INSTALL_TURTLEBOT2_DEMO_DEPS is false it's fine)

I don't have quite enough context to know if this is the appropriate fix (maybe these lines need to be completely reworked?), but it's one thing that works. @clalancette could you provide input please?

Turtlebot CI jobs:
linux:
linux-aarch64:

[clalancette]

Hm, that is pretty weird. We need these workarounds for aarch64 as well, and they do work on my local Pine64 board. I'll take a look at this today and see if I can figure something out.

[dhood]

Are you able to build the dockerfile with this invocation locally? I suspect it would have the same issue but haven't checked myself, I've just been doing investigations on CI jobs.

---

This are the jobs I was investigating with: http://ci.ros2.org/view/All/job/test_ci_turtlebot-demo_linux-aarch64/ (they will be deleted after this issue is resolved). The first 10 jobs I was messing around with the docker cache on the host (eventually I just completely removed /var/lib/docker/aufs). You'll see that this job had a clear cache and had to rebuild the entire image from scratch, and it still failed.

Realising that the cache probably wasn't the issue, jobs 11-20 I was playing around with the job configuration. In those jobs, the only ones that successfully built+ran the docker image were ones that either:

• didn't pass the INSTALL_TURTLEBOT2_DEMO_DEPS var, or
• passed it as INSTALL_TURTLEBOT2_DEMO_DEPS=false or
• passed it as INSTALL_TURTLEBOT2_DEMO_DEPS=true but used this branch to avoid those lines

That's what led me to the conclusion that it's the dockerfile that's causing it to not build properly, not docker issues on the host.

[clalancette]

Hm, this is getting stranger. If I log into the packet.net server, and run:

$ docker build --build-arg PLATFORM=arm --build-arg INSTALL_TURTLEBOT2_DEMO_DEPS=true -t ros2_batch_ci_turtlebot_demo linux_docker_resources

by hand, it builds the docker image just fine. I did have to make a few edits to the Dockerfile to make this work; namely, I had to edit FROM ubuntu:xenial to be FROM aarch64/ubuntu:xenial, and I had to comment out everything having to do with RTI, but other than that it seemed to work. I'm not sure what is going on in the context of jenkins that is causing it to fail. Still looking into it.

[clalancette]

OK. So having the RTI stuff commented out actually materially affects this issue in ways I don't understand. If I have all of the RTI stuff in the Dockerfile, I can see the problem clearly when running by hand on the packet.net server. Oddly, if I comment out the very last ADD of the rticonnext-dds_tools debian file, then things start working again. Similarly, if I comment out one or the other of the

RUN if test ${INSTALL_TURTLEBOT2_DEMO_DEPS}

(it doesn't matter which one), it also starts working. I can't say I understand any of this, but that last point lends itself to a workaround; a single RUN statement that does all of the workarounds at once. I don't like it, because I don't understand the underlying issue, but that could be a workaround for now. I'm going to try that out in a CI build.

[clalancette]

TB aarch64:
TB amd64:

[clalancette]

All right, that looked pretty good. Here's some builds that should actually build the PCL libraries, to make sure that the workarounds still work:

TB aarch64:
TB amd64:

[nuclearsandwich]

It irks me that the issue is worked around by reducing the number of RUN directives rather than any change specific to the generated image. I wonder if the problem is related to aufs itself. It looks like the packet.net machines are currently rocking 3.13 which means we miss overlayfs (in kernel as of 3.18) by a few revisions and the overlay2 driver available in Docker 1.12 requires 4.0 or newer.

If we hit other errors that seem to be related to image layering, looking closer at aufs bugs might be worthwhile.

[clalancette]

@nuclearsandwich It also concerns me a lot. There is clearly something wrong, because removing some combination of ADD and RUN statements makes the thing work. Further, this only happens on aarch64, so I'm pretty convinced it is a bug in the lower layers (Docker, aufs, or the kernel). Unfortunately, I can't afford to spend another day mucking around with this, so I'd like to go with the workaround for now.

[clalancette]

All right, the turtlebot builds that actually use PCL work (the unstable bit of that is because of a slight packaging bug in cartographer, which I will address later). One more CI, just to check that we don't affect the "regular" jobs, then I'll open a new PR with my changes:

• Linux
• Linux-aarch64
• macOS
• Windows

[nuclearsandwich]

I can't afford to spend another day mucking around with this, so I'd like to go with the workaround for now.

Yeah that's fine, I just wanted to put a breadcrumb down before I even forgot that I once knew of errors like that.

[dirk-thomas]

You might be seeing this one: ros-infrastructure/ros_buildfarm#377

[clalancette]

@dirk-thomas Oh, interesting. Let me try that out.

[clalancette]

The magic fix didn't work. Also, looking at the linked github issue, the symptoms are different. I think we'll still have to go with my workaround for now.

[clalancette]

All right, all of the jobs we care about succeeded. I've opened #74 instead to fix this, so I'm going to close this out. Also, I've opened up #75 to track down the "real" issue.