Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide ARMv6l binaries #5525

Closed
wants to merge 1 commit into from

Conversation

bltavares
Copy link

ARM is a quite popular for small computing clusters, and an interesting
pet project to try Nomad on.

Unfortunately, the current ARM builds don't run on Raspberry Pi's 1,
Zero or Zero W as it is compiled with some operations not available
there.

When executing the current binary, you receive this message:

Illegal instruction (core dumped)

The difference is caused by the version of the ARM instructions are
available on each version of the Pi.

| Version | Compiler arch |
|-------------+----------------------------|
| Pi v1, Zero | armel, floating point (v6) |
| Pi v2, v3 | armhf, floating point (v7) |

Fortunately, cross-compiling using Go is not so troublesome. It is possible to
produce an executable using the non-hf version of gcc.

This commit introduces a new make target for armel binaries. It renames the
previous one so it is clear the difference between artifacts versions.

A couple of extra compilation dependencies are needed on the build machine.

Tested armel o a Pi Zero W 1.3 and Pi 3, which executes, but there are not yet
any jobs running. The armel binary should work on all Raspberry Pi models,
while the armhf only works on Pi 2/3 and it is optimized.

Fixes #2517

ARM is a quite popular for small computing clusters, and an interesting
pet project to try Nomad on.

Unfortunately, the current ARM builds don't run on Raspberry Pi's 1,
Zero or Zero W as it is compiled with some operations not available
there.

When executing the current binary, you receive this message:

```
Illegal instruction (core dumped)
```

The difference is caused by the version of the ARM instructions are
available on each version of the Pi.

| Version     | Compiler arch              |
|-------------+----------------------------|
| Pi v1, Zero | armel, floating point (v6) |
| Pi v2, v3   | armhf, floating point (v7) |

Fortunately, cross-compiling using Go is not so troublesome. It is possible to
produce an executable using the non-hf version of `gcc`.

This commit introduces a new `make` target for armel binaries. It renames the
previous one so it is clear the difference between artifacts versions.

A couple of extra compilation dependencies are needed on the build machine.

Tested `armel` o a Pi Zero W 1.3 and Pi 3, which executes, but there are not yet
any jobs running. The `armel` binary should work on all Raspberry Pi models,
while the `armhf` only works on Pi 2/3 and it is optimized.

Fixes hashicorp#2517
@bltavares
Copy link
Author

I was able to test this on a local cluster, using a mix of Pi Zero, Pi3 and amd64 machines and got it running the Redis deployment example.

If it was not for the bugs [1][2] on Docker image selection logic, it would have worked on the first allocation try.

Nomad already provides the processor architecture, but it does not include the variant. I've then added two meta tags on each consul client, to help selecting the image to run on each node based on the variant:

client {
  enabled           = true
  node_class = "armv6l" # value from $(uname -m)

  meta {
    "arch" = "armv6l" # value from $(uname -m)
    "arch_base" = "arm32v6" # mapping logic to Docker multiarch repos prefix (eg: arm32v6, amd64, arm64v8...)
  }
}

With that, I was able to run the redis example:

job "redis" {
	datacenters = ["dc1"]

	constraint {
		attribute = "${attr.kernel.name}"
		value = "linux"
	}

	update {
		stagger = "10s"
		max_parallel = 1
	}

	group "cache" {
		count = 3
		restart {
			attempts = 2
			interval = "1m"
			delay = "10s"
			mode = "fail"
		}

		task "redis" {
			driver = "docker"

			config {
				image = "${meta.arch_base}/redis:alpine" # arm32v6/redis does not have latest tag
				port_map {
					db = 6379
				}
			}

			service {
				name = "${TASKGROUP}-redis"
				tags = ["global", "cache"]
				port = "db"
				check {
					name = "alive"
					type = "tcp"
					interval = "10s"
					timeout = "2s"
				}
			}

			resources {
				cpu = 500 # 500 MHz
				memory = 256 # 256MB
				network {
					mbits = 10
					port "db" {
					}
				}
			}
		}
	}
}

Precompiled binaries here: https://github.com/bltavares/nomad/releases/tag/v0.9.0-armel

@cgbaker
Copy link
Contributor

cgbaker commented Apr 15, 2019

Hi @bltavares , thank you for submitting this. However, we are going to leave the older ARM architectures up to community support. A preferred avenue for supporting these (and other niche) platforms will be external repos like @nicholasjackson https://github.com/nicholasjackson/nomad-arm6l.

@cgbaker cgbaker closed this Apr 15, 2019
@bltavares
Copy link
Author

bltavares commented Apr 15, 2019

@cgbaker Thanks for letting us know. Its unfortunate that armel will not be part of the in-tree deployment process, yet understandable.

The suggested project, unfortunately, produces a Nomad binary which does not work when scheduling tasks. It fails due to to some cross-compilation issues I haven't been able to resolve yet (it's related to how os/user deals with cgo).

The binary compile with these changes do work tho. I could try to configure a new Docker builder with the same setup as the Vagrant machine as an alternative project, but I'll most likely just keep my fork updated then.

@github-actions
Copy link

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 12, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Illegal instruction (core dumped) on armv6l
2 participants