Add support for ARM architecture

[russorat]

I would like the ability to run Flux (via InfluxDB) when it is installed on an ARM based processor.

[ajoberstar]

I'm running Influx 2.0.0-beta.9 on a Raspberry Pi 4B on the arm64 kernel with Raspbian. So far I've only been seeing issues with the mean and sum operations like the referenced issue. Other ops like median, last, max, difference, work fine.

[mich43l]

The mean operation is really important for the use of Flux and InfluxDB. Please fix it on ARM devices like Raspberry Pi. Is there any workaround or fix? Using Influx 1.8 with Flux on RPi 3.

[russorat]

@mich43l or @ajoberstar this is the code for mean specifically: https://github.com/influxdata/flux/blob/master/stdlib/universe/mean.go

Anything jumping out at you as to why mean would break on ARM? If so, would you be willing to contribute a fix? Unfortunately, I don't have any ARM devices to test this on.

[Kloetenhofer]

Same Here.
We choose Influxdb because it supports inserting data with custom timestamps, which is not possible with most of the other time series db systems like prometheus. We also use a raspberry pi to gather about 300 measurements each 50-300 milliseconds from an industry PLC, controlling mining/construction machines over CAN bus. To compensate for transmit delays and write buffer stacking, we need those custom timestamps to keep data in perfect order as is was created in the PLC. Obviously you can not join data on _time from multiple measurements when timestamps are not equal, so we need aggregateWindow() with mean() function before filtering by _value and doing the join. So far everything works fine on a regular debian server, but we're getting that

{"error":"panic: runtime error: invalid memory address or nil pointer dereference"}

when doing the same thing on the ARM architecture

[ajoberstar]

@russorat I'm not familiar with Go, but I can do my best. As a starting point, how do I get a more detailed error message? I'm more familiar with the JVM ecosystem, so is there some equivalent to a stack trace that can help point to the line it's erroring on?

[cinhcet]

any updates on this?

[FixTestRepeat]

Just tested this on a Pi 4B running with the new beta aarch64 kernel on raspbian/RaspberryOS. (4.19.118-v8+ #1311 SMP PREEMPT).

The issue is still occurring, and is easily reproducable under beta Influxdb v2 (influxdb_2.0.0-beta.10 ) in the Data Explorer.
For any query constructed, add sum or mean and then click submit. The results pane shows the error:

panic: runtime error: invalid memory address or nil pointer dereference

I tried enabling debug logging mode, but there's nothing helpful why the error is returned. I'm willing to test further if you can provide steps/areas to investigate to help.

[spacejay01]

I have the same issue. Any update on this? I was happy that with Flux it is finally possible to aggregate to months. Now Flux QL does not work on ARM. I think many users run it on a Raspberry. Hope there is a solution soon.

[PhilDay-CT]

Is there anyway to get some attention on this ? Flux is core to 2.0, and mean() is pretty much core to any down sampling, so without this it pretty much rules out using influx on ARM

[PhilDay-CT]

If anyone does need a workaround for this in the short term the following seems to work

|> window(every: 3h)
|> reduce(fn: (r, accumulator) => ({
      count: accumulator.count + 1,
      total: accumulator.total + r._value,
      avg: (accumulator.total + r._value) / float(v: accumulator.count)
    }),
    identity: {count: 1, total: 0.0, avg: 0.0}
  )
  |> map(fn: (r) => ({ r with _time: r._stop, _value: r.ave }))
  |> drop(columns: ["_start", "_stop", "count", "total"])

[pcgeek86]

I'm running into a specific limitation where Alerts cannot work, because there's no ability to specify a custom Flux query in the web management interface. I'll check to see if the influx CLI supports writing Alerts with custom Flux queries.

I filed feedback around the specific Flux error messages on InfluxDB v2 RC3 (Release Candidate 3) here: influxdata/influxdb#19874

This is how I got Ubuntu Server 20.04.1 LTS Focal Fossa running on Raspberry Pi. It wasn't trivial, but not terribly hard either.

https://trevorsullivan.net/2020/10/30/install-ubuntu-20-04-1-lts-64-bit-on-raspberry-pi/

[jpbarcia]

Same issue here, looked through the code a bit, and the only difference that I could spot was:

func (a *SumFloatAgg) DoFloat(vs *array.Float64) {
	if l := vs.Len() - vs.NullN(); l > 0 {
		if vs.NullN() == 0 {
			a.sum += math.Float64.Sum(vs) <---------
			a.ok = true
		} else {
			for i := 0; i < vs.Len(); i++ {
				if vs.IsValid(i) {
					a.sum += vs.Value(i)
					a.ok = true
				}
			}
		}
	}
}

This is somewhat different from how the mode and max are calculated (without the arrow.math functions). So... I went to the apache arrow repo: https://github.com/apache/arrow/tree/master/go. And run this in the raspberry pi 4 that I have:

$ go test -bench=8192 -run=. ./math
--- FAIL: TestFloat64Funcs_Sum (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x19dbd0]

goroutine 6 [running]:
testing.tRunner.func1.1(0x1c5a60, 0x37cc20)
        /snap/go/6724/src/testing/testing.go:1072 +0x240
testing.tRunner.func1(0x4000001b00)
        /snap/go/6724/src/testing/testing.go:1075 +0x34c
panic(0x1c5a60, 0x37cc20)
        /snap/go/6724/src/runtime/panic.go:969 +0x15c
github.com/apache/arrow/go/arrow/math.Float64Funcs.Sum(...)
        /home/barcia/dev/arrow/go/arrow/math/float64.go:38
github.com/apache/arrow/go/arrow/math_test.TestFloat64Funcs_Sum(0x4000001b00)
        /home/barcia/dev/arrow/go/arrow/math/float64_test.go:35 +0x190
testing.tRunner(0x4000001b00, 0x213610)
        /snap/go/6724/src/testing/testing.go:1123 +0xdc
created by testing.(*T).Run
        /snap/go/6724/src/testing/testing.go:1168 +0x244
exit status 2
FAIL    github.com/apache/arrow/go/arrow/math   0.018s
FAIL

Looks like the only correlation that I could find for now is the Runtime Error, also logs from Influx (influxdata/influxdb#19874 (comment)) suggest that the problem may come from the arrow.math functions. I wonder if we could get a build excluding those functions for arm64, haven't tried to compile the project though.

[jpbarcia]

@russorat An update on the last comment that I left: Searching for a solution, using the build tag "noasm" for the apache arrow math functions made the tests to pass =).
Also looks like this commit: influxdata/influxdb@333640d introduced the noasm build tag and basically solved this issue, I believe there should be defined export GOARCH=$(shell go env GOARCH) in the Makefile though, but I don't really know 100% how Makefiles work, so I edited the commands directly. My raspberry 4 crashed twice while doing the yarn build:ci but succeeded the third time (I'm looking forward for the performance of the Apple M1 ;).
Mean and Sum are now working fine, and wayyyyy faster than the workaround with the reduce function.
Side note: I didn't run the tests :(. And it's not as straight forward to build influxdb from source, it took me a while, e.g. yarnpkg should have an alias "yarn" (for ubuntu), we need to install cargo/rust (aarch64-unknown-linux-gnu toolchain works fine)

[mgodlewski]

@jpbarcia, I've tried the 1.8.3 version on docker, released today but the commit you pointed out is on v2 branche only (not available on docker yet), too bad :/
Do you think this commit could be backported on v1.8 branch ?

[jpbarcia]

My understanding (which comes from 10 minutes looking on how to build influxdb 1.8.3) is that you can also specify in the build.py the noasm tag, something like this: go build -tags "noasm" ... may work.

Anyway, I compiled influxdb at head (2.0.1?) for arm64 (with the raspi4), I also have a docker image (locally hosted - unofficial ;) that I can share if you want. It's working fine, last I tested 1M records take a second - if not less - to process for a simple window average, which is awesome =).

[PhilDay-CT]

Hi Jean Pierre, I'd be interested in having access to your docker image if possible please ? Thanks Phil

…

On Fri, 20 Nov 2020 at 17:45, Jean Pierre ***@***.***> wrote: My understanding (which comes from 10 minutes looking on how to build influxdb 1.8.3) is that you can also specify in the build.py <https://github.com/influxdata/influxdb/blob/v1.8.3/build.py> the noasm tag, something like this: go build -tags "noasm" ... may work. Anyway, I compiled influxdb at head (2.0.1?) for arm64 (with the raspi4), I also have a docker image (locally hosted - unofficial ;) that I can share if you want. It's working fine, last I tested 1M records take a second - if not less - to process for a simple window average, which is awesome =). — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2505 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQVLOMSW3FUFLX7QMKYU4NDSQ2TMJANCNFSM4KSVHTZA> .

[jpbarcia]

Sure. Here is the link to the saved container (influxdb2) in my Google Drive, file is influxdb_aa1cefa9e2.tar.
Just load the image with docker load, it should load a localhost/influxdb:aa1cefa9e2.
This is more or less the dockerfile I used (with the binaries - influx, influxd - that are also in the link), you can also use just influxd as the start command:

FROM  arm64v8/ubuntu:20.04
COPY ./bin/influx /usr/bin/influx
COPY ./bin/influxd /usr/bin/influxd
EXPOSE 8086
CMD influxd run --bolt-path /var/lib/influxdb/influxd.bolt --engine-path /var/lib/influxdb/engine --store bolt

---

Just in case:
This is not intended for production usage, since I haven't run all the tests.
Happy to help if there are any issues, but bear in mind that I'm not in any way involved with the development of this product. So, use it at your own risk =)

[danxmoran]

While testing this out as part of influxdata/pkg-config#26, I found this tool from the Rust Tools team. It could be useful to incorporate into the build/test process for this repo.

[zbot473]

@jpbarcia How'd you do it? I'm able to build InfluxDB but the UI gives a 500 error.

[jpbarcia]

@zbot473 make sure to build all the project (run make, or make all maybe), there should be some step in there to build the UI, which I believe running just make influxd or go build cmd/influxd won't do it.

[zbot473]

One thing, if anyone comes across this and gets OOM error (code 137) on the Pi4, I configured it to have a 2GB swapfile, and it worked fine after that. This was on a 4GB model, so you might have to adjust for more swap on a lower end, to even none on the 8GB model.

[danxmoran]

The InfluxDB 2.0.3 release includes an updated ARM64 build. You can download the binaries here: https://dl.influxdata.com/influxdb/releases/influxdb2-2.0.3_linux_arm64.tar.gz

Please let me know how it goes if you try it out!

[zbot473]

Thanks! I'll check it out.

[FixTestRepeat]

Thanks, downloaded it this morning. Initial checks look good

[zbot473]

This works well, and I've been using it for the past few days on my RasPi! You must download a 64-bit OS like Ubuntu (server or desktop) or Raspbian 64 bit.

[miettal]

influxdb 1.x version still has ARM compatibility problem.
mean aggregation will be stucked.

It seem latest flux package is not applied to influxdb 1.x release.
is there planning of deploying?

[timhallinflux]

No. Recommendation is to move to 2.x -- so long as you have 64 bit ARM.

[miettal]

OK. thank you for reply.

// I think flux interface of 1.8.x is for migration to 2.x. but if no planning of bugfix, the migration will be difficult...