Longhorn manager not run on RancherOS

[aleksey005]

I try run Longhorn on RancherOS node, install open-iscsi after show error:

time="2019-10-23T13:36:56Z" level=info msg="Start overwriting built-in settings with customized values"
time="2019-10-23T13:36:56Z" level=info msg="Start Longhorn replica controller"
time="2019-10-23T13:36:56Z" level=info msg="Start Longhorn engine controller"
time="2019-10-23T13:36:56Z" level=info msg="Start Longhorn websocket controller"
time="2019-10-23T13:36:56Z" level=info msg="Start Longhorn Engine Image controller"
time="2019-10-23T13:36:56Z" level=info msg="Start Longhorn node controller"
time="2019-10-23T13:36:56Z" level=info msg="Start Longhorn Setting controller"
time="2019-10-23T13:36:56Z" level=info msg="Start kubernetes controller"
time="2019-10-23T13:36:56Z" level=info msg="Starting Longhorn instance manager controller"
time="2019-10-23T13:36:56Z" level=info msg="Start Longhorn volume controller"
time="2019-10-23T13:36:56Z" level=fatal msg="Error starting manager: cannot get disk info of directory /var/lib/rancher/longhorn/: Failed to execute: nsenter [--mount=/host/proc/5423/ns//mnt stat -fc {\"path\":\"%n\",\"fsid\":\"%i\",\"type\":\"%T\",\"freeBlock\":%f,\"totalBlock\":%b,\"blockSize\":%S} /var/lib/rancher/longhorn/], output , stderr, nsenter: failed to execute stat: No such file or directory\n, error exit status 1"

[NagleIT]

Also getting this error, Brand new cluster and node setup on RancherOS.

[NagleIT]

Changing to Ubuntu and ditching RancherOS fixed this. Must be some incompatibility with the latest RancherOS?

[TheGrumpyBadger]

Same issue.
RancherOS 1.5.4, Rancher 2.3, Longhorn 0.70. Fresh installed cluster,

[Rancheroo]

Same issue.
RancherOS 1.5.4, Rancher 2.3.3, Longhorn 0.70. Fresh installed cluster,

[yasker]

@Rancheroo @greyman-hh Can you raise an issue in RancherOS and link here? Seems something was missing in the RancherOS.

[Rancheroo]

New issue created

rancher/os#2937

[Sellto]

Same issue.
RancherOS 1.5.5, Rancher 2.3.3, Longhorn 0.8.0. Fresh installed cluster,

[swedesoft]

Same issue as well.
RancherOS 1.5.5, Rancher 2.3.3, Longhorn 0.8.0. Fresh installed cluster,

Looks like RancherOS doesn't include the stat tool.

[odensc]

I was able to get Longhorn working on RancherOS by running the following set of commands:

sudo wget https://busybox.net/downloads/binaries/1.31.0-i686-uclibc/busybox_STAT -O /bin/stat
sudo wget https://busybox.net/downloads/binaries/1.31.0-i686-uclibc/busybox_FLOCK -O /bin/flock
sudo chmod +x /bin/stat
sudo chmod +x /bin/flock
sudo ros s enable open-iscsi
sudo ros s up open-iscsi

This basically just downloads the missing utilities (stat, flock) from busybox and enables open-iscsi.

The issue is that the /bin folder doesn't persist between reboots, so this isn't a real solution. Seems like stat and flock would just need to be included in RancherOS and Longhorn would work fine.

[TheGrumpyBadger]

thx, works fine, I hope we see this soon in the next release.

[normanschlatter]

I was able to get Longhorn working on RancherOS by running the following set of commands:

sudo wget https://busybox.net/downloads/binaries/1.31.0-i686-uclibc/busybox_STAT -O /bin/stat
sudo wget https://busybox.net/downloads/binaries/1.31.0-i686-uclibc/busybox_FLOCK -O /bin/flock
sudo chmod +x /bin/stat
sudo chmod +x /bin/flock
sudo ros s enable open-iscsi
sudo ros s up open-iscsi

This basically just downloads the missing utilities (stat, flock) from busybox and enables open-iscsi.

The issue is that the /bin folder doesn't persist between reboots, so this isn't a real solution. Seems like stat and flock would just need to be included in RancherOS and Longhorn would work fine.

I asked in the last k3s meetup about this Issue, I got following response: "this is off topic from the meetup, but ill get it to the devs. you should know that we are no longer focusing heavily on rancher os."

Looks like its time to ditch RancherOS for me.

[cjellick]

That was me. I don't know all the technical details of this issue. I also don't know if this bug applies specifically to just rancher os or some other subset of linux OSes. Just wanted to give the full picture.

[marnel007]

@normanschlatter That worked for me

[gknepper]

Please @yasker and @superseb, promote this changes on RancherOS in order to support Longhorn natively:

For all on this thread that want to implement the change - Explained:

Changes on Container os-base - https://github.com/rancher/os-base/blob/master/config/busybox-dynamic.config:

Enable Flock on busybox:

from:
	# CONFIG_FLOCK is not set
to:
	CONFIG_FLOCK=y

Stat is already set by default:

CONFIG_STAT=y
CONFIG_FEATURE_STAT_FORMAT=y
CONFIG_FEATURE_STAT_FILESYSTEM=y

Change fix-up-image.sh - https://github.com/rancher/os-base/blob/master/scripts/fix-up-image.sh:

Remove lines

     usr/bin/flock \
     usr/bin/stat \

• Publish the image on a http server as explained at: https://github.com/rancher/os-base/blob/master/README.md

Changes on RancherOS:

• Dockerfile.dapper
  ARG OS_BASE_URL_amd64=http://192.168.1.15:8080/os-base_amd64.tar.xz

The new ISO will have flock and stat necessaries by longhorn installed by default at the distribution.

[uhcl-lund]

Just installed longhorn 1.0.1 and, although this article helped (adding stat and flock), I found I was also missing lsblk, which I "stole" from an overlay file system:

cp /var/lib/docker/overlay2/84b6637f5e5e0d1a160300637d3a183f0d1cf0cb3c483545fb9e4ad72cf9a1c3/merged/bin/lsblk /bin/

(your long hex string WILL be different, I used "find / -type f -name lsblk -print" to find one. Make sure you get one from the "merged" path).

I will update here if I ever figure out how to make these three executables "permanent" after boot.

[gknepper]

@uhcl-lund I modified RancherOS to include stat and flock. I'll also include lsblk. I was wondering if you can test it ? Thanks.

[uhcl-lund]

I’d be happy to. I am new to GitHub, but not to git in general. Do you have a document to help me set up my workstation to do the pull/build? Sorry if it is obvious, but I didn’t poke around much, just submitted the comment. Thanks to Longhorn, I now have my own docker-registry, so… Kurt From: Gabriel Knepper Mendes <notifications@github.com> Sent: Tuesday, August 4, 2020 10:35 PM To: longhorn/longhorn <longhorn@noreply.github.com> Cc: Lund, Kurt <lund@uhcl.edu>; Mention <mention@noreply.github.com> Subject: Re: [longhorn/longhorn] Longhorn manager not run on RancherOS (#828) @uhcl-lund<https://urldefense.com/v3/__https:/github.com/uhcl-lund__;!!BCR0FSePrR4x!UZQmQlYf4qtTHW30IGwk_ONeUMPr6HNBqy8rrOdONQ6KWRdWCnwR0HgbkBeV$> I modified RancherOS to include stat and flock. I'll also include lsblk. I was wondering if you can test it ? Thanks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/longhorn/longhorn/issues/828*issuecomment-668960111__;Iw!!BCR0FSePrR4x!UZQmQlYf4qtTHW30IGwk_ONeUMPr6HNBqy8rrOdONQ6KWRdWCnwR0OulZNiq$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AQQFXDQNEUD2BFW7565WX6DR7DHM3ANCNFSM4JEBQF6Q__;!!BCR0FSePrR4x!UZQmQlYf4qtTHW30IGwk_ONeUMPr6HNBqy8rrOdONQ6KWRdWCnwR0NBglMmf$>.

[raelix]

Hi everyone, any updates on this? I still facing the same issue...

[gknepper]

@raelix I think because RancherOS it's in EOL, people are looking for other options.

[prosolitnl]

Switch console to alpine and reboot the node

sudo ros console switch alpine

[joshimoo]

@gknepper RancherOS is eol and not officially supported, some community members had success doing the necessary modifications to make it work. If you are looking for a replacement you can give k3os a try or ubuntu / sles / rhel.

[gknepper]

Thanks @joshimoo for the info. That's more or less what I assumed on January. I migrate my workload to ubuntu but I still miss the speed, the simplicity and the objectiveness of RancherOS. I'm trying OpenSUSE MicroOS. Concept wise it's promising but automation wise I'm still facing some issues when running in a VMware cluster with RancherOS node driver. BTW this thread is from 2019 maybe due covid-19 looks like almost a century to me. Take care you all!

[chaos95]

I tend to agree with @gknepper - neither k3os nor the other options suggested really meets the brief that RancherOS did. That said, there is an effort to maintain it moving forward over at BurmillaOS, so perhaps it might be worth submitting the proposed changes there.