Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Blockers for systemd? #994

Open
fpqc opened this issue Aug 27, 2016 · 78 comments
Labels

Comments

@fpqc
Copy link

@fpqc fpqc commented Aug 27, 2016

For various reasons, it isn't obvious to me exactly how to determine what features are missing from WSL's implementation in order to get something like systemd up and running. From the documentation, we know for sure that WSL does not yet implement any of the syscalls related to cgroups, and doesn't include any of the kernel namespaces, which is probably a blocker at the moment. Are there any other missing syscalls or socket options that would block systemd from running?

I also am attaching the output of
strace systemd --test

systemd-trace.txt

(using the docker image of Ubuntu yakkety installed with RoliSoft's python script)

Also, I know that WSL currently uses a kind of hardcoded init process (which sets up the network interfaces, remakes the /etc/hosts file, does some other stuff, then (I think) forks /bin/bash by default). Would the whole initialization process need to be rewritten from scratch to use systemd as the init, assuming all of the other dependencies were eventually implemented (cgroups, aforementioned missing syscalls and socket options)?

@iz0eyj

This comment has been minimized.

Copy link

@iz0eyj iz0eyj commented Aug 27, 2016

Something in this old thread ( @jackchammons ), but it seems that WSL (and all the Win10 kernel design) is a Top Secret. :)
https://blogs.msdn.microsoft.com/wsl/2016/06/08/wsl-system-calls/

@rodrymbo

This comment has been minimized.

Copy link

@rodrymbo rodrymbo commented Aug 27, 2016

Considering how open the Devs have been hereabouts, I'd say the issue isn't so much "keeping it secret" as "it's a moving target".

I don't think it will take too long for the Devs to figure out that what they are presenting to us is "single-user, console". To a Linux person it is like Safe Mode is to a Windows person. Yes you can do stuff in Safe Mode, and sometimes you are amazed that it even works at all, but it's not the same as running Windows, just as running WSL without cron or syslog or sshd or a terminal emulator (what we have got is sort of a console emulator) and other servers/daemons/background tasks, is not the same as running an Ubuntu developer machine.

I can be happy without upstart/init/systemd running everything that comes with a fresh Ubuntu install. But the things I work on generally do logging to syslog, and don't have an option for logging to a file; so trying to work on them without rsyslogd is quite difficult. It's nice to be able to run a bash script from a shell prompt, but what would be nicer is being able to run a few from cron, to automate things. And so forth.

So let's see if we can figure out what is blocking systemd, but I'd guess that when the Devs decide it's something they need to implement, they'll figure out a way.

@fpqc

This comment has been minimized.

Copy link
Author

@fpqc fpqc commented Aug 27, 2016

@rodrymbo I'm just interested for my own curiosity. Like, is the main blocker just cgroups, or is it something more difficult? Like, is WSL architected in such a way that its init process needs to be a special one generated by MS, or did they just make a stripped-down init process because they haven't implemented cgroups and didn't want to bother supporting upstart when they knew it was already obsolete?

Basically, I want to compare against this:
Gentoo Kernel Options for Systemd

Aside from cgroups, the requirements don't seem that crazy to emulate. I don't see the specific syscalls mentioned in that guide listed in the list of supported syscalls for WSL, but on the face of it, you need cgroups, "open by fhandle syscalls", eventpoll syscall, signalfd syscall, and timerfd syscall. The rest of the requirements appear to be things that are at various stages of implementation.

P.S. Mercifully, it looks like the Redhat devs have given up for now on pushing things like dbus into the kernel for the forseeable future (there was a big blowout on the kernel.org mailing list over kdbus, and kdbus was dropped from Fedora Rawhide a bit later).

@iz0eyj

This comment has been minimized.

Copy link

@iz0eyj iz0eyj commented Aug 27, 2016

@fpqc init, upstart etc etc requires a continuous operation; in this moment WSL exists only when calling bash ... perhaps in the future will be different ... (?)
I think now you could only simulate an init or an upstart, but does not implement them really (emulate them)
for system calls I think that certain things cannot be implemented (emulated) because WSL is not a standalone operating system, but I see no reason why they cannot be simulated at least providing a dummy response.
I think the reason is only the young age of WSL, just a few months, but at the end we will have a very close to the "true" Linux.

Sorry for my bad English.

@fpqc

This comment has been minimized.

Copy link
Author

@fpqc fpqc commented Aug 27, 2016

@iz0eyj That's not true, do
$ ps -A

You will see that the init daemon is running at pid 1.

@rodrymbo

This comment has been minimized.

Copy link

@rodrymbo rodrymbo commented Aug 27, 2016

@fpqc - The Microsoft init (/init, rather than /sbin/init or whatever) is definitely proprietary. It and any other parts of the WSL environment are probably written to meet the bare minimum for whatever requirements they thought would be needed. If the Devs decided they needed cgroups, it will be there, and if they didn't, it probably won't. :P

Not saying you shouldn't be curious, or that trying to figure it out based on clues might not provide an interesting way to pass time.

It looks to me as though they could implement most of a whole linux userland system if they wanted, and much more of the rest of the system, by which I mean to a user everything would work as expected including services. So the issue will likely be how much of that can users like us convince them (in Uservoice or whatever) is worthwhile.

@iz0eyj - right now, most of what you describe is happening: when Bash.exe runs, init is running (apparently launched or managed from svchost.exe or lxssmanager or something). No, it doesn't start when Windows starts (yet) but neither does each Bash.exe console start another init. And yes, it does stop (and tear down the environment) when the last Bash.exe is closed; but until that last Bash.exe closes, init is there, running along in the background. From my perspective, the tearing down of the linux environment is just a matter of the Devs requiring that the system "shutdown -h now" when the last Bash.exe closes. So in a sense, init is running continuously ... when the environment is running.

I agree that for many things it would be good enough for init (or whatever accepts system calls) to return dummy responses that satisfy whatever we are trying to run. For example, ifconfig could return what appear to be two devices, a loopback device and an ethernet device -- even though they don't really exist. (Or maybe there is a pseudo-device that would work without pretending to be a hardware adapter.) As long as networking works, I won't be too picky. I suppose someone will need a VPN, or more control over a tun/tap, and maybe that can come in the future.

@fpqc - For some things it is helpful to look in Process Explorer to see that init is running under a svchost.exe. What you see from ps -A (or ps -Af) could just be a different init in each different bash.exe console, but checking Process Explorer shows that it's the same one (or at least it appears to be) and it is running independently of Bash.exe (aside from being shutdown when the last Bash.exe exits).

@rodrymbo

This comment has been minimized.

Copy link

@rodrymbo rodrymbo commented Aug 29, 2016

@fpqc - is there a reason for implementing systemd in WSL other than bragging rights? It looks to me like that would bring along a lot of baggage we might not need. Could we go to something closer to sysvinit or whatever for starting a few services, or would having systemd implemented give us a lot of benefits I'm not seeing?

@cerebrate

This comment has been minimized.

Copy link

@cerebrate cerebrate commented Oct 9, 2016

For myself, I would like to be able to run systemd not to replace the WSL init, but because being able to start a pseudo-init with systemd --system --unit=foo.target would be a convenient way to start and subsequently manage a set of services/daemons required for a particular development, etc., configuration. By having multiple .target configuration files, that way systemd could be used to set up whatever service-set happened to be needed for a given session in a suitably Linuxy way.

(And, I suppose, those who want services running all the time could put bash.exe -c /bin/systemd into a startup shortcut, but that's just gravy compared to the convenient start-and-manage service functionality that'd come out of it.)

((I added a uservoice request for this here, BTW: https://wpdev.uservoice.com/forums/266908-command-prompt-console-bash-on-ubuntu-on-windo/suggestions/16571479-support-for-running-systemd .))

@schmitch

This comment has been minimized.

Copy link

@schmitch schmitch commented Apr 7, 2017

Well this future would be great to use PostgreSQL or MySQL in Xenial instead of the Windows Installer.
Which would be really really huge, especially if systemd would be invoked on Windows startup and not on bash Startup.

The creators update did many things right, but this would be extremly huge as well. The best thing would be that Windows would be clean and the Development environment can be trashed and cleaned really really easy.

@carwyn

This comment has been minimized.

Copy link

@carwyn carwyn commented Jun 8, 2017

If the desire is to manage a set of services under WSL in a more "linuxy" way it may be worth generalizing this discussion to look at other light weight approaches of doing this. SysV init is one option but there are others that have been developed for more constrained environments that may be of interest for example:

I'd hazard a guess that systemd may be one of the trickier ones to implement let alone track.

I suspect the main motivation for WSL is twofold: for developer tools and for running containers. In both cases they tend to be single thing or chain/group of things per invocation.

@fpqc

This comment has been minimized.

Copy link
Author

@fpqc fpqc commented Jun 8, 2017

@carwyn All of the distro devs with whom MS has announced support (Ubuntu, Redhat, and OpenSuse) are now firmly in the Systemd camp, and all of their package managers and packages assume systemd is running. MS does not want to maintain its own distribution of Linux userland for WSL. Presumably if they do make a switch over to systemd from WSL-init, there should be nothing stopping you from writing shellscripts to wrap the interop daemon to work with sysVinit scripts.

@carwyn

This comment has been minimized.

Copy link

@carwyn carwyn commented Jun 8, 2017

@fpqc granted, although there is also a MS relationship with Docker although that seems more about container management rather than what's in them. Having said that Docker seem to be sniffing around Unikernels a lot.

@bugparty

This comment has been minimized.

Copy link

@bugparty bugparty commented Nov 9, 2017

one of Ubuntu official develops says M$s intent is not a complete linux, its just something like chroot.

@seadonfrank

This comment has been minimized.

Copy link

@seadonfrank seadonfrank commented Nov 14, 2017

You may want to try using Upstart to run and manage third-party services
Also, Follow me on linkedIn , I am going to publish an article very soon on WSL and running opensource project with WSL. Hopefully, it should give you a comprehensive guide while you start working with WSL

@therealkenc

This comment has been minimized.

Copy link
Collaborator

@therealkenc therealkenc commented Dec 26, 2017

@kenshen112 writes in another issue:

...all .service files in wsl are by default located in the /lib folder mentioned above whilst they should be in the /etc equivalent.
...
...If the wsl team could move the file locations and fix permissions we might see systemd start.

The locations of those files are controlled by Canonical via the Debian systemd Maintainers via Freedesktop.org, not WSL. If you believe moving the file locations will help you could create a launchpad PPA for systemd.

My /etc/systemd/system looks alright enough on WSL. Which it does because it is just the stock Ubuntu image and systemd package. Could be missing something, but even if I am, that isn't the end of your voyage. Getting closer though (for some relative value of 'close`).

@kenshen112

This comment has been minimized.

Copy link

@kenshen112 kenshen112 commented Dec 27, 2017

@therealkenc as far as I understand all .service files should be in /etc/systemed which most aren't and that's what causes the error as far as I can see if i execute the command in my post copying the files from there previous location to /etc/ it works aside from the permission's issue. if that's how conical has it by default though then I wonder if there's a symlink missing somewhere???

@therealkenc

This comment has been minimized.

Copy link
Collaborator

@therealkenc therealkenc commented Dec 27, 2017

I haven't looked into systemd for a few months, and I am not sure why the .service entries you think might be missing aren't there. This is what mine looks like (unmodified) for what it is worth. But missing entries shouldn't cause systemd to fail (as far as it goes); it would just mean less services start. What methodology are you using to start systemd, if any? You are running systemctl status in #2787, but that is putting the horse before the cart because systemd isn't running per #2209 #1579.

@kenshen112

This comment has been minimized.

Copy link

@kenshen112 kenshen112 commented Dec 27, 2017

so first off https://askubuntu.com/questions/894419/systemd-on-ubuntu-16-04-gives-no-such-interface-error one small example of the same problem / fix occurring in ubuntu 16.04
after I make the change my etc folder looks like https://pastebin.com/u3UkEeeH but now there's permission issues as previously stated.

sysctrl is a tool that allows one to create and manage service files used by systemd - the reason I said it could help us https://www.digitalocean.com/community/tutorials/how-to-use-systemctl-to-manage-systemd-services-and-units it manages the stopping and starting of systemd services a key piece of what systemd does as a whole so whilst it might not fully fix systemd it'll give us some of the functionality of it at least for sysadmin's that's incredibly useful.

@fpqc

This comment has been minimized.

Copy link
Author

@fpqc fpqc commented Dec 28, 2017

@kenshen112 no. Only changes or custom systemd unitfiles go into /etc/systemd.

default unitfiles belong in /usr/lib/systemd/system

Systemd provides tooling to build and store diffs of default unitfiles in your /etc/systemd/system as well as completely custom unitfiles.

@therealkenc

This comment has been minimized.

Copy link
Collaborator

@therealkenc therealkenc commented Jun 14, 2019

@therealkenc I at least had to manually mount the cgroups or else systemd errored out on startup.

Try the one liner off a clean boot and clean install without whatever it is was futzed in /etc/systemd (tbh I didn't look). If it errors out post a screencap whatever other details you can think of, and we can dive a bit. The PID 1 systemd doesn't expect a cgroups to be there before it was run. It is the first thing running (see, because PID 1).

it seemed to me that this adds a fair bit of overhead in terms of things I don't want or need (looking at the above, timesyncd, irqbalance, lvmetad, accounts-daemon, sshd

Sure, absolutely valid to decide you don't want systemd-timesyncd. But not special. If you don't fancy systemd-timesyncd (maybe you prefer chrony instead), then inside your container do:

$ systemctl stop systemd-timesyncd
$ systemctl disable systemd-timesyncd

The next time you start up it won't be there.

But n.b., none of those daemons are there for their health. Or see, they wouldn't fire up on Ubuntu. I'll give ya irqbalance, for us, maybe, for our not-quite-type-1 hypervisor. But then irqbalance isn't hurting anybody either. There might even be some weird use case that wishes it were running. Disable it. Or not.

That all said, a custom target is of course a perfectly valid way to go in some special cases. That's why custom targets exist. And indeed, containers are 'Exhibit A' of a special case.

Bringing us to the one elephant in the room which you mention: agetty. Ubuntu's stock targets launch agetty, and we don't really have a tty1 (or tty2 or tty3) in a container scenario (unless you were to fake them).

When you start up WSL you'd see more than one /init process. One of those is agetty by another name. In a more perfect world, WSL /init would be broken up and we could use it.

I'd need to enter the namespace every time I run a new shell or wsl something.

Sure do.

A complete solution would cleverly wrap pretty how to enter (log in to) the namespace and slowly converge on lxc in a death march. You can also forgo wsl.exe entirely and enter your container with Windows ssh.exe (wrapped up all pretty natch). Tempting for me to post something minimal on github, but then there is the risk of it being treated as some kind of official "WSL" systemd solution.

The only drawback I've run into so far

No. You aren't even in the right cgroup tree let alone the right mount and PID namespace. Not really worth getting into the weeds here, since none of this is related to WSL and this post is TL already. Lots of sites dedicated to systemd and containers. I just trying to head off a spiritual repeat of # 637 and have a dupe target.

Keep experimenting though and hopefully have some fun doing so! ❤️

@cerebrate

This comment has been minimized.

Copy link

@cerebrate cerebrate commented Jun 15, 2019

I'd presumed we'd specifically not want systemd-timesyncd because the Windows host is already doing time sync for the system clock, and doing it twice over seems unhealthy.

Anyway: thanks muchly for the education. Since no-one seems likely to treat anything of mine as remotely official, I've hacked together my own fairly-minimal helper tool for this, which anyone is welcome to use in an entirely unofficial way:

https://github.com/arkane-systems/genie

@therealkenc

This comment has been minimized.

Copy link
Collaborator

@therealkenc therealkenc commented Jun 16, 2019

Windows host is already doing time sync for the system clock

The word is "only" not "already". As in: WSL only sets the time at startup. There is no chrony equivalent built into WSL's house brand init. That is, there is no "sync". You could probably live with the clock being set just at startup on Real Linux too. A long time ago in ye Olden Tymes Linux worked like that. But I bet you didn't disable systemd-timesyncd on your Real Ubuntu box because (quoth) "it seemed to me that this adds a fair bit of overhead in terms of things I don't want or need". Amirte?

FWIW there's no equivalent of systemd-logind in WSL's /init either. Interesting to learn there exists a project elogind created just for distros that don't have systemd. Funny quote answering "why bother" on that page. Sounds like us, eh?

One last since I'm here. Bet your tmux zombies get reaped properly if they are hung off the correct part of the process tree (inside the bottle) with your genie tool. So workaround-available on your other recent submission. ;) Don't know if you've noticed yet but shutdown(8) works properly now too (#3253). sudo shutdown now inside the container.

Fantastic to see code up on github. 👍

@cerebrate

This comment has been minimized.

Copy link

@cerebrate cerebrate commented Jun 16, 2019

That (that the time is only set on startup) I knew, but I was thinking more that as I understand it, systemd-timesyncd writes updates to the hardware clock as well as the system clock, and so I'd end up with it and the Windows Time service both trying to make corrections to it. I can always set up a timer to do a periodic hwclock -s to keep in sync.

And yes, the tmux zombies do indeed get reaped properly inside the bottle! I was wondering if there'd be problems with that due to /bin/login's zero ppid, but seems not.

@therealkenc

This comment has been minimized.

Copy link
Collaborator

@therealkenc therealkenc commented Jun 16, 2019

timesyncd writes updates to the hardware clock as well as the system clock ... both trying to make corrections to it

Oh no no no. It is a virtual clock in WSL2 (read: Hyper-V). Not a real clock. If you set the hw clock manually inside WSL, nothing happens in Windows. Because they aren't the same clock "hardware". In fact if you set the hwclock in WSL, do a wsl.exe --shutdown, and fire wsl back up, the hwclock changes will be lost. Because the fake clock doesn't have a fake battery.

image

@kidpollo

This comment has been minimized.

Copy link

@kidpollo kidpollo commented Jun 21, 2019

@therealkenc yay got your one liner working on Arch. I assume in the future Distros with wsl will come with systemd running correctly?

@therealkenc

This comment has been minimized.

Copy link
Collaborator

@therealkenc therealkenc commented Jun 21, 2019

I assume in the future Distros with wsl will come with systemd running correctly?

Dunno (wouldn't know). I wouldn't personally assume one way or the other, at least in the medium term.

@fpqc

This comment has been minimized.

Copy link
Author

@fpqc fpqc commented Aug 7, 2019

@mscraigloewen I saw the announcement for the new global options for WSL. Do you think it would be possible to add an option to enable systemd support in WSL2 containers using the techniques here? It looks like it's basically all up and working.

Or I guess alternatively it would be nice to know if this is something in the pipeline.

@amaothree

This comment has been minimized.

Copy link

@amaothree amaothree commented Nov 7, 2019

Find an interesting solution here.
yuk7/ArchWSL#20 (comment)

You can work around the issues with systemctl by replacing it with this script by @gdraheim:
https://github.com/gdraheim/docker-systemctl-replacement/blob/master/files/docker/systemctl.py
just do the following (assumes /usr/bin/systemctl) :
mv /usr/bin/systemctl /usr/bin/systemctl.old
curl https://raw.githubusercontent.com/gdraheim/docker-systemctl-replacement/master/files/docker/systemctl.py > /usr/bin/systemctl
chmod +x /usr/bin/systemctl

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.