Use the default username as provided by cloud-init

[Saviq]

To keep as close as possible with cloud deployments, we were asked to use the default username when issuing multipass shell, as provided by cloud-init.

It depends on the image being launched (cloud init's template), so we might need to have our own "service" user and use that to find out what user we should shell into.

[townsend2010]

For now, we are going to replace the multipass user with an ubuntu user on all newly launched instances regardless of the distro being launched. Any existing instances will still have the multipass user. I opened #949 to track this.

In a future enhancement, we will use the cloud images default user and also add a multipass user during cloud-init which will mainly be used to figure out the default user.

[basak]

It depends on the image being launched (cloud init's template), so we might need to have our own "service" user and use that to find out what user we should shell into.

That sounds like a really horrible hack. Could the required information perhaps be added to image metadata - simplestreams in the case of Ubuntu? It seems to me that defining this against individual images and tracking cloud-init userdata overrides if necessary would be a more appropriate layer for this ti fit into.

[townsend2010]

@basak,

Could the required information perhaps be added to image metadata - simplestreams in the case of Ubuntu?

We need a single source of truth for this if we are to get it from an external source. Other cloud images (such as Fedora) don't use simplestreams, so I don't think that is a place for it. We need a cloud image agnostic way of determining the default user.

Our proposed "hack" is to add a multipass user in addition to whatever the default user cloud-init sets up is. Then through issuing a command via ssh using the known multipass user into the instance, we can detect what the name of the default user is, so when someone does shell or exec, they get the default user regardless of the cloud image being used.

If you have any other ideas of how we can obtain the default user regardless of the cloud image being used, we are all ears 😁

[raharper]

IIRC, multipass uses the NoCloud datasource. I think we can combine a few features to make this work better.

Cloud-init supports adding a user and having it generate emit a redirect to default user message over ssh. I propose that multipass use this configuration to create the multipass user with redirection enabled (or some other username if you want to retain multipass user for some other purpose) and the message from the instance will reply with whatever the default user for that image was; this handles not having multipass have to know how cloud-init is configured in the target image.

Here's the config to make this happen:

In meta-data you need to add the user's public ssh key like so:

% cat meta-data
instance-id: e0c77a90-2bf7-45c0-b3b8-09c196582e32
public-keys:
  - ssh-rsa AAAA...

And in user-data, enable the default and multipass user with ssh redirect like so

#cloud-config
users:
  - default
  - name: multipass
    ssh_redirect_user: true

After the instance has booted, you can ssh to the instance as the redirect user and cloud-init emits this messge:

% ssh multipass@localhost /bin/true 2>dev/null
Please login as the user "ubuntu" rather than the user "multipass".

[townsend2010]

@raharper,

Cool stuff! A couple of questions.

1. Is that string that cloud-init prints stable? Since it's a message we'll have to key on, any changes to that message will mess up how we parse the string to find the user we should use.
2. Is this supported on older Ubuntu cloud image versions such as Trusty & Xenial?

[basak]

On Thu, Nov 07, 2019 at 08:33:47AM -0800, Chris Townsend wrote: @basak, > Could the required information perhaps be added to image metadata - simplestreams in the case of Ubuntu? We need a single source of truth for this if we are to get it from an external source. Other cloud images (such as Fedora) don't use simplestreams, so I don't think that is a place for it. We need a cloud image agnostic way of determining the default user.

I agree. I'm proposing that you move that single source of truth to the abstraction you're using for the image source. Make it the responsibility for the layer that supplies the image to also tell you the default username at the same time as providing you the image. In the case of simplestreams, this could be hardcoded to 'ubuntu' if you're only fetching Ubuntu, or it could come from simplestreams data itself (we can add it if we decide that's the way to go), or it could be supplied as a mapping from your list of simplestreams URLs if you have multiple image sources that way. In the case of other non-simplestreams image sources, those sources should know the default username to use as they specialise in those image sources, right? So they could supply that together with the image. Essentially I'm saying: consider the default username to be metadata associated with the image itself. I think this is a clean model and avoids hacks. You might not consider the truth quite "single source" any more because the source is the image source and there are multiple image sources, leading to multiple username sources. However I think this is still a clean model: you don't have a single source of images, and the default username varies according to the source, so it follows that the source of the default username would map to the source of the images. The default username belongs to an image.

Our proposed "hack" is to add a `multipass` user *in addition* to whatever the default user cloud-init sets up is. Then through issuing a command via ssh using the known `multipass` user into the instance, we can detect what the name of the default user is, so when someone does `shell` or `exec`, they get the default user regardless of the cloud image being used.

My concern about this is that I understand that a key goal of multipass is for developers to be able to test what would happen in production, and any difference can be very frustrating. An additional user added by the developer would in your proposal end up being uid=1002 with multipass (because the multipass user would have taken 1001) and yet uid=1001 in production. Even if you use system UIDs, the equivalent issue would apply in that part of the uid namespace. This sort of difference is really frustrating for developers when they run into them and should be avoided. Ideally multipass would run production images with no modification whatsoever. This is the point of cloud-init and currently a key differentiator against Vagrant. Hacking the image seems like the start of slippery slope going against that. Where you need it for UX, IMHO we should work out how to resolve it with the help of upstreams as required.

[raharper]

1. Yes. The format is hard-coded:
   https://github.com/canonical/cloud-init/blob/master/cloudinit/ssh_util.py#L44

2. This is supported back through Xenial. ssh_redirect_user was landed in Sept 2018, released in 18.4 and newer.

[basak]

Ideally multipass would run production images with no modification
whatsoever. This is the point of cloud-init and currently a key
differentiator against Vagrant. Hacking the image seems like the start
of slippery slope going against that.

To be clear, I realise that you're not actually "hacking the image" or modifying it. That was a poor choice of words, sorry.

What you are doing though is hacking the cloud-init userdata to be different from default in a way the developer won't do in production. I think that every instance of this should be avoided whenever possible for the reason I stated, and in this case it can be avoided so it should be avoided.

[raharper]

I'm proposing that you move that single source of truth to the abstraction you're using for the image source. Make it the responsibility for the layer that supplies the image to also tell you the default username at the same time as providing you the image.

Unfortunately, the "default" user in images has diverged over time; specifically in Non-ubuntu images. Over time as cloud-images update to newer cloud-init releases, they've been able to set a consistent name as supplied by cloud-init in it's default configuration in packaging. Additionally, downstream provides, both in packaging and in image publishing may also override these defaults with whatever name/values they choose.

This leaves few solutions. The primary means to knowin the value is going to be either provide the default user name and groups (groups are harder) or accept the default user as defined in the image and query the value.

[raharper]

What you are doing though is hacking the cloud-init userdata to be different from default in a way the developer won't do in production. I think that every instance of this should be avoided whenever possible for the reason I stated, and in this case it can be avoided so it should be avoided.

User-data is arbitrary; there are no "default" or "production" user-data values.

[townsend2010]

What you are doing though is hacking the cloud-init userdata to be different from default in a way the developer won't do in production.

Right, which is what this bug is about is figuring out a way for Multipass to be able to ssh into an instance, regardless of what cloud image is being used, in a consistent manner and be able to preserve the "default" user that cloud-init sets up. Yes, we did change the default to be "multipass" and as has been pointed out over time, this is wrong. As a stop-gap measure, we changed that to "ubuntu", but again, that breaks other non-Ubuntu cloud-images.

Since we allow launching cloud images via URL or even file based, there is absolutely no way for Multipass to know what distro's cloud image is being launched. This is why we need some way to "probe" the instance to figure out what the default user name is. Also, there is absolutely nothing that precludes a user to set their own "default" user via the --cloud-init option to multipass launch, so even if an Ubuntu cloud image is launched via SimpleStreams, the user may very well have changed the default user to foo using their own custom cloud-init data they passed in.

[townsend2010]

@raharper,

Thanks for the answers. So you are guaranteeing that the hard coded message will never change and will always be non-localized?

[raharper]

@townsend2010

No, and no. Software changes as do requirements.

What we'll want is to allow user-data to include a template for the redirect response so you know what to expect.

There are no pending changes to that string. Cloud-init does not currently have any localization.

[townsend2010]

@raharper,

Yeah, I kind of said that tongue-in-cheek, but I wanted to bring up that we need some sort of way to always be able to parse out the real default user name regardless of changes there. I appreciate your pointers and help on this!

[ricab]

No, and no. Software changes as do requirements.

I guess we wouldn't need a hard guarantee that it won't change, just that it wouldn't change silently along a version track.

So, a related question is: is it (or could it become) part of the interface in any way? Otherwise, I share Chris's unease depending on an implementation detail.

[raharper]

@ricab

Certainly. It's been untouched since it landed. Cloud-init team is aware. We can do two things.

1. merge in comments around the ssh_redirect_user documentation and code, as well as where the message is currently defined, warning against changing that. Multipass is not the only user of ssh_redirect_user config.

2. Accept a PR to allow user-data to provide a template for the redirection message.

[basak]

On Thu, Nov 07, 2019 at 09:58:54AM -0800, Ryan Harper wrote: > What you are doing though is hacking the cloud-init userdata to be different from default in a way the developer won't do in production. I think that every instance of this should be avoided whenever possible for the reason I stated, and in this case it can be avoided so it should be avoided. User-data is arbitrary; there are no "default" or "production" user-data values.

Well, there is. Default is what you get when you don't override with userdata. Put another way: I'd like multipass' default to be the same as cloud-init's default. It should, as much as is possible, minimise supplying userdata by default that causes cloud-init to change away from its default behaviour.

[basak]

On Thu, Nov 07, 2019 at 10:25:48AM -0800, Chris Townsend wrote: Since we allow launching cloud images via URL or even file based, there is absolutely no way for Multipass to know what distro's cloud image is being launched. This is why we need some way to "probe" the instance to figure out what the default user name is. Also, there is absolutely nothing that precludes a user to set their own "default" user via the `--cloud-init` option to `multipass launch`, so even if an Ubuntu cloud image is launched via SimpleStreams, the user may very well have changed the default user to `foo` using their own custom cloud-init data they passed in.

I don't think it's necessary for multipass to magically default to the correct username in _all_ cases. Clearly that's impossible because I could always come up with a userdata customisation such that it won't be able to guess (eg. move ssh to a different port number, or replace all users with a set defined from a configuration management tool, etc). So it would be sufficient to cover enough reasonable cases. With cloud-init's help, parsing the custom userdata the user passed in would I think be a better hack than modifying the userdata to install an extra user account that the user isn't expecting. This is surely especially true in the case that the user is customising user accounts via userdata!