This patch provides a friendly interface with the masterzen/winrm client #249

hoenirvili · Nov 1, 2016

For a better integration in the juju ecosystem enabling many windows
machine interactions. This will also help us execute remote commands and scripts
across the wire.

dooferlad · Nov 9, 2016

Hi,

Thanks for the PR!

This is big and introduces new dependencies. Since at least https://github.com/masterzen/winrm is under a different license to juju/utils and juju/juju I would like to defer to someone who is able to work out what the implications are, if any, of incorporating them.

Please provide CI steps so the reviewer can confirm that the change works.

Please split the change up into smaller chunks of work. It seems that at least the changes in the series and shell folders could land independently of the winrm change.

The x509 certificate looks generic - is that right? If it is a specific winrm formatted certificate then I see the reason for including it in the winrm folder. Wouldn't combining this with github.com/juju/juju/cert/cert.go be useful? If so, moving code out of core into a security library is a good thing.

axw · Nov 10, 2016

@hoenirvili I second @dooferlad's question about using juju/cert. I'm away this week, will review this when I get back.

gabriel-samfira · Nov 10, 2016

@dooferlad The license for the winrm package is Apache 2.0 which should be consumable by juju/utils without any issues.

The x509 is mostly generic. The only winrm specific thing it does, is that it adds a UPN (User Principal Name) extension to the certificate. That extension is used to map the certificate to a particular system user (not unlike adding a OpenSSH key to the users ~/.ssh/authorized_keys).

Moving the x509 code to juju/cert should be fine. The reason we added it to the utils package was that we wanted to mirror the ssh package as closely as possible. Conceptually they have the same purpose (including the key generation part), so we wanted to have all the code dealing with access keys and remote commands in one place.

axw · Nov 10, 2016

@gabriel-samfira another option is to split out the more generic parts of juju/cert, and move them into utils/cert (say). Then we can have winrm-specific bits inside winrm, and delegate to the generic bits in utils/cert.

gabriel-samfira · Nov 10, 2016

@axw sounds good!

hoenirvili · Nov 10, 2016

@axw sounds good ! I will move the generic code from juju/cert to utils/cert.
(edit)
@axw how does the manual provisioning (linux side) is integrated in the CI?

hoenirvili · Nov 10, 2016

@dooferlad the juju/cert is creating CA x509 certs but the winrm cert generation creates client certs to use for authentication purposes.

axw · Nov 10, 2016

@axw how does the manual provisioning (linux side) is integrated in the CI?

We have a subset of CI tests which run the exact same thing as for manual machines as for others. The only difference is that they use "add-machine ssh:..." and then deploy --to. We have some machines (I think LXD instances) running all the time, and each test run reprovisions the same machine.

For this branch, I don't think there are any QA steps required, as none of this is hooked up yet. Once winrm provisioning is hooked up, then QA steps will need to be listed describing how we can test your changes. e.g. start a windows VM, , juju add-machine winrm:...

hoenirvili · Nov 14, 2016

@axw Could you take now another look?

axw · Nov 16, 2016

axw requested changes Nov 16, 2016

View changes

Looking good overall. My main gripes are about the reliance on globals. I haven't gone into depth on the tests, as I expect they'll change significantly based on interface changes.

can you please update the cert.NewLeaf function so that you can pass in pkix.Extension, and then rewrite this on top of that function?

Could you please add more details? Why should I add a pkix.Extensions to the NewLeaf function? And why should I rewrite this NewClientCert using the NewLeaf function?

AFAICS, the only difference we care about between NewClientCert and NewLeaf is that NewClientCert adds the UPN extension. So to avoid repeating all the generic certificate generation code here, I'd rather we used the NewLeaf to generate the cert, and provide the UPN extension as an argument to NewLeaf.

Fixed !

please start error messages with lowercase, and prefer "cannot" over "can't"

i.e. errors.Annotate(err, "cannot construct ...")

Fixed !

I think it would be more useful to use a shorter input, and check the exact output of the function.

Fixed !

s/Sec/Secure/ please

Fixed !

This is not goroutine-safe. Please use winrm.NewClientWithParameters instead.

You could take a copy of winrm.DefaultParameters here and override the TransportDecorator field, but please don't write to the global variable.

Fixed !

Can you please add a param struct for New(Secure)Client, and include the client cert, key, CA cert, and timeout? The struct should have a "Validate() error" method which this function calls. The Validate method should check things like cert&key are both provided, or both nil; timeout is non-zero; CA cert is provided, or explicitly disabled.

If you do all that, then you may as well just have a NewClient function and no NewSec(ure)Client. You can control the behaviour by setting or not setting the client cert/key.

Fixed !

s/clinet/client/

Fixed !

OK, so param struct should have: client cert, client key, and a GetPassword function. Validate should take care of ensuring that client cert/key are mutually exclusive with password (if that's indeed the right thing to do).

Fixed !

Please leave printing to stdout to the GetPasswd function.

Fixed !

This seems like a strange interface. Why is this stateful? Why wouldn't you just have this return the error directly?

i.e.

_, err := c.conn.Run(command, stdout, stderr)
return err

If you look at the entire utils/winrm , the package is made with this principle in mind, of recording the error inside the client itself and future(client) operation return immediately if there was a previous error, making them no-op. More on here: https://blog.golang.org/errors-are-values

I don't think the Scanner pattern is appropriate here. Run and Ping are not typically going to be used in a series, but as independent calls.

At the bottom of the blog article there is this comment, which is relevant:

There is one significant drawback to this approach, at least for some applications: there is no way to know how much of the processing completed before the error occurred. If that information is important, a more fine-grained approach is necessary. Often, though, an all-or-nothing check at the end is sufficient.

Ok @axw I see your reason, but please give me some advice, suggestions on how to make the API interactions better. We should also keep in mind that we need just only one client connection across the entire provisioning process.
Something like:

  conn := winrm.NewConn()
  doAllTheStuff(conn);
  winrm.Close(conn);

In my opinion I tried to do this using a global instance inside the utils/winrm package and make some wrapper methods around that, in order to make it more friendly I avoid using the api likewinrm.ThatInstance.Method(), instead we have. winrm.Method() and with methods like UseClient(), UseSecureClient() we are telling the client of the API to specify the method we should interact with the winrm listener, similarly we could introduces some global var to be set. Also if you saw, I was trying to mimic as much as I can the ssh pkg code, to not make it radically different.

What I think would be natural is:

package winrm

func NewClient(Config) *Client

type Client struct {...}

func (*Client) Close() error
func (*Client) Ping() error
func (*Client) Run(cmd string, stdout, stderr io.Writer) error

(I would prefer "Run" were "Command", like in ssh, but it may be overkill.)

Re mimicking the ssh package: I think the way the ssh package was done was a mistake, which I'd rather we didn't repeat. Maybe you can point me at some code that you think will be awkward if you get rid of the global?

@axw if you look on the other PR(juju/juju#6523) you can see I extensively use winrm.Run() after I initialise it in the InitAdministratorUser function, It would be a little unmaintainable to make NewClient() and Close() every time I run winrm.Run() into a function, I should not create more connections.We should just one connection for trying https auth vs password auth interactions for the entire provisioning process or other processes that will be used in the juju-core.

@hoenirvili sorry, I was confused because there's a bit of interchanging of "Client" and "Connection" in this thread. Connection implies, to me, something that needs to be closed. The client isn't connection-oriented, though, so Close isn't necessary. So the API would just be:

package winrm

func NewClient(Config) *Client

type Client struct {...}

func (*Client) Ping() error
func (*Client) Run(cmd string, stdout, stderr io.Writer) error

My previous statement about there being no sensible place for a single CA cert for all servers remains, so I still don't think it's sensible to have a global client. Not to mention it's racy: if you had two things in the same process trying to manually provision two different machines, we'd be in trouble. This isn't just academic, as we may want to move the bulk of manual provisioning responsibility to the controller sooner or later.

Seeing as you're passing in "DefaultClient" and "SecureClient" anyway, I see no reason why you would want to then pass that into a global and then use the global. Why not use the values directly? Doing so avoids concurrency issues, and makes testing easier.

Finally, on usage in PR juju/juju#6523. As above I was mistaken about the need for Close, so you needn't create a new Client per Run. Instead, just pass the pair of clients in (or a Client constructor function), and use them directly for the lifetime of the ProvisionMachine call. The Ping calls would be more idiomatic if the method returned an error, e.g.

at https://github.com/juju/juju/pull/6523/files#diff-8eee7a4e81e6a28a0cbc14fa50ae0b6bR128

winrm.Ping()
first := winrm.Error()
if first == nil {

would become

first := args.SecureClient.Ping()
if first == nil {

The Run calls would be slightly more verbose (i.e. with immediate error checking), but (a) there aren't that many Run calls, and (b) the errors will be directly relatable to the source (rather than "an error happened somewhere along the way", which you may or may not be able to determine the context of from the error message).

In case all of that is unclear, here is what I think InitAdministratorUser should look like:

func InitAdministratorUser(args manual.ProvisionMachineArgs) error {
        logger.Infof("Initalising %q, user %q", args.Host, args.User)
        logger.Infof("Using WinRM secure client as user %q on %q", args.Host, args.User)
 
        first := args.SecureClient.Ping()
        if first == nil {
                logger.Infof("WinRM secure authentication is already enabled on the host %q", args.Host)
                return nil
        }
        logger.Debugf("Secure authentication is not enabled on user %q", args.User)
 
        if args.DefaultClient == nil { // if it's not already custom set use the default
                // at this state the secure listener is not enabled and
                // we must enable it first
                args.DefaultClient = winrm.NewClient(args.User, args.Host, winrm.TTYGetPasswd, args.Stdout)
        }

        last := args.DefaultClient.Ping()
        if last == nil {
                logger.Infof("WinRM normal connection is enabled on the host %q", args.Host)
                return nil
        }
        logger.Errorf("Normal authentication is not enabled on the host %q", args.Host)
 
        if first != nil && last != nil {
                return errors.Annotatef(first, last.Error())
        }
 
        if first != nil {
                return first
        }
 
        return last
}

(Bear in mind that ProvisionMachineArgs is a value type, so the update to DefaultClient won't persist beyond the end of the function. I think it might be better to require that ProvisionMachineArgs has a SecureClient and a DefaultClient on entry to the exported functions, using a Validate method.)

Fixed !

this last sentence doesn't belong here, but on the top-level Ping method

Fixed !

as with Run, I think this should just return the error directly

Fixed !

please define a constant for the string that we're expecting back

const echoPayload = "HI!"
c.Run("echo " + echoPayload, ...)
...
if stdout.String() == echoPayload {
...
}

Fixed !

are the errors coming back from winrm not structured? what's a sample error? slightly concerned that just checking for "401" in the string is a bit too loose.

The errors that are coming back from winrm are not structured. That's why I check if 401(unauthorized request) is in the error string to know if we failed to access the endpoint because of unathorization problems or because the endpoint doesn't have winrm listener enable.

can we be more precise than Contains? I think we can do equality, probably stripping off newline from the right?

Fixed !

The content of ErrPing ("Ping failed, can't recive any response form target") doesn't match the behaviour here. I'm not convinced there's a need for a top-level ErrPing var. Instead, I would do:

    return errors.Errorf("unexpected response: expected %q, got %q", echoPayload, stdout.String())

Fixed !

What value are the default client & top-level functions providing? I don't want to see them used in Juju, and this code is all about making winrm easier for Juju... so I'm inclined to think that we should drop the default client, UseClient, and other related top-level functions.

Check the upper last comment.

start errors with lowercase please

Fixed !

I'm not keen on the use of globals here. I guess this is derived from the ssh/clientkeys.go code, which frankly is a bit ugly. The forces that directed ssh/clientkeys.go do not apply to winrm. In particular, we need keys for ssh on disk because we shell out to openssh. We don't do that for winrm.

Can you please change the code to parse and return certs/keys in-memory. I'd also prefer if we separated generation from loading. The caller can intercept an errors.IsNotFound(err), and generate a cert/key pair and write it to disk. I expect we'll only need to do that in one location in Juju, like we do for ssh.LoadClientKeys.

Please call them "keyFile", "certFile" or "keyPath", "certPath". Named as they are, I would expect them contain the key/cert values.

Fixed !

As before, please don't use globals. It seems unlikely that we would have a well-defined location for a CA cert on disk, as it's going to be different for each server. In Juju, I guess there's two places where we would need to specify a CA cert:

when manual-provisioning, using a CA cert specified by the user on the command line
when doing the equivalent of "juju ssh" ("juju winrm"?), using a CA cert published by the Juju agent on the target machine

Fixed !

axw · Nov 24, 2016

axw reviewed Nov 24, 2016

View changes

Thanks, I am happy with the interface now. Can you please update the remaining tests and delete unused code, so I can give the final review (hopefully!)

Please update this function to call utils/cert.NewLeaf, instead of duplicating logic here. This can be in a follow up if you like.

Fixed !

This is not fixed. Did you miss a commit? Maybe I'm being unclear.

What I want to see is:

no call to x509.CreateCertificate in this function
calling NewLeaf from github.com/juju/utils/cert (https://github.com/juju/utils/blob/master/cert/cert.go#L119) to create the certificate

To do that, github.com/juju/utils/cert would need to be updated so that you can pass x509.ExtraExtensions into NewLeaf. Let's just leave a TODO for now, and tidy that up later.

This call of x509.CreateCertificate is inside of NewLeaf function. If I put the call of NewLeaf here, it will make it in a recursive loop.
And I was having some issues with the commit log.

I'm not saying to call the NewLeaf function that is defined in this package. I'm referring to the one in utils/cert. utils/winrm.NewLeaf should initialise pkix.Extensions, and then call utils/cert.NewLeaf.

Anyway, let's defer this for now, this PR has gone on long enough ;)

Wait... this is the cert package. I think something was moved and I didn't notice, I'm sorry for the confusion.

Hrm, I'm not so sure about changing the common utils/cert interface, but let's go with it for now. I may end up changing it a bit.

Sorry, what I was asking for was for the function in this package to set up the ExtKeyUsage (any any other bits specific to winrm), and then call utils/cert.NewLeaf. There's a bunch of duplicated code in this package that isn't specific to winrm.

Fixed !

please move the fmt.Fprintf prompts to the GetPasswd implementation. this code shouldn't assume GetPasswd is interactive.

I think these tests all need to be updated now. There's no ResetClientCert, LoadClientCert, ClientCert, ClientKey, etc. Probably you should just have pre-generated certs/keys checked in the winrm_test package, and use them in your tests. Avoid generating new certs/keys for each test (or even the suite, if possible), because it's expensive and slows down tests. Not avoidable in all cases though.

Fixed !

hoenirvili · Nov 24, 2016

Sorry for the delay @axw I will make a new commit.... somehow git doesn't' include the last fixes after rebasing with master, I don't know what happened.

axw · Nov 24, 2016

axw reviewed Nov 24, 2016

View changes

A few last things, and this will be good to merge. Main thing is to either fix tests for, or remove the x509.go code.

Please update this function to call utils/cert.NewLeaf, instead of duplicating logic here. This can be in a follow up if you like.

Fixed !

This is not fixed. Did you miss a commit? Maybe I'm being unclear.

What I want to see is:

no call to x509.CreateCertificate in this function
calling NewLeaf from github.com/juju/utils/cert (https://github.com/juju/utils/blob/master/cert/cert.go#L119) to create the certificate

To do that, github.com/juju/utils/cert would need to be updated so that you can pass x509.ExtraExtensions into NewLeaf. Let's just leave a TODO for now, and tidy that up later.

This call of x509.CreateCertificate is inside of NewLeaf function. If I put the call of NewLeaf here, it will make it in a recursive loop.
And I was having some issues with the commit log.

I'm not saying to call the NewLeaf function that is defined in this package. I'm referring to the one in utils/cert. utils/winrm.NewLeaf should initialise pkix.Extensions, and then call utils/cert.NewLeaf.

Anyway, let's defer this for now, this PR has gone on long enough ;)

Wait... this is the cert package. I think something was moved and I didn't notice, I'm sorry for the confusion.

Hrm, I'm not so sure about changing the common utils/cert interface, but let's go with it for now. I may end up changing it a bit.

I think you want Cloudbase here.

Fixed !

The tests are all commented out, and this isn't used in the winrm package. Will you be using it in the juju code? If yes, please uncomment the tests and make sure they're working. If not, please delete the code.

Ohh sorry I forgot to uncomment these. Fixed !

axw · Nov 27, 2016

axw approved these changes Nov 27, 2016

View changes

please do not use a hard-coded temporary dir

Fixed !

gocheck.C has a method called MkDir that will create a temporary dir, and return its path. Please use that instead of the hard-coded "/tmp/base".

Fixed !

hoenirvili · Nov 28, 2016

$$merge$$

jujubot · Nov 28, 2016

Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju-utils

jujubot · Nov 28, 2016

Build failed: Tests failed
build url: http://juju-ci.vapour.ws:8080/job/github-merge-juju-utils/154

hoenirvili · Nov 28, 2016

$$merge$$

jujubot · Nov 28, 2016

Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju-utils

jujubot · Nov 28, 2016

Build failed: Tests failed
build url: http://juju-ci.vapour.ws:8080/job/github-merge-juju-utils/155

hoenirvili · Nov 28, 2016

$$merge$$

jujubot · Nov 28, 2016

Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju-utils

hoenirvili referenced this pull request in juju/juju Nov 1, 2016
Merged
Enable manual provisioning for windows machines. #6523

hoenirvili referenced this pull request Nov 10, 2016
Merged
Move juju/cert pkg into utils. #250

jujubot added a commit that referenced this pull request Nov 10, 2016

Merge pull request #250 from hoenirvili/move-cert-pkg
Move juju/cert pkg into utils. As discussed in here #249 @axw @dooferlad

a84e60d

Update dependencies
Signed-off-by: Giulitti Salvatore <sgiulitti@cloudbasesolutions.com>

No user is associated with the committer email.

Learn about signing commits

5765b88

jujubot merged commit 7018575 into juju:master Nov 28, 2016

hoenirvili deleted the hoenirvili:enable-winrm-support branch Nov 28, 2016

hoenirvili referenced this pull request Dec 2, 2016
Merged
Adding ExtraExtensions overwirtes or makes dns field in template to disappear. #254

+        +	var err error
+        +	script, err = newEncodedPSScript(script)
+        +	if err != nil {
+        +		return "", errors.Annotatef(err, "Can't construct powershell command for remote execution")

@@ -77,3 +77,17 @@ func (s powershellSuite) TestMkdirAll(c *gc.C) {
          		`mkdir 'C:\some\dir'`,
          	})
          }
+        +
+        +func (s powershellSuite) TestNewPSEncodedCommand(c *gc.C) {

+        +	// will be used to make a new client conneciton to a endpoint
+        +	// TransportDecorator will enable us to switch transports
+        +	// this will be used for https client x509 authentication
+        +	winrm.DefaultParameters.TransportDecorator = func() winrm.Transporter {

+        +	// - no CA if there is not any and yes if there is
+        +	// - pass only client cert and key
+        +	// - 30 sec timeout
+        +	if ca := CACert(); ca != nil {

+        +		endpoint = winrm.NewEndpoint(host, httpsPort, true, true, nil, ClientCert(), ClientKey(), 30*time.Second)
+        +		logger.Warningf("Skipping CA server verification, using the --insecure option")
+        +	}
+        +	logger.Debugf("Creating WinRM secure clinet connection on %s %d", host, httpsPort)

juju/utils

This patch provides a friendly interface with the masterzen/winrm client #249

hoenirvili commented Nov 1, 2016

hoenirvili referenced this pull request in juju/juju Nov 1, 2016

Enable manual provisioning for windows machines. #6523

dooferlad commented Nov 9, 2016

axw commented Nov 10, 2016

gabriel-samfira commented Nov 10, 2016

axw commented Nov 10, 2016

gabriel-samfira commented Nov 10, 2016

hoenirvili commented Nov 10, 2016 •

Edited 1 time

hoenirvili
Nov 10, 2016

hoenirvili commented Nov 10, 2016

axw commented Nov 10, 2016

hoenirvili referenced this pull request Nov 10, 2016

Move juju/cert pkg into utils. #250

jujubot added a commit that referenced this pull request Nov 10, 2016

hoenirvili commented Nov 14, 2016

axw requested changes Nov 16, 2016

View changes

axw reviewed Nov 24, 2016

View changes

hoenirvili commented Nov 24, 2016

axw reviewed Nov 24, 2016

View changes

axw approved these changes Nov 27, 2016

View changes

hoenirvili commented Nov 28, 2016

jujubot commented Nov 28, 2016

jujubot commented Nov 28, 2016

hoenirvili commented Nov 28, 2016

jujubot commented Nov 28, 2016

jujubot commented Nov 28, 2016

hoenirvili commented Nov 28, 2016

jujubot commented Nov 28, 2016

jujubot merged commit `7018575` into juju:master Nov 28, 2016

hoenirvili deleted the hoenirvili:enable-winrm-support branch Nov 28, 2016

hoenirvili referenced this pull request Dec 2, 2016

Adding ExtraExtensions overwirtes or makes dns field in template to disappear. #254

jujubot added a commit that referenced this pull request Dec 2, 2016

juju/utils

Join GitHub today

This patch provides a friendly interface with the masterzen/winrm client #249

Conversation

hoenirvili commented Nov 1, 2016

hoenirvili referenced this pull request in juju/juju Nov 1, 2016

dooferlad commented Nov 9, 2016

axw commented Nov 10, 2016

gabriel-samfira commented Nov 10, 2016

axw commented Nov 10, 2016

gabriel-samfira commented Nov 10, 2016

hoenirvili commented Nov 10, 2016 • edited Edited 1 time hoenirvili Nov 10, 2016

hoenirvili commented Nov 10, 2016

axw commented Nov 10, 2016

hoenirvili referenced this pull request Nov 10, 2016

jujubot added a commit that referenced this pull request Nov 10, 2016

hoenirvili commented Nov 14, 2016

axw requested changes Nov 16, 2016 View changes

hoenirvili Nov 17, 2016 • edited Edited 1 time hoenirvili Nov 17, 2016

axw reviewed Nov 24, 2016 View changes

hoenirvili commented Nov 24, 2016

axw reviewed Nov 24, 2016 View changes

axw approved these changes Nov 27, 2016 View changes

hoenirvili commented Nov 28, 2016

jujubot commented Nov 28, 2016

jujubot commented Nov 28, 2016

hoenirvili commented Nov 28, 2016

jujubot commented Nov 28, 2016

jujubot commented Nov 28, 2016

hoenirvili commented Nov 28, 2016

jujubot commented Nov 28, 2016

jujubot merged commit 7018575 into juju:master Nov 28, 2016

hoenirvili deleted the hoenirvili:enable-winrm-support branch Nov 28, 2016

hoenirvili referenced this pull request Dec 2, 2016

jujubot added a commit that referenced this pull request Dec 2, 2016

hoenirvili commented Nov 10, 2016 •

Edited 1 time

hoenirvili
Nov 10, 2016

axw requested changes Nov 16, 2016

View changes

hoenirvili Nov 17, 2016 •

Edited 1 time

hoenirvili
Nov 17, 2016

axw reviewed Nov 24, 2016

View changes

axw reviewed Nov 24, 2016

View changes

axw approved these changes Nov 27, 2016

View changes

jujubot merged commit `7018575` into juju:master Nov 28, 2016