SSH/SFTP support for Windows

[cawale]

Currently the SFTP protocol is not supported if the end point type is set to windows. The Windows command set is not used when doing Status checks or deploys.

Please add this functionality to Deployhub.

Thanks.

[pgibbs-openmake]

Scheduling this for next release. Need an equivalent of mkdir -p for windows. One possible solution would be to send a sequence of mkdir commands, ignoring error returns (mkdir /tmp;mkdir /tmp/tgt;mkdir /tmp/tgt/finaldir) instead which would be identical for Windows and Linux but this could be problematic with deep directory nesting. Perhaps issue them as separate command lines?

The biggest issue is that the underlying protocol is not passed the target operating system so we would struggle to switch logic depending on whether the target is Windows or a flavour of Linux/Unix.

[cawale]

If it helps this command string will do the same thing in windows.

"cmd /c d: && mkdir \tmp && mkdir \tmp\tgt && mkdir \tmp\tgt\finaldir"

[sbtaylor15]

mkdir on windows automatically creates all of the intermediate directories.

For example: mkdir d:\temp\a\b\c\d

No -p is needed

[pgibbs-openmake]

That's true but as discussed above the underlying script does not know the target operating system. So we either have to pass that through somehow or come up with a strategy that works for any target OS.

[sbtaylor15]

Do you have the fix in place for enable sftp for a windows machine?

Also, will "mkdir /tmp;mkdir /tmp/tgt;mkdir /tmp/tgt/finaldir" work creating the directory structure when you ssh into the windows box?

[cawale]

No I have do not have a fix right now for the SFTP for windows. From the command line I can do it but not from DeployHub.

I believe the correct syntax would be something like "cmd /c d: && mkdir -f \tmp\tgt\finaldir"

[sbtaylor15]

What is the ssh server that you are using on Windows, cygwin?

[cawale]

We are using FreeSSHd.

[cawale]

Hi all, Any eta on this functionality? Thanks. Chris From: Steve Taylor [mailto:notifications@github.com] Sent: February 20, 2018 11:31 AM To: OpenMake-Software/DeployHub Cc: Wale, Chris (MOHLTC); Author Subject: Re: [OpenMake-Software/DeployHub] SSH/SFTP support for Windows (#16) What is the ssh server that you are using on Windows, cygwin? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#16 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AgloGTJAsQd_-aCplVoGlUq-9EIj5bU5ks5tWvM-gaJpZM4Qyz92>.

[sbtaylor15]

Can you run the following from the Solaris server against the Windows SSH server?

ssh uname

Let me know the output.

[cawale]

This what you are looking for? bash-3.2$ ssh -l WaleC 10.77.9.108 WaleC@10.77.9.108's password: From: Steve Taylor [mailto:notifications@github.com] Sent: March 9, 2018 1:29 PM To: OpenMake-Software/DeployHub Cc: Wale, Chris (MOHLTC); Author Subject: [Possible SPAM] Re: [OpenMake-Software/DeployHub] SSH/SFTP support for Windows (#16) Can you run the following from the Solaris server against the Windows SSH server? ssh uname Let me know the output. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#16 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AgloGesNzMDtnbYlVOGICAQne6fgG8t9ks5tcsnzgaJpZM4Qyz92>.

[sbtaylor15]

Sorry github stripped out the command:

ssh -l WaleC 10.77.9.108 uname

Just tack on uname as the command to run. I am guessing that the output will be something about the program uname not being found. Just need to double check the real output.

Thanks Steve

[cawale]

Hi Steve, I was able to run the command you suggested below. Here is what was returned. bash-3.2$ ssh -l WaleC 10.77.9.108 uname WaleC@10.77.9.108's password: Unable to execute command or shell on remote system: Failed to Execute process. Chris From: Steve Taylor [mailto:notifications@github.com] Sent: March 9, 2018 1:39 PM To: OpenMake-Software/DeployHub Cc: Wale, Chris (MOHLTC); Author Subject: [Possible SPAM] Re: [OpenMake-Software/DeployHub] SSH/SFTP support for Windows (#16) Sorry github stripped out the command: ssh -l WaleC 10.77.9.108 uname Just tack on uname as the command to run. I am guessing that the output will be something about the program uname not being found. Just need to double check the real output. Thanks Steve — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#16 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AgloGYRruHbiil2ukP8gWPOR1GXIAD69ks5tcswogaJpZM4Qyz92>.

[sbtaylor15]

In the process of updating sshexec to check the remote end point OS by running uname. If we get back 'Unable to execute command' then we can assume that its windows.

@cawale can you try another command which may make it a bit cleaner:

ssh -l WaleC 10.77.9.108 set

Send me the output from the command above.

[cawale]

The result is this

bash-3.2$ ssh -l WaleC 10.77.9.108 set
WaleC@10.77.9.108's password:
Unable to execute command or shell on remote system: Failed to Execute process.

[sbtaylor15]

Changed the logic to check the remote OS and the run a modified command for the OS specific syntax.
sshexec.zip

Backup the existing engine/lib/sshexec and put the new one in place.

Let me know the results.

Thanks, Steve

[sbtaylor15]

I forgot to comment out line 2 of the script. Here is the updated one.

sshexec.zip

[cawale]

Seeing this error. Maybe my os does not have something you need?

bash-3.2$ ./sshexec -m 10.77.9.108 -port 22 -usr yyyy -pw xxxxxxxx "mkdir -p /export/home/walec/deploy;cd /export/home/walec/deploy;pwd"
./sshexec: syntax error at line 12: `HOME=$' unexpected

[cawale]

bash-3.2$ finger whoami | awk '/^Directory/ {print $2}'
bash: whoami: command not found

[cawale]

found it in /usr/usb

[cawale]

so this

finger /usr/ucb/whoami | awk '/^Directory/ {print $2}'

returns

/opt/deployhub

but the sshexec still fails with

./sshexec: syntax error at line 12: `HOME=$' unexpected

[cawale]

Your #!/bin/sh needs to be #!/bin/ksh on my system for this script to work.

[cawale]

ok. So I am getting farther. Now I get.

bash-3.2$ ./sshexec -m 10.77.9.108 -port 22 -usr xxxxxx -pw xxxxx "mkdir -p /export/home/walec/deploy;cd /export/home/walec/deploy;pwd"
/opt/deployhub
grep: illegal option -- q
Usage: grep -hblcnsviw pattern file . . .
./sshexec[139]: sshpass: not found
./sshexec[189]: sshpass: not found

[sbtaylor15]

You will need to install the sshpass package. This package enables ssh to be run without a password prompt. See: https://www.opencsw.org/package/sshpass/

[cawale]

ok so the grep that you needed was not on my path it was here /usr/xpg4/bin/

[cawale]

sshpass was already there just not where it was expecting it.

[cawale]

Where do I find rev?

[sbtaylor15]

Don't look like Solaris has the rev command. Let me rework the that line.

[cawale]

I found a script that does the same thing. I created a rev script.

[cawale]

ok. So I got the script working from the command line with some changes. I can send it to you.

It does not yet work with the UI. I have just tested it with the status check at this point. I have not tried an install.

There are a few caveats though. The server MUST be specified as a unix server. If not then the '' in the UI causes problems. After you set the server to unix then you need to use single quotes around the 'base directory' for the server if you have a leading / in it or else the path will not get saved or used.

Also, I didn't need the rev routine after all.

[sbtaylor15]

Where you able to get the mkdir of directories and other commands to run on the Windows machine from the command line (not via the UI) invocation of the sshexec script?

[cawale]

yup

[cawale]

How do I turn on more debug logging? I cannot see anything worthwhile that the system is doing when it is doing the Status call. I don't see the request being send to sshexec or any thing that it returns.

[cawale]

ok. so I found the debug.out log. From that file I see this when I run the Status function in the UI.

protocol_cd('/temp/deploy')
myDir=['/temp/deploy']
cmd=[sshexec -m "10.77.9.108" -port "22" -usr "WaleC" -pw 'Jan202018' "mkdir -p "'/temp/deploy'";cd "'/temp/deploy'";pwd"]
line=[Unable to execute command or shell on remote system: Failed to Execute process.
]

I think this is the first uname failure not the failure of the hole thing. I am going to look and see if I can get it not to print out the uname failure.

[cawale]

So it looks like the UI or something is doing a uname after the initial request??? I am seeing this in an sshexec log I created.

sshexec -m 10.77.9.108 -port 22 -usr xxxxx -pw xxxx mkdir -p /temp/deploy;cd /temp/deploy;pwd
WINDOWS 1: mkdir -p /temp/deploy
WINDOWS 1: cd /temp/deploy
WINDOWS 1: pwd
WINDOWS 2: /c (if not exist \temp\deploy mkdir \temp\deploy ) && cd \temp\deploy && cd
sshexec -force -m 10.77.9.108 -port 22 -syn -usr xxxxx -pw xxxxxx uname
WINDOWS 1: uname
WINDOWS 2: /c uname

[cawale]

So I been able to get a successful dmtransfer working but still not through the UI. It looks like the UI try's to be smart and converts '\' to '/' and then the match at the end fails even though the sshexec works as expected and returns the right directory.

I have been able to set the server up as a windows server and set the base directory to C:\temp\deploy. This is what gets sent in to sshexec though. (from debug.out)

protocol_cd(C:\temp\deploy)
myDir=[C:/temp/deploy]
cmd=[sshexec -m "10.77.9.108" -port "22" -usr "xxxx" -pw 'xxxxx' "mkdir -p "C:/temp/deploy";cd "C:/temp/deploy";pwd"]

If I run it from dmtransfer, from the command line, then I can see that the correct response if coming back from windows.

protocol_cd(C:\temp\deploy)
myDir=[C:/temp/deploy]
cmd=[sshexec -m "10.77.9.108" -port "22" -usr "xxxx" -pw 'xxxxx' "mkdir -p "C:/temp/deploy";cd "C:/temp/deploy";pwd"]
line=[C:\temp\deploy
]

I think maybe this is something you need to fix in code.

[cawale]

Just to confirm to you the dmtransfer from the command line looks like this.

bash-3.2$ ./dmtransfer "-protocol" "sftp" "-server" "10.77.9.108" "-user" "xxx" "-password" "xxxx" "-checkbase" "C:\temp\deploy"
Base directory "C:\temp\deploy" is valid

[cawale]

Now trying to remove the paths to whoami and grep that I needed to get this working.

I have tried updating the path in my .profile file as well as in the rectl file, logged out and back in again and restarted Deployhub but when I do a Status request for the server it is not finding either.

Where do I need to set the path to include the new required directories so that the UI is setup correctly to use the sshexec script?

I can run dmtransfer and sshexec from the command line and all works as expected.

[sbtaylor15]

It should be under the omreleng userid. Update that .profile. Or you can update the global profile in /etc/profile (I believe)

[cawale]

I have just hardcoded the paths. I cannot seem to find the right place for it to work.

[sbtaylor15]

You can try putting the scripts in the /engine/bin directory. That should be in the path.

[cawale]

Is there a reason that the UI would put extra double quotes around paths?

sshexec -m "10.77.9.108" -port "22" -usr "xxxx" -pw 'xxxxx' "mkdir -p "C:/temp/deploy";cd "C:/temp/deploy";pwd"

I don't specify them in the base directory path???? I think this is what is messing up the UI call to sshexec.

[sbtaylor15]

Yes, I agree that UI is messing up the command. I need to change the UI so you can specify Windows as the OS and sftp as the protocol making sure the slashes and quoting are correct. Let me work on that.

[cawale]

ok. Thanks.

[cawale]

You may want to hold off on that. I just ran the sshexec command from the command line. The same one with all the extra double quotes and it worked fine. The script dealt with the extra quotes.

So now I don't really have any idea why the UI does not see this call as a successful one???

Unless it is having issues dealing with the '\' characters in the response.

[cawale]

One thing I have noticed is that when I run the dmtransfer command from the command line, with the same syntax as the UI does, I see an extra debug line in the debug.out file.

protocol_cd(C:\temp\deploy)
myDir=[C:/temp/deploy]
cmd=[sshexec -m "10.77.9.108" -port "22" -usr "xxx" -pw 'xxxx' "mkdir -p "C:/temp/deploy";cd "C:/temp/deploy";pwd"]
line=[C:\temp\deploy
]

I do not see this when it gets initiated from the UI.

[cawale]

Now I am getting into some weird results. Wondering if the ssh connections are having issues. None of my sshpass commands come back with anything. They complete but the result is "".

That is through the UI only. If I run them from the command line in either ksh or bash everything works fine?

[sbtaylor15]

Is the password being passed in from the UI? Also, I don't think you can run sshpass by itself.

[cawale]

Yes the password is being sent in. I am able to run sshpass on it's own.

[cawale]

I think my problem is with the ssh daemon. I have 3 sshd tasks running and I think I should only have one.

We have seen this issue before where sshpass stops working for a while and then miraculously starts again at some point. Maybe a bug in Solaris' sshd and maybe some issue with sshpass where it is not catching an error correctly?

[cawale]

So the issue ended up being a bug I introduced in the script which I resolved. This does not seem to fix the UI integration issue but all of the commands that the UI is trying to do work from the command line. both the sshexec and dmtransfer commands.

It also still works for unix endpoints fine from the UI.

When I look at the debug.out log the only difference I see from the successful unix requests and the windows ones are that there are no 'line [ ]' debug line after the 'cmd' debug line.

I don't know what this means but I think I have taken it as far as I can. Let me know how you want me to send the updated script file and if there is anything else I can do to help.

[cawale]

Is there any ETA on looking at the code for this?

[sbtaylor15]

Can you send me a screen shot of the General Tab for the Windows End Point that is using ssh for the protocol? I want to make sure that my test case matches up with your definition.

Thanks, Steve

[cawale]

Here you go.

[sbtaylor15]

Windows will be supported using WinRM via Ansible.