Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Upgrade SSH #31

Closed
wernight opened this Issue Jun 19, 2012 · 35 comments

Comments

Projects
None yet

Msysgit is using OpenSSH 4.6p1 while the current version of Cygwin is at least 5.3p1.

Issues with the old one include:

Owner

hvoigt commented Jun 19, 2012

I am all for it! @wernight do you want to work on this?

@dscho where does our OpenSSH come from? I do not find an openssh directory underneath src, I would suggest to create one there and add a release.sh for download, compilation and installing. Is that the correct path?

Member

ciaranj commented Jun 19, 2012

commit: 6fb732b looks like it could be of use (I'm also interested, but am tied up still on the instaweb stuff)

edit: ah but of course openssl != openssh ;) doh!

Member

ciaranj commented Jun 19, 2012

So, in an attempt to look (slightly) less than an idiot, I did a little research and came across this (year-old!) post from the mailing list that effectively details how one might set about such a thing:

https://groups.google.com/forum/#!msg/msysgit/hoFU1pO2GFE/aFl-dFRXzrUJ

Unfortunately (as far as I can tell) the currently available .lzma files depend on a later version of mingw (specifically msys-crypto-1.0.0.dll not msys-crypto-0.9.8.dll) so I'd guess we either have to compile from scratch against the version being shipped or update the version of mingw being used..which .. presumably is a PiTa :(

Owner

dscho commented Jun 19, 2012

@ciaranj that link is a good one! I think that it should be possible to make a /src/openssh/release.sh on the 'msys' branch with that information. However, given the fact that a new OpenSSH is not my itch and given the fact that I have many more itches I need and want to take care of these days, it will not be me who makes that script ;-)

Thank you ciaranj. I could help on providing a compatible rsync (what I'm already using now). What are the requirements for that "script"? Where is it to be executed, from what checkout, and what should it do/not do?

Owner

hvoigt commented Jun 20, 2012

@wernight as @dscho pointed out that script should live on the msys branch with the name /src/openssh/release.sh.

That script should download the sources, unpack them, apply local patches (also committed in that directory), compile the new openssh and install it in the msysgit environment. Have a look how /src/rt/release.sh is doing that for bash.exe and the msys.dll with our local patches.

Here is a description how to work on msys.dll itself:

https://github.com/msysgit/msysgit/wiki/WorkingOnMsysGit#wiki-Compiling_a_new_msys_dll

a similar approach should probably be taken to compile openssh using msys.dll.

Owner

dscho commented Jun 21, 2012

@wernight I'd be very interested in having rsync. Did you have a look at the /src/*/release.sh scripts? They're not at all difficult to understand and copy-edit. However, if at all possible, I would prefer rsync.exe to be a MinGW program instead of an MSys program (i.e. not linked to msys-1.0.dll). As rsync uses fork() on Linux, that might not be possible, though, since MSys provides a fork() emulation while MinGW does not.

The only rsync I got working among something like the 5 major ones, is one of cygwin. One version in particular. And, there is still a big issue with it, it's that it expects cygwin paths: /cygdrive/c/... which are impossible in Git. Still it's better than nothing. That mostly for that that I was looking for a newer SSH but I've no knowledge of Git's SSH requirements. From rsync, it seems very fragile and requiring the exact version it was compiled with.

Owner

dscho commented Jun 23, 2012

@wernight sorry, we do not just take binaries as-are as a rule. But did you actually look at any of these release.sh files? They are not at all hard to understand and copy-edit.

Member

ciaranj commented Jun 25, 2012

Surprisingly this: https://gist.github.com/2991314 seems to mostly work (in that it creates the binaries) but doesn't seem to play nicely with the existing repository (doesn't seem to spot that it is part of a Git repository). (OpenSSH)

Owner

dscho commented Jun 25, 2012

@ciaranj since you probably already have a clone of msysgit in which you tested this, why not make a branch and push it so it is much easier for me to test it and see what goes wrong? (I do not yet get what you mean by OpenSSH not realizing that it is part of a Git repository...)

Member

ciaranj commented Jun 25, 2012

@dscho sure, please see #35 .. I was really aiming the gist at @wernight though rather than hassle you :) ... I also think my msysgit install might be deeply b0rked I'm getting libconv2 errors everywhere (presumably related to the stalled work on instaweb)

Member

ciaranj commented Jun 26, 2012

@dscho ughhh just re-read your comment, you never wanted a pull request (thankfully, as it isn't suitable for pulling!) so I'll close that. The branch you asked for can be found here: https://github.com/ciaranj/msysgit/tree/msys_openssh_script

edit: And I've just re-tested on a separate 'clean' machine and discovered that my OpenSSH binaries depend on cygwin somehow .. I think I'll blat my msysgit installation and try again :)

Member

ciaranj commented Jun 26, 2012

So I have OpenSSH 5.9 compiled and working:

21:52:24 ciaranj@CNJ /tmp
>ssh -v
OpenSSH_5.9p1, OpenSSL 0.9.8r 8 Feb 2011

But it isn't any faster :( Also my approach may not be liked (it depends on A.N.Other port of OpenSSH), I'll throw it into a branch if anyone wants to proceed. ( https://github.com/ciaranj/msysgit/commits/msys_openssh_script please be aware that this is atop devel, not msys as I failed to get that to work!)

Owner

dscho commented Jul 5, 2012

Yeah, numeric IPs in download links do not make me very confident ;-)

Have you compared the patched vs the original OpenSSH? Maybe we can extract patches like we did e.g. with curl and have a setup that will make future updates a breeze?

Member

ciaranj commented Jul 10, 2012

@dscho I've extracted all the changes out as a single patch. But the resulting binaries seem screwed, to be more specific they seem to struggle with terminal inputs and outputs, I lack the capability to figure this stuff out, still hopeful that @wernight might have some ideas ;)

Owner

dscho commented Jul 12, 2012

@ciaranj any chance to have a branch where I can look at the single patch?

Member

ciaranj commented Jul 13, 2012

@dscho Hmmm, i thought I'd pushed it Up into the existing branch (https://github.com/ciaranj/msysgit/tree/msys_openssh_script ). Can you see the commit in there yet?

torgil commented Jan 4, 2013

Is anyone working on this issue?

I found a bug where the use of --upload-pack to git-fetch (and -u flag to git clone) gave the belowerror connecting to a ssh host (at least with tcsh shell). I think I'll await this to complete before I debug or file an issue.

Cloning into 'testrepo'...
Badly placed ()'s.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

aefimov commented Feb 8, 2013

Hello,
@ciaranj Is any troubles to send your branch msys_openssh_script as Pull Request here? It is really blocker for using Git on Windows over SSH.

Ok, I think I'm ready to start discussing this issue.

I have also run across the issue of msysgit's ssh traffic being limited to ~2 mbps on a gigabit network. I found a thread online that suggested it was the version of ssh packaged with msysgit that was the problem and it suggest using cygwin's ssh. I wasn't really into using cygwin's ssh so I grabbed the ssh files from the MinGW/MSYS project and dropped them into msysgit. The result was ~25-30 mbps transfers.

Originally I came to the msysgit project with the intent of providing (rather simple) instructions of how I did the above with the hope that that would be sufficient. This thread and its associated links lend me to beleave otherwise so I spent a bunch of time with the msysgit project attempting to compile openssh using the sources provided by the MSYS project.

My attempt has actually been successful and most of the problems encountered have been due to things being out of date or not available in msysgit's msys branch.

In particular I have the following issues I'd like better work arounds for or that prevent me from writing a script that can be run from the msys branch.

  1. No lzma or xz to extract sources downloaded from MinGW/MSYS project.
    1. My work around was to use a MinGW environment to extract these archives.
  2. Old gcc does not support -mtune and -fno-unti-at-a-time compiler flags.
    1. My work around was to remove them.
  3. msys-build script for minires (needed by openssh) wants lndir.
    1. My work around was to manually run make after their script configured things.
  4. openssl is not present in the msys branch.
    1. I just grabbed all the files from the master branch while in the msys branch.
    2. Ran into some issues at first as this exposed openssh to the mingw headers so I moved the openssl files out of the mingw directory.

There are a few more things that I had to do though those can be discussed once these are out of the way as I believe dealing with these should allow me to write some scripts similar to /src/rxvt/release.sh so I can share the work.

Owner

sschuberth commented Feb 14, 2013

@patrik-gornicz FYI, all the issues you had while trying to build OpenSSH are (to be) addressed by the mingwGitDevEnv project. Most notably, thanks to @t-b, we now have a current version of OpenSSH incl. HPN-SSH patches there. So maybe it would make sense for msysgit to just take what we already have in mingwGitDevEnv, or in the best case get more people involved in mingwGitDevEnv so that it can replace msysgit in the mid term.

@sschuberth a new dev environment sounds nice. As for the HPN patches, ya I ran into the same problems mentioned in a current version of OpenSSH though when I worked around them the resulting ssh seems to hang when I do a git clone. Couldn't figure out why, though didn't spend much time on it. One of the patches I had to tweak where with respect to calls to pthread_cleanup push/pop so it's possible my change is totally wrong or the pthread's provided by msys (at least in msysgit's msys branch) has an issue.

Yes please update OpenSSH! I have speed approximately 100Kb/s cloning repository through SSH, but using HTTPS approximately 700-900Kb/s

Owner

dscho commented Mar 20, 2013

@KindDragon any help is greatly appreciated. If you cannot contribute help to upgrade OpenSSH, I'll have to ask you to be patient until anybody who can help with the upgrade has time and motivation enough to do it.

Speedy37 commented Dec 8, 2013

I've simply copied msys-crypto-1.0.0.dll, msys-minires.dll, msys-z.dll, ssh.exe, ssh-add.exe, ssh-agent.exe, ssh-keygen.exe and ssh-keyscan.exe from latest msys git version (notice that msgit actually use msys-crypt-0.9.8.dll).
And I got a very important network boost (I was at 400Kb/s, now I got 1.6Mb/s and with cygwin git I got 1Mb/s)
So upgrade msys, I sick of patching every msysgit installation I made.
If you to test things fast, here are the binaries I use : https://files.speedy37.fr/public/GitSSH.zip

Owner

sschuberth commented Dec 8, 2013

And I'm sick of people who do not appreciate the work of others, but just use it without giving anything back, and being offensive on top of it. If you'd like to have OpenSSH updated, go ahead and do so as described in dscho's comment. Alternatively, see my comment about the mingwGitDevEnv project.

wernight commented Dec 8, 2013

@sebastian: I'm sure you guys have plenty to do, so does most of us. Speedy
is also doing work for others in that he found a way to use and tested
upgraded binaries. I'm sure what is missing is more the difficulty to
patch the Git project, and that part should be the trivial on GitHub to
have good pull requests.

On Sun Dec 08 2013 at 2:07:00 PM, Sebastian Schuberth <
notifications@github.com> wrote:

And I'm sick of people who do not appreciate the work of others, but just
use it without giving anything back, and being offensive on top of it. If
you'd like to have OpenSSH updated, go ahead and do so as described in dscho's
commenthttps://github.com/msysgit/msysgit/issues/31#issuecomment-6440174.
Alternatively, see my comment about the mingwGitDevEnv projecthttps://github.com/msysgit/msysgit/issues/31#issuecomment-13537020
.


Reply to this email directly or view it on GitHubhttps://github.com/msysgit/msysgit/issues/31#issuecomment-30081155
.

Speedy37 commented Dec 8, 2013

I appreciate the work you people did. I just don't understand why this project hasn't updated msys since about 3 years. If you open the zip, you can see I use this patch since 2011...
I'll for sure try the mingwGitDevEnv project. But a lot of people simply download msysgit and found the ssh performance very bad.
I haven't look yet at how to make a git pull request for updating msys. But I think it should not take very long for the msysgit team to update msys as they know their own dev methods.
An everyday msysgit user.

Owner

dscho commented Dec 30, 2013

@wernight @Speedy37 I understand your points of view. And I disagree.

Unfortunately it looks like all the people who would actually be capable of doing anything about this issue disagree, too, so realistically, I will have to close this issue.

@dscho dscho closed this Dec 30, 2013

@Speedy37 It works. Many thanks!
btw: I try mingwGitDevEnv in ubuntu, but failed to build with many errors,it is not so easy to try(if I have time I will try again).

Contributor

t-b commented Jan 15, 2014

@zhangchunlin mingwGitDevEnv and "Git for windows" are supposed to be run on windows. They might work with wine on linux but this is not the intended usage.

Owner

dscho commented Jan 15, 2014

mingwGitDevEnv in Ubuntu? LOL... ;-)

@t-b @dscho , I get it now. I saw "wine=$(type -p wine)", so I thought it should be under linux to run it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment