Windows wrapper #10

Closed
wants to merge 3 commits into
from

Conversation

Projects
None yet
4 participants
@interfect
Contributor

interfect commented Sep 1, 2016

On top of #9, adds an .exe file that just runs the shell script in the same directory under Git Bash. Needed because Git Annex can't invoke the shell script as an executable on Windows.

I'm using this and it appears to be working for me, against an Amazxon Cloud Drive rclone remote.

interfect added some commits Sep 1, 2016

Strip trailing whitespace from commands
On Windows, I was having a problem with Git Bash's case builtin not
matching "INITREMOTE\r\n" against "INITREMOTE". This should fix that by
stripping off CRLFs (and all whitespace) at the end of a command.
Add a Windows wrapper
It's written in C# and basically makes it so that when git annex tries
to createProcess on "git-annex-remote-rclone" it works.
Actually fix Windows support
The script now properly runs on Windows, through a wrapper I will PR
separately.

To make this work, I've had to not strip backslashes from the input (by
adding -r to the read calls). They are being specified as path
separators, and stripping them screws up paths.

I also swapped out the right-trim for a more specific
remove-trailing-CR-if-present, which is now uniformly applied to both
instances of read.

I also left in my check for trying to upload a nonexistent file (which
happened when backslashes were stripped). It *should* never happen, but
it's definitely an error if it does.
@DanielDent

This comment has been minimized.

Show comment
Hide comment
@DanielDent

DanielDent Sep 1, 2016

Owner

Thank you for this as well as your other pull request. I'd like to merge Windows support. I'm not an active Windows user these days, so it's a little harder for me to support properly.

In terms of handling builds/releases:

  1. If it were easy do this in a way that allows cross-compilation (ala mingw-w64) that might be one way to simplify builds.
  2. Another approach is that I could provide a Windows instance with a Gitlab CI runner. This could be used for both building and testing (This project is also hosted at https://gitlab.com/DanielDent/git-annex-remote-rclone, and I have a Linux CI runner hooked up to the project which is triggered automatically. This helps catch issues with new releases of dependencies). If you look at the .gitlab-ci.yml file, you'll see a very simple test script which is run before a release is made. Running a similar test suite on Windows would be nice.
  3. I'm open to Appveyor or any other approach to Windows builds.

In terms of the code itself:

  1. I don't know what the purpose of the contrib/GitAnnexRemoteRcloneWrapper/.vs/GitAnnexRemoteRcloneWrapper/v14/.suo binary is
  2. I don't think it's safe to assume Program Files is on C:. I know Windows has an environment variable (http://stackoverflow.com/questions/17688758/using-programfilesx86-on-windows-os-32bit, http://stackoverflow.com/questions/9594066/how-to-get-program-files-x86-env-variable), but I have not done enough Windows development recently to know the 'correct' way of figuring out the path of another program. Perhaps there's a registry entry which should be consulted?

In any event, I'm pretty sure that simply assuming that it's only a 32-bit machine and program files is on C: is incorrect.

https://git-annex.branchable.com/install/Windows/ talks about the need for using 32-bit binaries, so that part of things may be correct. It might also be appropriate to check both the location where a 64 bit binary would be expected as well as where a 32 bit binary would be expected.

Owner

DanielDent commented Sep 1, 2016

Thank you for this as well as your other pull request. I'd like to merge Windows support. I'm not an active Windows user these days, so it's a little harder for me to support properly.

In terms of handling builds/releases:

  1. If it were easy do this in a way that allows cross-compilation (ala mingw-w64) that might be one way to simplify builds.
  2. Another approach is that I could provide a Windows instance with a Gitlab CI runner. This could be used for both building and testing (This project is also hosted at https://gitlab.com/DanielDent/git-annex-remote-rclone, and I have a Linux CI runner hooked up to the project which is triggered automatically. This helps catch issues with new releases of dependencies). If you look at the .gitlab-ci.yml file, you'll see a very simple test script which is run before a release is made. Running a similar test suite on Windows would be nice.
  3. I'm open to Appveyor or any other approach to Windows builds.

In terms of the code itself:

  1. I don't know what the purpose of the contrib/GitAnnexRemoteRcloneWrapper/.vs/GitAnnexRemoteRcloneWrapper/v14/.suo binary is
  2. I don't think it's safe to assume Program Files is on C:. I know Windows has an environment variable (http://stackoverflow.com/questions/17688758/using-programfilesx86-on-windows-os-32bit, http://stackoverflow.com/questions/9594066/how-to-get-program-files-x86-env-variable), but I have not done enough Windows development recently to know the 'correct' way of figuring out the path of another program. Perhaps there's a registry entry which should be consulted?

In any event, I'm pretty sure that simply assuming that it's only a 32-bit machine and program files is on C: is incorrect.

https://git-annex.branchable.com/install/Windows/ talks about the need for using 32-bit binaries, so that part of things may be correct. It might also be appropriate to check both the location where a 64 bit binary would be expected as well as where a 32 bit binary would be expected.

@DanielDent

This comment has been minimized.

Show comment
Hide comment
@DanielDent

DanielDent Sep 1, 2016

Owner

@joeyh It just occurred to me that rather than have git-annex launch a wrapper for bash which launches this wrapper for rclone, perhaps when git-annex launches a remote on Windows it could pay attention to the shebang and launch this remote using bash without the use of the wrapper contributed by @interfect . But perhaps that adds unnecessary work for you or is a bad idea for some other reason I haven't considered.

Owner

DanielDent commented Sep 1, 2016

@joeyh It just occurred to me that rather than have git-annex launch a wrapper for bash which launches this wrapper for rclone, perhaps when git-annex launches a remote on Windows it could pay attention to the shebang and launch this remote using bash without the use of the wrapper contributed by @interfect . But perhaps that adds unnecessary work for you or is a bad idea for some other reason I haven't considered.

@interfect

This comment has been minimized.

Show comment
Hide comment
@interfect

interfect Sep 1, 2016

Contributor

Getting Git Annex itself to try and interpret shebangs might sort of solve
things; you could just set your shebang on Windows to point to where your
Windows build of Bash lives. On the other hand, that's really not supposed
to be Git Annex's job, and it would have to manually go wandering all over
your PATH looking for things.

It does make sense to check the Windows environment variables, and maybe
try for both 32 and 64-bit builds of Bash. I have it pointing at the 32-bit
bash right now because the Git Annex Windows installer depends specifically
on 32-bit Git for Windows, even if you have a 64-bit machine.

Those binary files can probably be dropped from the PR. I will see about
that.

This EXE is actually done in C#, which runs on .NET, so one should be
able to build and run it with Mono (maybe through MonoDevelop). As is the
hardcoded C:\ paths aren't going to do anything in a Unix environment, but
if it consults the environment variables or has "/bin/bash" on its list of
paths to search for bash in, it ought to be able to build and run and do
the exact same thing under Mono on Linux. I will see about setting that up.

On Thu, Sep 1, 2016 at 7:34 AM, Daniel Dent notifications@github.com
wrote:

@joeyh https://github.com/joeyh It just occurred to me that rather than
have git-annex launch a wrapper for bash which launches this wrapper for
rclone, perhaps when git-annex launches a remote on Windows it could pay
attention to the shebang and launch this remote using bash without the use
of the wrapper contributed by @interfect https://github.com/interfect .
But perhaps that adds unnecessary work for you or is a bad idea for some
other reason I haven't considered.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#10 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAt5tk8cOn-Hc7romRuZAU2gVOAsaygJks5qluJogaJpZM4JyWYD
.

Contributor

interfect commented Sep 1, 2016

Getting Git Annex itself to try and interpret shebangs might sort of solve
things; you could just set your shebang on Windows to point to where your
Windows build of Bash lives. On the other hand, that's really not supposed
to be Git Annex's job, and it would have to manually go wandering all over
your PATH looking for things.

It does make sense to check the Windows environment variables, and maybe
try for both 32 and 64-bit builds of Bash. I have it pointing at the 32-bit
bash right now because the Git Annex Windows installer depends specifically
on 32-bit Git for Windows, even if you have a 64-bit machine.

Those binary files can probably be dropped from the PR. I will see about
that.

This EXE is actually done in C#, which runs on .NET, so one should be
able to build and run it with Mono (maybe through MonoDevelop). As is the
hardcoded C:\ paths aren't going to do anything in a Unix environment, but
if it consults the environment variables or has "/bin/bash" on its list of
paths to search for bash in, it ought to be able to build and run and do
the exact same thing under Mono on Linux. I will see about setting that up.

On Thu, Sep 1, 2016 at 7:34 AM, Daniel Dent notifications@github.com
wrote:

@joeyh https://github.com/joeyh It just occurred to me that rather than
have git-annex launch a wrapper for bash which launches this wrapper for
rclone, perhaps when git-annex launches a remote on Windows it could pay
attention to the shebang and launch this remote using bash without the use
of the wrapper contributed by @interfect https://github.com/interfect .
But perhaps that adds unnecessary work for you or is a bad idea for some
other reason I haven't considered.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#10 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAt5tk8cOn-Hc7romRuZAU2gVOAsaygJks5qluJogaJpZM4JyWYD
.

@joeyh

This comment has been minimized.

Show comment
Hide comment
@joeyh

joeyh Sep 1, 2016

Oh, git-annex already deals with this particular windows nonsense elsewhere. When it needs to run a git hook, it parses it for a shebang. Git for windows does the same.

So, if you can please open a todo item in git-annex, I can refactor that existing code to be used in more places.

joeyh commented Sep 1, 2016

Oh, git-annex already deals with this particular windows nonsense elsewhere. When it needs to run a git hook, it parses it for a shebang. Git for windows does the same.

So, if you can please open a todo item in git-annex, I can refactor that existing code to be used in more places.

@DanielDent

This comment has been minimized.

Show comment
Hide comment
@DanielDent

DanielDent Sep 1, 2016

Owner

@joeyh As usual, you are awesome :). I've opened http://git-annex.branchable.com/todo/refactor_shebang_handling_code_for_wider_use/

@interfect Joey's proposal seems like the most elegant approach here. If for some reason that doesn't work out, your Mono suggestion seems sensible

Owner

DanielDent commented Sep 1, 2016

@joeyh As usual, you are awesome :). I've opened http://git-annex.branchable.com/todo/refactor_shebang_handling_code_for_wider_use/

@interfect Joey's proposal seems like the most elegant approach here. If for some reason that doesn't work out, your Mono suggestion seems sensible

@interfect

This comment has been minimized.

Show comment
Hide comment
@interfect

interfect Sep 4, 2016

Contributor

I'm going to close this, then, and wait on that change to git-annex itself.

Contributor

interfect commented Sep 4, 2016

I'm going to close this, then, and wait on that change to git-annex itself.

@interfect interfect closed this Sep 4, 2016

@joeyh

This comment has been minimized.

Show comment
Hide comment
@joeyh

joeyh Sep 5, 2016

Done, please test the git-annex change..

joeyh commented Sep 5, 2016

Done, please test the git-annex change..

@PythonNut

This comment has been minimized.

Show comment
Hide comment
@PythonNut

PythonNut Dec 26, 2016

Hm... so I gave this a shot with the most recent version of git-annex, rclone, and git-annex-remote-rclone.

I moved both rclone and git-annex-remote-rclone into the /usr/bin folder on the Git for Windows build, and ensured that they both were executable.

However, trying to enable the remote still yields:

git-annex: Cannot run git-annex-remote-rclone -- Make sure it's in your PATH and is executable.

Is there anything obvious that I may be doing wrong?

Hm... so I gave this a shot with the most recent version of git-annex, rclone, and git-annex-remote-rclone.

I moved both rclone and git-annex-remote-rclone into the /usr/bin folder on the Git for Windows build, and ensured that they both were executable.

However, trying to enable the remote still yields:

git-annex: Cannot run git-annex-remote-rclone -- Make sure it's in your PATH and is executable.

Is there anything obvious that I may be doing wrong?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment