Native Windows port

[ThomasWaldmann]

If borg could run directly on a normal Windows installation (via cmd, not requiring cygwin or other major runtime support), it could be useful for windows machines that do not have cygwin installed.

Notes:

Doing a native windows port is likely quite some work over a longer time, due to all the differences between windows and UNIX-like operating systems. Whether it can be done in a pretty (regarding to source code changes) and good working (regarding to practical use) way has to be seen.

Because of that, it will have to live in a separate branch of the repository at first (which should be regularly updated from master branch, so it is kept up-to-date and without potential merge conflicts).

At some time in the future, that branch might or might not get merged into master.

---

💰 there is a bounty for this

I opened a (first) bounty for this - if you would like to see this happen, feel free to support this or a future, related bounty.

The goal and scope of this bounty is:

• have a borg codebase that is able to run on native windows (without cygwin or similar runtime requirements)
• working basic borg commands: init, create, list, extract, delete
• working unit tests (failures in some of them that test stuff not in scope of the bounty are acceptable)
• initial documentation for developers and users (how to set up dev environment, how to install on a user system)
• not required: encryption, special windows file attributes support (like ACLs, xattrs), VSS support, installer

[ThomasWaldmann]

See PR #947 (has been merged into "windows" branch now).

Superceded by: #986 (has been merged into "windows" branch now).

[ThomasWaldmann]

Thanks to @Anakonda for doing the native windows port!

The basics work, but it will for sure need more testing, fixes and enhancements - please file tickets for anything that shows up (make sure you mention "windows branch" for problems only happening in that branch).

[henfri]

Hello,

I would like to try borg under Windows.
I am aware of the standalone installer using cygwin. But for the project, I think testing of the native windows port would be useful, don't you think?

Is there any documentation on the native port? E.g. what python packages are needed.
For another project I use cx_freeze to freeze to create an executable. That would certainly help on this respect. What do you think?

Regards,
Hendrik

[henfri]

Hello,

I just saw that an executable is already created here:
https://github.com/borgbackup/borg/blob/windows/deployment/windows/buildwin32.py

That's good. Before deploying borg as native executable, an installer would be even better, but for testing, it is not needed I think.

If you would like people to test the native windows, I think that a compiled version of this branch would be needed. Is that available from somewhere?

Regards,
Hendrik

[Anakonda]

@henfri latest version is available here https://ci.appveyor.com/project/borgbackup/borg/build/artifacts

[Neurrone]

@Anakonda is this a native windows port, or does it install sigwin?

[henfri]

Cygwin, you mean?
By the looks it is the native built and does not need cygwin

Greetings,
Hendrik

[Neurrone]

@henfri thanks. Will try it, curious to see how it stacks up against rclone and duplicati

[henfri]

Hello,

no, it does not run. It is missing ctypes.wintypes:

from .windows import acl_get, acl_set
File "src\borg\platform\windows.pyx", line 13, in init borg.platform.windows (src/borg/platform\windows.c:8460)
ImportError: No module named 'ctypes.wintypes'

Greetings,
Hendrik

[Anakonda]

Here is working one: https://www.dropbox.com/s/tolumpufc179j77/borg.zip?dl=0
I will do pull request later.

[henfri]

Thanks.
That runs.

But I get an error message doing

borg list "borg@openmediavault:/srv/_dev_disk_by-id_ata-ST5000DM000/borgbackup
Using a pure-python msgpack! This will result in lower performance.
Could not find supported ssh client. Supported clients are ssh and plink.

So, I suppose that I need "ssh" installed somewhere?
But 'what' ssh and where?

Greetings,
Hendrik

[Anakonda]

You are trying to backup to a server? Where did you get that address?
Borg uses protocol called ssh to communicate with server so you need ssh client. Windows version support openssh (ssh) and putty (plink)

[henfri]

Hello,

yes. The adress is the one that I use for backing up my linux machine.

I will try plink.
Before I ask more questions: Is there a documentation for the specifics of the windows version?

Greetings,
Hendrik

[henfri]

Hello,

I tried plink now.

AssertionError: cleanup happened in Repository.del
Local Exception.
Traceback (most recent call last):
File "C:\Users\Hendrik Friedel\Downloads\borg1\bin\borg\archiver.py", line 2287, in main
exit_code = archiver.run(args)
File "C:\Users\Hendrik Friedel\Downloads\borg1\bin\borg\archiver.py", line 2220, in run
return args.func(args)
File "C:\Users\Hendrik Friedel\Downloads\borg1\bin\borg\archiver.py", line 83, in wrapper
repository = RemoteRepository(location, create=create, lock_wait=self.lock_wait, lock=lock, args=args)
File "C:\Users\Hendrik Friedel\Downloads\borg1\bin\borg\remote.py", line 184, in init
version = self.call('negotiate', RPC_PROTOCOL_VERSION)
File "C:\Users\Hendrik Friedel\Downloads\borg1\bin\borg\remote.py", line 272, in call
for resp in self.call_many(cmd, [args], **kw):
File "C:\Users\Hendrik Friedel\Downloads\borg1\bin\borg\remote.py", line 324, in call_many
raise Exception('FD exception occurred')
Exception: FD exception occurred

Platform: Windows LaptopHendrik 10 10.0.14393 AMD64 Intel64 Family 6 Model 69 Stepping 1, GenuineIntel
Borg: 1.1.dev0-201-g7b37b16d Python: CPython 3.5.3
PID: 16908 CWD: C:\Users\Hendrik Friedel\Downloads\borg1
sys.argv: ['borg', 'list', 'borg@openmediavault:/srv/_dev_disk_by-id_ata-ST5000DM000/borgbackup']
SSH_ORIGINAL_COMMAND: None

I can also try openssl. Can you point me at a download link? There seem to be several ports.

Greetings,
Hendrik

[Anakonda]

There is bug in windows version, You need to authenticate via ssh key. See some guides how to set that up.
As far as I know only openssh builds for windows are for cygwin and msys.

[henfri]

Hello,

I did my first backup from Windows now.
Some notes:
Make sure you can connect from putty to your server using public-key. I used a dedicated user ("borg") on the server. I created the key with puttygen.exe

In putty go to settings->connection->SSH->auth --> private key for authorization and select the private key that was created by puttygen.
Add the public key to /home/borg/.ssh/authorized_keys

Test the connection from putty. No password should be needed.
Now test the connection with plink:
plink borg@server ls
should give a directory listing from the server.

If this runs, you are set.

[henfri]

Hello,

would you be willing also to provide an 32bit executable?

Regards,
Hendrik

[henfri]

... and with the latest Version, please as I am also suffering from this:
#3192

[henfri]

Hello,

I have used this version of Borg for a while now. I had in the past not seen errors (but I was running it in Background). But now I continiously get:

FD exception occurred

Local Exception.
Traceback (most recent call last):
  File "C:\Program Files\borg\bin\borg\archiver.py", line 95, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "C:\Program Files\borg\bin\borg\archiver.py", line 311, in do_create
    create_inner(archive, cache)
  File "C:\Program Files\borg\bin\borg\archiver.py", line 287, in create_inner
    archive.save(comment=args.comment, timestamp=args.timestamp)
  File "C:\Program Files\borg\bin\borg\archive.py", line 378, in save
    self.cache.add_chunk(self.id, Chunk(data), self.stats)
  File "C:\Program Files\borg\bin\borg\cache.py", line 386, in add_chunk
    self.repository.put(id, data, wait=False)
  File "C:\Program Files\borg\bin\borg\remote.py", line 405, in put
    return self.call('put', id_, data, wait=wait)
  File "C:\Program Files\borg\bin\borg\remote.py", line 272, in call
    for resp in self.call_many(cmd, [args], **kw):
  File "C:\Program Files\borg\bin\borg\remote.py", line 324, in call_many
    raise Exception('FD exception occurred')
Exception: FD exception occurred

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\borg\bin\borg\archiver.py", line 2287, in main
    exit_code = archiver.run(args)
  File "C:\Program Files\borg\bin\borg\archiver.py", line 2220, in run
    return args.func(args)
  File "C:\Program Files\borg\bin\borg\archiver.py", line 95, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "C:\Program Files\borg\bin\borg\remote.py", line 209, in __exit__
    self.rollback()
  File "C:\Program Files\borg\bin\borg\remote.py", line 385, in rollback
    return self.call('rollback')
  File "C:\Program Files\borg\bin\borg\remote.py", line 272, in call
    for resp in self.call_many(cmd, [args], **kw):
  File "C:\Program Files\borg\bin\borg\remote.py", line 324, in call_many
    raise Exception('FD exception occurred')
Exception: FD exception occurred

How can I further debug this issue?

Greetings,
Hendrik

[ThomasWaldmann]

@henfri often it is a very fundamental problem with ssh not working.

[henfri]

Hello Thomas,

wouldn't it be good to make the output a bit more clear and even isolate the problem further in the output?

What's surprising: The error happens after a long time. I am able to backup successfully if I select smaller folders.

It's a bit of a locked situation.. I cannot do anything, I fear.

Any Ideas?

[ThomasWaldmann]

Check if you ssh works if you invoke it with similar params manually.

Not sure how we can do better as the failure is in ssh.

[anarcat]

so what's the status of the Windows port? the install docs refer to the Linux subsystem or Cygwin but do not seem to refer to this .zip build... was that intentionally removed? or is that code still sitting only in a feature branch?

i may need to work on windows for a client and I'm wondering what the best way to go would be - should i try that windows branch? or cygwin?

[henfri]

Hello,
Where do I find the most recent windows binary?
Greetings,
Hendrik

[Anakonda]

https://ci.appveyor.com/project/borgbackup/borg/build/artifacts

[quicktrick]

Hi guys! Could you please give me some points on how to use this port on Windows 10? I extracted windowsbuild.zip to some folder, extracted library.zip too, launched a command prompt as Administrator, but when I try to start the program I get:

d:\borgbackup>borg help
Fatal Python error: Py_Initialize: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000a3c (most recent call first):

d:\borgbackup>

What am I doing wrong?

[Anakonda]

Do not extract 'library.zip'
Those are automated builds and may be broken.

[quicktrick]

Thanks for your reply, @Anakonda
With not extracted library.zip the output is:

D:\borgbackup>borg help
Failed to import the site module
ModuleNotFoundError: No module named 'site'

D:\borgbackup>

[enkore]

If you're looking for a Windows backup software for productive use, it's probably a better idea to stick to something better tested on Windows.

[quicktrick]

You are probably right @enkore. I'm usually using another backup solutions on Windows. And I'm also using Borg backup on about 30 Linux and FreeBSD servers. Thanks for your reply anyway. There are some mixed solutions of Borg backup on Linux + Windows I'm going to consider.

[henfri]

Hello,

sorry, but where under https://ci.appveyor.com/project/borgbackup/borg/build/artifacts do I find the download?

Appart from that: What's the future of the windows-builds.
I find it a pitty, that this work is not merged into the dev-branch. Is that even still possible? Or would it be easier to re-start the modifications from the current dev-branch?

Regards,
Hendrik

[henfri]

Hello -again,

apart from my previous question:
I still have an older executable (1.1) for windows.
I run

pageant my_key.ppk
plink borg@myserver

--> I can login without password.
But If I run borg now, I am asked for a bassword. What could be the reason for that?

Regards,
Hendrik

[ThomasWaldmann]

@henfri can't tell much about the windows stuff except that it needs more developers, reviewers, testers and a more continued effort. I can only review very simple windows related changed, to review more advanced stuff it needs developers with experience of windows fs and apis.

[henfri]

Hello,
this is a bit of a hen and egg problem:
I am sure I can test and fix -but only from something that is running. Also, I am concerned about the big gap between dev and the windows branch.
For me, it seems neccessary to first integrate the existing modifications into the master branch to avoid the branches diverging.

Greetings,
Hendrik

[ThomasWaldmann]

There is more recent windows related work in master branch that was done independently of the windows branch. It was smaller, limited scope stuff that I could review and so it got merged. Maybe doing the smaller, simpler steps first makes sense.

[henfri]

yes. Do you know, whether the current master branch works in windows?

I think, that also we can distinguish between:
a) running in windows in a python distribution
b) a 'compiled' (frozen) executable

In order not to loose the existing work though, it would be good if someone familiar with it could review the old commits in the windows branch, adopt them into the current master and create (small) pull requests.

[ThomasWaldmann]

I don't know.

I only rarely test borg on cygwin, but i don't have a wsl nor a native win32 development system.

Yeah, someone(tm) could do that. Start with the simple stuff.

I think the initial goal should be something like "backup and restore my documents folder" rather than something with ntfs permissions / ACLs, special features, etc.

[Anakonda]

@henfri as the link says appveyour keeps artifacts only for 6 months. New ones are generated automatically as soon as something is pushed to the windows branch.
If you have windows 10 I would suggest that you start using the openssh from microsoft. https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse

---

I have not tested the cygwin port, running with wsl should work as is.

Most of the code in windows branch can be used again when it is time to do Windows spesific stuff.

I could maybe do something if it were to master branch and not to windows branch.

[henfri]

Hi Thomas,

I understand.
From a pure git perspective: Is there any help?
I mean some way to look at the commits/changes in the windows branch and then try to merge them into the current master, showing conflicts and then, I could try to resolve them.

Regards,
Hendrik

[ThomasWaldmann]

git show, git cherry-pick. not sure if there is something better / more comfortable.

[henfri]

@Anakonda thanks for your reply

I could maybe do something if it were to master branch and not to windows branch.

What do you mean by that?

[Anakonda]

I meant that its discouraging to see lot of commits pushed to master thinking I should merge them to windows branch. The merges I ended up doing were not easy. I don't want that mess again.

[henfri]

Hello,

I fully understand that. That is also my concern. I suppose you suggest that first your work should be merged into master, right?

@ThomasWaldmann What's your view on this? I think @Anakonda has a point here. I assume, you would merge, if the changes come in smaller pull requests?

Regards,
Hendrik

[ThomasWaldmann]

I won't merge the windows branch "as is". It is beyond my capability to review or maintain it.

Also (IIRC) at least some of the changesets aren't very "clean" / "focussed".

Smaller, focussed, clean, "simple enough" PRs are welcome.

More difficult ones are welcome, too, if there is somebody reviewing them and there are windows developers being able to maintain the code.

[henfri]

Hello Thomas,

I fear, this way there is no way forward. I can understand Anakondas frustration/unwillingness to maintain the current approach of having to merge everything that is done in master into windows.

I have the impression that anakonda is willing to maintain the code. And I would use and test it.

So, what would be needed:

1. you to clarify what you did not find clean/focussed
2. anakonda to check whether smaller PRs are possible and whether he is willing to do these.

What do you think?

Greetings,
Hendrik

[ThomasWaldmann]

the code should be reviewed by somebody with a clue of win32 api.

if it is unclear to @Anakonda what's unclean/unfocussed, i can point at it, but it might be rather obvious.

[henfri]

Hello,

I reviewed this one:
126921d
which is the initial commit.
What I see that could be unclean is:

• UID/GID set to 0
• Mix of changes to make compatible to windows AND changes in order to be able to compile/install.

Apart from that, I do not see anything that could break functionality on non win32 platforms and nothing that would be 'dirty'.

Greetings,
Hendrik

[Anakonda]

I suggest that we forget the windows branch for now, it is no way mergeable with master, instead start making (very) small changes to master.
Maybe get pip install -e . to pass as first commit.

[luke-jr]

Can AppVeyor be restarted to regenerate the Windows builds? Presumably it's better than nothing?

[TheRealAlexV]

Would also like to see a windows version

[z11k]

+1 for win build

[maxkratz]

+1