Add Support for Windows

[chinmayshah99]

Expected Behavior

Add a script to generate a .dll file as part of build process. Also, edits needs to made in setup.py to include this generated dll file. This dll file will be generated using bazel build.

Current Behavior

Right now, pydp only supports Linux systems. This is due to the fact that we are just generating the .so file as seen here.

Logs

>>> import pydp as dp
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\ProgramData\Anaconda3\lib\site-packages\pydp\__init__.py", line 1, in <module>
    from .pydp import *
ModuleNotFoundError: No module named 'pydp.pydp'

System information

• PyDP version: 0.1.0dev
• Operating System: Any windows system

[leclair-7]

I'll be working on this one starting later today. Updates to come!

[BenjaminDev]

@leclair-7 I have access to a windows machine and can look at into this tonight. Are there any learnings you can share that might assist me?

[chinmayshah99]

Hey @BenjaminDev,
Here is the stuff @leclair-7 tried. Some of this might not be relevant (make sure you understand our build process).

Pls note that the reason we are doing this exercise is that we want to release the package for windows machine, so the goal is to generate a .dll file and integrate it into setup.py

1 Installed Visual Studio Community 2019 (bazel for windows said to use "Visual C++ Redistributable for Visual Studio 2015", but my cl.exe (compiler) seemed to work)
2 Installed anaconda (python 3.7.4)
3 Followed the steps on the bazel installer on https://docs.bazel.build/versions/master/install-windows.html
4 Installed Windows subsystem (bazel at one point looked for a bash.exe):
    - Go to microsoft store URL: https://aka.ms/wslstore
    - download/install Ubuntu subsystem
5 Change python_configure.bzl to point to your python bin and lib directories (however we don't need this):
    For me on default installs that was:
        _PYTHON_BIN_PATH = "../../AppData/Local/Continuum/anaconda3"
        _PYTHON_LIB_PATH = "../../AppData/Local/Continuum/anaconda3/Lib"
6 In all BUILD files in the third_party/differential-privacy repo, I ran the vim command :g/Wno-sign/d to delete all lines with Wno-sign-compare
        modified:   differential_privacy/algorithms/BUILD
        modified:   differential_privacy/base/BUILD
        modified:   differential_privacy/example/BUILD
        modified:   differential_privacy/postgres/BUILD
        modified:   differential_privacy/proto/BUILD
        modified:   differential_privacy/testing/BUILD
7 <Path to PyDP>/PyDP ran:
bazel build src/python:bindings_test  --verbose_failures
(it fails due to parsing the python paths from python_configure.bzl)
Errors were 
INFO: Analyzed target //src/python:bindings_test (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
INFO: From Compiling external/com_google_absl/absl/synchronization/internal/graphcycles.cc:
cl : Command line warning D9002 : ignoring unknown option '-std=c++17'
INFO: From Compiling external/com_google_absl/absl/time/format.cc:
cl : Command line warning D9002 : ignoring unknown option '-std=c++17'
INFO: From Compiling external/com_google_absl/absl/time/civil_time.cc:
cl : Command line warning D9002 : ignoring unknown option '-std=c++17'
ERROR: C:/users/hagel/_bazel_hagel/7kb5qlv3/external/local_config_python/BUILD:67:1: Executing genrule @local_config_python//:python_include failed (Exit 1): bash.exe failed: error executing command
  cd C:/users/hagel/_bazel_hagel/7kb5qlv3/execroot/PyDP
  SET PATH=C:\Windows\system32;C:\WINDOWS;C:\WINDOWS\System32;C:\WINDOWS\System32\WindowsPowerShell\v1.0;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\libnvvp;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\NVIDIA Corporation\Nsight Compute 2019.4.0\;C:\Program Files\Git\cmd;C:\Program Files\dotnet\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\bazel;C:\Users\hagel\AppData\Local\Continuum\anaconda3;C:\Users\hagel\AppData\Local\Microsoft\WindowsApps;C:\Users\hagel\AppData\Local\atom\bin;C:\Users\hagel\AppData\Local\Programs\Microsoft VS Code\bin;C:\Program Files (x86)\FAHClient;C:\Users\hagel\AppData\Local\Microsoft\WindowsApps;C:\bazel;C:\Users\hagel\AppData\Local\Continuum\anaconda3;
    SET RUNFILES_MANIFEST_ONLY=1
  C:/Windows/system32/bash.exe bazel-out/x64_windows-fastbuild/bin/external/local_config_python/python_include.genrule_script.sh
Execution platform: @local_config_platform//:host
cp: cannot stat 'C:/Users/hagel/AppData/Local/Continuum/anaconda3/include/LIEF/ART.hpp': No such file or directory
Target //src/python:bindings_test failed to build
ERROR: C:/users/hagel/desktop/pydp/src/bindings/BUILD:3:1 Executing genrule @local_config_python//:python_include failed (Exit 1): bash.exe failed: error executing command
  cd C:/users/hagel/_bazel_hagel/7kb5qlv3/execroot/PyDP
  SET PATH=C:\Windows\system32;C:\WINDOWS;C:\WINDOWS\System32;C:\WINDOWS\System32\WindowsPowerShell\v1.0;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\libnvvp;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\NVIDIA Corporation\Nsight Compute 2019.4.0\;C:\Program Files\Git\cmd;C:\Program Files\dotnet\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\bazel;C:\Users\hagel\AppData\Local\Continuum\anaconda3;C:\Users\hagel\AppData\Local\Microsoft\WindowsApps;C:\Users\hagel\AppData\Local\atom\bin;C:\Users\hagel\AppData\Local\Programs\Microsoft VS Code\bin;C:\Program Files (x86)\FAHClient;C:\Users\hagel\AppData\Local\Microsoft\WindowsApps;C:\bazel;C:\Users\hagel\AppData\Local\Continuum\anaconda3;
    SET RUNFILES_MANIFEST_ONLY=1
  C:/Windows/system32/bash.exe bazel-out/x64_windows-fastbuild/bin/external/local_config_python/python_include.genrule_script.sh
Execution platform: @local_config_platform//:host
INFO: Elapsed time: 1.570s, Critical Path: 1.34s
INFO: 6 processes: 6 local.
FAILED: Build did NOT complete successfully

[BenjaminDev]

There are a few blockers with the windows build:

1. copts = ["-Wno-sign-compare"], are not supported on windows. This is a minor issue and we can simply remove them and it all works. I am unsure the best way to resolve this - if you use bazel on windows I'd love to hear ideas.
2. The SHA that we are using includes #include <libgen.h> in base/logging.cc and this is not supported on windows. We can remove it and upstream has already done so but we would need to pull in a newer "version" (SHA).
3. The last hurdle is that upstream are using #include <unistd> in base/logging.cc which is not supported on windows. If I remove it e get the following errors:

external/google_dp/base/logging.cc(79): error C3861: 'mkdir': identifier not found
external/google_dp/base/logging.cc(82): error C3861: 'S_ISDIR': identifier not found
external/google_dp/base/logging.cc(139): error C2065: 'F_OK': undeclared identifier
external/google_dp/base/logging.cc(139): error C3861: 'access': identifier not found
external/google_dp/base/logging.cc(140): error C2065: 'W_OK': undeclared identifier
external/google_dp/base/logging.cc(140): error C3861: 'access': identifier not found
external/google_dp/base/logging.cc(216): error C2065: 'CLOCK_REALTIME': undeclared identifier
external/google_dp/base/logging.cc(216): error C3861: 'clock_gettime': identifier not found

So we need to find or create windows alternatives for these and include them. As far as I am aware this would need an someone from the upstream project to complete.

So 1) and 2) are not deal breakers but 3) seems to be a slight issue and will need assistance from upstream

[chinmayshah99]

For 1, google/differential-privacy#18, your solution seems the way to go right now.

For 2 and 3, what if temporarily remove logging and try to atleast build. We can see later on what can be done depending on the need of the logging module (we can also write our own custom codebase for same)

[BenjaminDev]

Update:

We have a functioning (but shaky) build on windows. If you would like to test/play with it please do and report issues here not as a new issue as it's a unofficial wheel "among friends":wink:

To test (python 3.8):

1. PS:> cd /to/a/tmp/directory
2. PS:> python -m venv .venv
3. PS:> .\.venv\Scripts\activate
4. (.venv)PS:> python -m pip install https://github.com/BenjaminDev/PyDP/releases/download/0.1.4-cp38-win_amd64-beta.2/python_dp-0.1.4_cp38_win_amd64_beta.2-cp38-cp38-win_amd64.whl
5. (.venv)PS:> python -c "import pydp; print(pydp.__version__)"
6. Test your current code. It works with the carrot demo!

This builds off a personal fork of the google differential privacy repo.

Todo:

• Clean up the patch and ask around for input and improvements - I'll post on slack.
• Put in a PR to the google differential privacy repo.

If the PR is rejected as I suspect it might be as there may be subtleties that I'm unaware of then we could support windows in the current fashion i.e our own fork.

We are moving the windows effort into a test-windows branch and so making a PR with your suggestion should be done against that branch.