Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixes #1631: Moved make shell commands into scripts for Windows #1636

Merged
merged 4 commits into from
Feb 22, 2019
Merged

Fixes #1631: Moved make shell commands into scripts for Windows #1636

merged 4 commits into from
Feb 22, 2019

Conversation

andy-maier
Copy link
Contributor

@andy-maier andy-maier commented Feb 3, 2019 •
edited
Loading

For details, see the commit message.

Test: Raamesh wantes to test this on native Windows, see issue #1631.

@andy-maier andy-maier self-assigned this Feb 3, 2019
@andy-maier andy-maier changed the title Moved make shell commands into scripts for Windows Fixes #1631: Moved make shell commands into scripts for Windows Feb 3, 2019
@andy-maier andy-maier added this to the 0.13.0 milestone Feb 3, 2019
@andy-maier andy-maier mentioned this pull request Feb 3, 2019
Closed
@coveralls
Copy link

coveralls commented Feb 3, 2019 •
edited
Loading

Coverage Status

Coverage decreased (-0.1%) to 84.067% when pulling 2e5b90f on andy/make-windows-no-quotes into 509436e on master.

@KSchopmeyer
Copy link
Collaborator

I reviewed the code with no issues.

However, trying to test in a windows system that has a make I hit several issues immediatly:

  1. does not install pbr. Did pip install pbr to get around that.

  2. Issue with curl. This environment includes curl and it can be called from the cmd line but the make environment does not find it.


(pywbem27) C:\Users\kschopmeyer\dev\pywbem\pywbem>make install
"makefile: Installing OS-level installation and runtime requirements"
"Debug: PATH=C:\Users\stduser\Envs\pywbem27\Scripts;C:\Python27\;C:\Python27\Scripts;C:\ProgramData\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\cvsnt;C:\bin;C:\bin\usr\local\bin;C:\Program Files (x86)\Skype\Phone\;C:\Program Files\Git\cmd;C:\Program Files (x86)\PuTTY\;C:\Program Files (x86)\Bitvise SSH Client;C:\WINDOWS\System32\OpenSSH\;C:\ProgramData\chocolatey\bin;C:\Users\stduser\AppData\Local\Programs\Python\Python37-32\Scripts\;C:\Users\stduser\AppData\Local\Programs\Python\Python37-32\;C:\Users\stduser\AppData\Local\Microsoft\WindowsApps;"
cmd /d /c pywbem_os_setup.bat install
Current Python is: Version 2,  64-bit
Installing OS-level prerequisite packages for install on platform Windows ...
Swig is already installed ... skipping
Installing Win64OpenSSL-1_1_0j ...
curl -o Win64OpenSSL-1_1_0j.exe -sSL https://slproweb.com/download/Win64OpenSSL-1_1_0j.exe
'curl' is not recognized as an internal or external command,
operable program or batch file.
Error: Command returned rc=9009
make: *** [install_os.done] Error 9009

(pywbem27) C:\Users\kschopmeyer\dev\pywbem\pywbem>which curl
which: no curl in (C:\Users\stduser\Envs\pywbem27\Scripts;C:\Python27\;C:\Python27\Scripts;C:\ProgramData\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\cvsnt;C:\bin;C:\bin\usr\local\bin;C:\Program Files (x86)\Skype\Phone\;C:\Program Files\Git\cmd;C:\Program Files (x86)\PuTTY\;C:\Program Files (x86)\Bitvise SSH Client;C:\WINDOWS\System32\OpenSSH\;C:\ProgramData\chocolatey\bin;C:\Users\stduser\AppData\Local\Programs\Python\Python37-32\Scripts\;C:\Users\stduser\AppData\Local\Programs\Python\Python37-32\;C:\Users\stduser\AppData\Local\Microsoft\WindowsApps;)

(pywbem27) C:\Users\kschopmeyer\dev\pywbem\pywbem>curl -h
Usage: curl [options...] <url>
     --abstract-unix-socket <path> Connect via abstract Unix domain socket
     --anyauth       Pick any authentication method
 -a, --append        Append to target file when uploading

(pywbem27) C:\Users\kschopmeyer\dev\pywbem\pywbem>curl --version
curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
Release-Date: [unreleased]
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL

(pywbem27) C:\Users\kschopmeyer\dev\pywbem\pywbem>make --version
GNU Make version 3.79.1, by Richard Stallman and Roland McGrath.
Built for Windows32
Copyright (C) 1988, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 2000
        Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

Report bugs to <bug-make@gnu.org>.

@KSchopmeyer KSchopmeyer closed this Feb 4, 2019
@KSchopmeyer KSchopmeyer reopened this Feb 4, 2019
@KSchopmeyer
Copy link
Collaborator

Accidently hit the close and comment button and then reopened.

This could all be my environment problems since trying just to get gnu make enabled on windows is a task but that is what I found on first tests.

@andy-maier
Copy link
Contributor Author

Hi karl. I have updated this branch by calling external commands from pywbem_os_setup.bat with "call", just in case someone has a curl.bat or curl.cmd. Also, I have added support for installing curl using Chocolatey if "where curl" fails. The "where" command is the built-in "which" of native Windows.

Please try again with this branch. Please try this on half-way set up environments as well as on environments that are reasonably fresh (e.g. freshly created virtual python env, run make clobber).

@andy-maier andy-maier force-pushed the andy/make-windows-no-quotes branch 2 times, most recently from 7312952 to c6ca8e5 Compare February 7, 2019 08:28
@KSchopmeyer
Copy link
Collaborator

KSchopmeyer commented Feb 8, 2019 •
edited by andy-maier
Loading

Started testing on windows. Testing with existing virtual env first. Note that this make I am using is the old gnumake but that there is also a bash but VERY old in the same directory.

  1. hit problem with clobber

    (pywbem3) C:\Users\kscho\dev\pywbem>make clobber
makefile: Removing temporary build products
/bin/find . -name '*.pyc' -delete
/bin/find: invalid predicate `-delete'
make: *** [clean] Error 1

  2. make build worked (executed install also)

  3. Had to manually install flake8. But that could be issue with my existing virt env.

    Installed flake8 with pip. flake8 works.

    However, make check fails. Cannot find flake8. This is probably because it is doing:

    /bin/bash: flake8: command not found

  4. Of course make test fails for same reason. Trying to use bash but under windows.

DISCUSSION: How far do we want to go with making the development environment work under window??? What all should be in the develop environment in order to do a make with pure windows?
i.e are we requiring bash??? That is a question right now, since I am not sure.

@KSchopmeyer
Copy link
Collaborator

lets try:

bash invocation in make that shows path. Looks like we are getting the windows bash now and for some reason, the path is not correct.

@andy-maier andy-maier force-pushed the andy/make-windows-no-quotes branch 6 times, most recently from c597dfd to c46fc2c Compare February 9, 2019 17:14
@andy-maier andy-maier force-pushed the andy/make-windows-no-quotes branch 4 times, most recently from eb155b1 to ecff41e Compare February 17, 2019 22:02
@andy-maier
Copy link
Contributor Author

The latest update also bringsthe Installation section of the documentation in sync with the code.

@andy-maier
Copy link
Contributor Author

Please test the installation and the setup of the development environment on Windows.

@andy-maier
Copy link
Contributor Author

andy-maier commented Feb 18, 2019 •
edited
Loading

I added retry support for choco installations, by using the "appveyor-retry" command. It retries 3 times (including the original attempt), and has no sleep time between retries.

In the Appveyor run for this latest update to the PR we had 3/3 successes without retries.

I rebased the "manual-ci-run" on this PR, and the Appveyor run there now has 32 jobs, and I already left out some combinations. In the recent run based on the latest update to this PR, its 32 jobs succeed or failed as follows:

So this got better than without the retries (I saw 5 jobs failing out of the 32, before the last update), but it is not yet safe enough for automatic runs. Of course, if it happens rarely enough, we can rerun the failing jobs (Appveyor offers that button once the whole build is finished).

Discussion point: I added a tools/retry.bat script in the latest update that replaces the use of the appveyor-retry command. It has several advantages:

  • It is used in pywbem_os_setup.bat also when run by the user (not just when running in Appveyor).
  • It has a sleep time between retries.
  • We can configure the number of retries.

The discussion point is whether we should keep that script around or fall back to using the appveyor-retry command?

@andy-maier andy-maier force-pushed the andy/make-windows-no-quotes branch 5 times, most recently from 67d477f to 981fbcf Compare February 18, 2019 11:48
@andy-maier andy-maier force-pushed the andy/make-windows-no-quotes branch 2 times, most recently from 06937ba to 230199d Compare February 22, 2019 07:30
@andy-maier
Copy link
Contributor Author

In our call today, Karl and I agreed to merge this one ot make progress. We would keep the retry.bat script.

@andy-maier
Test: Fixed a bytes/unicode error in validate.py
b688cde 
Signed-off-by: Andreas Maier <andreas.r.maier@gmx.de>
@andy-maier
Changed invocation of wbemcli in unit tests to be like in reality
1ffd392 
Signed-off-by: Andreas Maier <andreas.r.maier@gmx.de>
@andy-maier
Improved make on native Windows and in Appveyor
21a4f6e 
Details:

- Some GNU make versions on native Windows have an issue with double
  quotes in make $(shell ..) commands; removed the use of double quotes.
  As a result, most inline python commands have been moved into new small
  scripts in the tools directory.
  Also, several make targets that used to produce log files,
  no longer can do that and the user needs to redirect the make invocation
  in order to get a log file.

- Removed dependencies on most UNIX-like commands (touch, tee, bash, rm,
  find, xargs) when using make on native Windows.

- Encapsulated file removal and copy commands to be compatible between
  native Windows and other platforms.

- Updated the appveyor.yml file to check only the new, smaller, list of
  commands.

- Improved dependencies in makefile

- Fixed list of prerequisite commands in makefile

- Added list of prereq commands to pywbem_os_setup.sh

- Suppressed more builtin rules by adding -p to MAKEFLAGS

- Fixed use of too new option for pylint

- GNU make uses as a default shell bash or sh when it is in the PATH.

  For this reason, the internal commands in case of native Windows
  were invoked using "cmd /c".

  On Windows 10, there is a strange behavior with that, whereby
  the command processor is invoked in interactive mode whenever:
  - the command contains an asterisk
  - the command is put into double quotes

  Example:
    cmd /c del /s *.pyc
  invokes the command processor in interactive mode

  To avoid this problem, the "cmd /c" is removed.
  To avoid that bash or sh is used, the SHELL variable is set to
  cmd when running on native Windows.

- Eliminated dependency on xargs by using for loop when
  on Windows.

- Added automatic install of xmllint on Windows

- Removed PYTHONPATH=. from sphinx command. It fails on
  Windows and does not seem to be needed.

- Used $(error ...) when failing due to package version.

- Changed platform names to Windows_UNIX and Windows_native,
  to avoid the CygWin name which is too specific.

- Added a 'platform' target to the makefile that displays
  information about the OS platform detected by make and about
  the shell used by make to execute commands.

- Added an 'env' target to the makefile that displays the
  environment as seen by make.

- Added support to the Appveyor CI control file to allow
  specifying whether and which UNIX-like environment should be
  used, by adding a UNIX_PATH variable.

- The required OS commands are now installed by invoking
  pywbem_os_setup.bat with args install and develop. this became
  necessary in order to dynamically set the UNIX-like environment.
  Commands are installed only if not yet in the PATH, i.e. if not
  provided by the UNIX-like environment.

- The Python virtualenv in Appveyor is no longer created by tox,
  but by virtualenv.

Signed-off-by: Andreas Maier <andreas.r.maier@gmx.de>
@andy-maier
Using retries in Appveyor to overcome choco install timeouts
2e5b90f 
Details:

* Added a tools/retry.bat script that attempts a Windows command
  for a certain number of times (currently 3), with a certain sleep
  time in between attempts (currently 30 seconds).

* Used the retry.bat script whenever choco is used to install packages:
  * 'make' in appveyor.yml
  * some OS level packages in pywbem_os_setup.bat
    In pywbem_os_setup.bat, the retry.bat script is only used after testing
    that it exists. That ensures that the pywbem_os_setup.bat script can
    still be downloaded and run standalone, as described in the
    documentation.

Signed-off-by: Andreas Maier <andreas.r.maier@gmx.de>
@andy-maier andy-maier merged commit 934cb56 into master Feb 22, 2019
@andy-maier andy-maier deleted the andy/make-windows-no-quotes branch February 22, 2019 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@KSchopmeyer KSchopmeyer Awaiting requested review from KSchopmeyer

Assignees

@andy-maier andy-maier

Labels
None yet
Projects
None yet
Milestone
0.13.0
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@andy-maier @coveralls @KSchopmeyer