Add AppVeyor build with autoupload when commit is tagged with 'rel_*' #291

RazerM · 2016-07-19T22:10:25Z

Create an AppVeyor account and enable the SQLAlchemy repository (presumably the bitbucket one).
Encrypt PyPI credentials on AppVeyor by clicking your name at the top right and clicking "Encrypt data" under "Tools".

3. Put those values in appveyor.yml.

I've included the appveyor/getwheels.py script, which may be useful if you want to download the latest wheels for a build. I can remove it if you don't want it in the repo.

zzzeek · 2016-07-20T14:52:52Z

this is great, thanks so much! I'll hopefully find time to try this out in the coming days.

zzzeek · 2016-11-30T16:14:59Z

first observation: appveyors oauth for bitbucket wants to request global administrator perms over all my bitbucket repos. for github, it is much more appropriately targeted at read-only access. so pulling from github seems more likely here.

zzzeek · 2016-11-30T16:15:46Z

appveyor.yml

+    - PYTHON: "C:\\Python26-x64"
+    - PYTHON: "C:\\Python27-x64"
+    - PYTHON: "C:\\Python33-x64"
+      DISTUTILS_USE_SDK: "1"


what is the significance of DISTUTILS_USE_SDK? How do I add Python36 to this? do i need DISTUTILS_USE_SDK for that too ?

It's used in build.cmd and "sets up the SDK needed for 64-bit builds of Python 3.3 or 3.4". More info here. It's not required on Python 3.5 and onwards.

zzzeek · 2016-11-30T16:21:24Z

confirm I can put all these scripts into the appveyor GUI directly rather than committing them to the repo ? Looks like I can set build->script, after_build, etc. This would be a lot more portable.

RazerM · 2016-11-30T16:43:00Z

I'm not sure how that would work with the separate build.cmd script, but in principle yes.

zzzeek · 2016-11-30T18:49:04Z

OK, I didn't realize the support script was put out by pypa itself, that is a little better. basically my fear is having to support this. I can turn it on now and it will work, but then it's going to keep breaking every N months for whatever reason that adds to my workload.

zzzeek · 2016-11-30T19:01:27Z

appveyor.yml

+  # Note that you must use the environment variable %PYTHON% to refer to
+  # the interpreter you're using - Appveyor does not do anything special
+  # to put the Python version you want to use on PATH.
+  - "appveyor\\build.cmd %PYTHON%\\python.exe setup.py bdist_wheel"


I think this needs to be like:

python.exe setup.py egg_info --tag-build='' bdist_wheel

so that we don't have the 'dev' tag that comes from setup.cfg

zzzeek · 2016-11-30T20:02:56Z

I've got four environments building and also I had to rewrite the .bat file as powershell, now they are all failing like this:



C:\Python27\lib\site-packages\wheel\pep425tags.py:77: RuntimeWarning: Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
  warn=(impl == 'cp')):
C:\Python27\lib\site-packages\wheel\pep425tags.py:81: RuntimeWarning: Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect
  warn=(impl == 'cp')):
C:\Python27\lib\site-packages\wheel\pep425tags.py:87: RuntimeWarning: Config variable 'Py_UNICODE_SIZE' is unset, Python ABI tag may be incorrect
  sys.version_info < (3, 3))) \
Command executed with exception: C:\Python27\lib\site-packages\wheel\pep425tags.py:77: RuntimeWarning: Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
  warn=(impl == 'cp')):
C:\Python27\lib\site-packages\wheel\pep425tags.py:81: RuntimeWarning: Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect
  warn=(impl == 'cp')):
C:\Python27\lib\site-packages\wheel\pep425tags.py:87: RuntimeWarning: Config variable 'Py_UNICODE_SIZE' is unset, Python ABI tag may be incorrect
  sys.version_info < (3, 3))) \

I think I can add you as a "collaborator" that would allow you to poke at it.

RazerM · 2016-11-30T20:37:08Z

Those warnings can be ignored on Windows, see pypa/pip#3383 (comment)

Looking at the log, it appears as if PowerShell can't parse something

At line:17 char:1
+ & "${env:PYTHON}\python.exe" setup.py egg_info --tag-build='' bdist_w ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (C:\Python26\lib...ay be incorrect:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

zzzeek · 2016-11-30T21:37:36Z

It is invoking the command I'm giving it. It's running through the first few phases just fine and It's only failing at the "install_scripts" step. the "line:18 char:1" seems to only be showing what command was run that failed (e.g. the & command).

zzzeek · 2016-11-30T21:42:56Z

needed an artifact config. now it succeeded.

zzzeek · 2016-11-30T22:11:24Z

So I can't get the script https://raw.githubusercontent.com/pypa/python-packaging-user-guide/master/source/code/build.cmd to run as stated in appveyor's GUI, nor can I get the powershell version to actually run that alternate compiler. So I can't get the C extensions to build for Py33-64, 34-64.

Also, I don't see how to get this to build an existing tag. It only has build master or build a branch. I see a web API to build a specific version but I don't feel like writing a POST script, I just want to push a button and build rel_1_1_14.

Also I push those tags up before I do "python setup.py register", so I'd probably want to do this manually, so i need a script I guess to do that POST to build on a certain tag and also to figure out how to get the appveyor GUI to send the right magic to build those two python 3 versions, or otherwise just build for python3.5 and be done with it.

RazerM · 2016-11-30T22:41:56Z

nor can I get the powershell version to actually run that alternate compiler.

Is there a reason why you didn't use CMD? I'd be happy to take a look to get it working.

Also, I don't see how to get this to build an existing tag. It only has build master or build a branch. I see a web API to build a specific version but I don't feel like writing a POST script, I just want to push a button and build rel_1_1_14.

I don't think there's a way to do this on the client, but I wouldn't expect this to be required. I can see that it'd be useful to test that it works now, though.

Also I push those tags up before I do "python setup.py register", so I'd probably want to do this manually, so i need a script I guess to do that POST to build on a certain tag and also to figure out how to get the appveyor GUI to send the right magic to build those two python 3 versions, or otherwise just build for python3.5 and be done with it.

I don't understand why python setup.py register would be part of the release workflow, it's only useful for updating the classifiers and long_description etc., no?

zzzeek · 2016-11-30T22:50:13Z

if you look at the first few builds it kept giving me the cryptic error message "The system cannot find the batch label specified -". googling / stackoverflowing did not help, I made sure it was CR/LF, took out all the comments in case it was the colons, no luck. From what I see in appveyor's docs here: https://www.appveyor.com/docs/build-configuration/#script-blocks-in-build-configuration , this is possibly not like a regular .bat file. The powershell version at least had normal error messages.

zzzeek · 2016-11-30T22:53:14Z

I use python setup.py register to establish the new version number. If you only need to do "sdist upload" to do that, I was not aware. Pypi now has no way to remove a version once you put it up, a mistake means your version is ruined and I have to do a full repackage + retag + have a hole in my versions (this happened w/ Alembic once when I attempted to use some shiny new tool that was recommended) so I am extremely cautious and conservative in how I deal wtih it.

Not sure how the tagging thing is supposed to work. Suppose I push the tag for rel_1_1_15. Something goes wrong on appveyor, like a network problem. Now I can't rebuild ?

RazerM · 2016-11-30T23:05:43Z

Ah yes, I see now:

When you set Cmd script on the UI its body will be split into lines and executed as separate commands with an exit code check after each line.

I can try to get the PowerShell working then.

Yes, uploading via sdist upload or twine upload will establish the new version number.

If the build for the tag fails, you can always re-build a given commit:

zzzeek · 2016-12-01T16:18:42Z

still, if we put this up, I can't build wheels for the existing tags w/o writing a POST script.

RazerM · 2016-12-03T21:43:50Z

Ok I have an alternative solution, which includes building wheels for macOS and Linux, and lets you build for old tags easily

https://github.com/RazerM/sqlalchemy-wheels

After setting up the PyPI variables, you could build wheels on all 3 platforms for a given tag like this:

./pushbuild.sh -u rel_1_1_4

RazerM · 2017-03-28T22:04:07Z

I've been building wheels for every release using https://github.com/RazerM/sqlalchemy-wheels and it works well.

zzzeek · 2017-03-29T05:18:51Z

this PR is still in my inbox as something I need to look at (I do "inbox zero" so the 8 or 9 messages that are there are significant). I'd have to find the time to go into it again but I recall there were limitations of travis making it difficult to integrate this into my workflow.

agronholm · 2017-12-09T23:47:59Z

I'm just leaving a link here to my solution: http://alextechrants.blogspot.fi/2017/10/automating-packaging-and-uploading-of.html

zzzeek · 2018-12-02T23:51:25Z

Hi there -

I'd still like to do this but I'd need someone to help integrate it into my workflow, at the moment the best workflow is where I'd relase to pypi normally, then that would trigger the placement of a .whl along with the same pypi release. A more complex workflow where we wait for the windows .whl to complete succesfully before building would require some more effort to integrate here, as I'd like builds to come from jenkins jobs and in this case the jenkins job would need to coordinate with appveyor to do the windows build, since I have no way of hosting any windows OSes or hardware.

in any case, if contrbutors can continue to help with this, please reopen on the new repo at https://github.com/sqlalchemy/sqlalchemy, thanks!

Add AppVeyor build with autoupload when commit is tagged with 'rel_*'

0d1fb10

RazerM force-pushed the appveyor branch from 0c14eda to 0d1fb10 Compare July 19, 2016 22:11

zzzeek reviewed Nov 30, 2016

View reviewed changes

sqlalchemy-bot mentioned this pull request Nov 27, 2018

Windows wheels sqlalchemy/sqlalchemy#3747

Closed

zzzeek closed this Dec 2, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add AppVeyor build with autoupload when commit is tagged with 'rel_*' #291

Add AppVeyor build with autoupload when commit is tagged with 'rel_*' #291

RazerM commented Jul 19, 2016 •

edited

zzzeek commented Jul 20, 2016

zzzeek commented Nov 30, 2016

zzzeek Nov 30, 2016

RazerM Nov 30, 2016

zzzeek commented Nov 30, 2016

RazerM commented Nov 30, 2016

zzzeek commented Nov 30, 2016

zzzeek Nov 30, 2016

zzzeek commented Nov 30, 2016 •

edited

RazerM commented Nov 30, 2016

zzzeek commented Nov 30, 2016

zzzeek commented Nov 30, 2016

zzzeek commented Nov 30, 2016

RazerM commented Nov 30, 2016

zzzeek commented Nov 30, 2016

zzzeek commented Nov 30, 2016

RazerM commented Nov 30, 2016

zzzeek commented Dec 1, 2016

RazerM commented Dec 3, 2016

RazerM commented Mar 28, 2017

zzzeek commented Mar 29, 2017

agronholm commented Dec 9, 2017

zzzeek commented Dec 2, 2018

Add AppVeyor build with autoupload when commit is tagged with 'rel_*' #291

Add AppVeyor build with autoupload when commit is tagged with 'rel_*' #291

Conversation

RazerM commented Jul 19, 2016 • edited

zzzeek commented Jul 20, 2016

zzzeek commented Nov 30, 2016

zzzeek Nov 30, 2016

Choose a reason for hiding this comment

RazerM Nov 30, 2016

Choose a reason for hiding this comment

zzzeek commented Nov 30, 2016

RazerM commented Nov 30, 2016

zzzeek commented Nov 30, 2016

zzzeek Nov 30, 2016

Choose a reason for hiding this comment

zzzeek commented Nov 30, 2016 • edited

RazerM commented Nov 30, 2016

zzzeek commented Nov 30, 2016

zzzeek commented Nov 30, 2016

zzzeek commented Nov 30, 2016

RazerM commented Nov 30, 2016

zzzeek commented Nov 30, 2016

zzzeek commented Nov 30, 2016

RazerM commented Nov 30, 2016

zzzeek commented Dec 1, 2016

RazerM commented Dec 3, 2016

RazerM commented Mar 28, 2017

zzzeek commented Mar 29, 2017

agronholm commented Dec 9, 2017

zzzeek commented Dec 2, 2018

RazerM commented Jul 19, 2016 •

edited

zzzeek commented Nov 30, 2016 •

edited