Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MRG] Creates nightly builds for Python >= 3.6 on all platforms #17

Merged
merged 59 commits into from Feb 22, 2019

Conversation

Projects
None yet
6 participants
@thomasjpfan
Copy link
Contributor

thomasjpfan commented Feb 18, 2019

Resolves scikit-learn/scikit-learn#9485

To enable daily builds, it has to be enabled with travis-ci's cron-job and appveyor's scheduled builds.

thomasjpfan added some commits Jan 30, 2019

@thomasjpfan

This comment has been minimized.

Copy link
Contributor Author

thomasjpfan commented Feb 19, 2019

This PR has been updated to only build for 3.6/3.7 on osx, linux, windows(64 bit). This is a total of 6 daily builds.

@ogrisel

This comment has been minimized.

Copy link
Contributor

ogrisel commented Feb 21, 2019

If we want the osx nightly build, we can merge scikit-learn/scikit-learn#12651 into master.

I would rather just finish the review of the new K-Means implementation ( scikit-learn/scikit-learn#11950 ) which I believe would fix this unstable test on macOS. @jeremiedbb let me know if I am wrong.

@jeremiedbb

This comment has been minimized.

Copy link
Contributor

jeremiedbb commented Feb 21, 2019

I do think it fixes the test. However, originally I made this test in this PR and then moved it in a separate PR to ease reviews. So it would not be absurd to remove this test from master and put it back in my PR.

@ogrisel
Copy link
Contributor

ogrisel left a comment

LGTM besides the following comments:

Show resolved Hide resolved .travis.yml Outdated
Show resolved Hide resolved .travis.yml Outdated
Show resolved Hide resolved .travis.yml Outdated
Show resolved Hide resolved .travis.yml Outdated
@adrinjalali

This comment has been minimized.

Copy link

adrinjalali commented Feb 21, 2019

thanks for doing all of these in parallel @thomasjpfan. Do you think if we move to microsoft CI, we should also move these to that CI?

thomasjpfan added some commits Feb 21, 2019

@thomasjpfan

This comment has been minimized.

Copy link
Contributor Author

thomasjpfan commented Feb 21, 2019

@adrinjalali It is possible to move our build to azure. We would need to adapt https://github.com/matthew-brett/multibuild to run on azure.

For now, I think it is best to keep this running on Travis and AppVeyor.

@ogrisel
Copy link
Contributor

ogrisel left a comment

The container name should be updated in the appveyor configuration.

BTW I have created the sklearn-nightly container on rackspace files and added a CNAME entry so that its content should be available under http://nightly.scikit-learn.org/ at some point (it does not seem to work yet). I don't know how to get https for this though.

Show resolved Hide resolved appveyor.yml Outdated
Update appveyor.yml
Co-Authored-By: thomasjpfan <thomasjpfan@gmail.com>
@amueller

This comment has been minimized.

Copy link
Contributor

amueller commented Feb 22, 2019

The owner of this web site has set X-Container-Meta-Web-Index: index.html. However, this file is not found. @ogrisel is that what you saw? Looks like it requires some config on the server, not just the cname.

# Maybe get and clean and patch source
- clean_code $REPO_DIR $BUILD_COMMIT
- build_wheel $REPO_DIR $PLAT
# Maybe get and clean and patch source

This comment has been minimized.

@amueller

amueller Feb 22, 2019

Contributor

patch? for the wheel building?

@amueller amueller merged commit 89029a7 into MacPython:master Feb 22, 2019

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@amueller

This comment has been minimized.

Copy link
Contributor

amueller commented Feb 22, 2019

Ah, wheelhouse_uploader creates the index, so all is good, never mind.

@amueller

This comment has been minimized.

Copy link
Contributor

amueller commented Feb 22, 2019

enabled for travis, no idea how to do it for appveyor, maybe need to log in with the appveyor admin account?
Also this: https://www.appveyor.com/blog/2017/11/08/sheduled-builds-for-free-accounts/

@amueller amueller referenced this pull request Feb 22, 2019

Closed

[WIP] Nighthly builds #4

@thomasjpfan

This comment has been minimized.

Copy link
Contributor Author

thomasjpfan commented Feb 22, 2019

@ogrisel The first windows build has been uploaded here: http://68bea3902784d69c0aa8-ef5ddc96fef712c596c71d053dc2ad71.r77.cf2.rackcdn.com

The rest of the builds say they are uploaded to the same location. The index file has not been updated on the CDN because the TTL was set to 3 days. You can set this to 15 minutes by:

screen shot 2019-02-22 at 3 46 31 pm

screen shot 2019-02-22 at 3 48 22 pm

We can set the TTL to one hour after we confirm that everything works.

@ogrisel

This comment has been minimized.

Copy link
Contributor

ogrisel commented Feb 25, 2019

I changed the TTL, deleted the old sklearn-nightly container that I created in the North Virginia datacenter (IAD) instead of Chicago (ORD), and updated the CNAME record for http://nightly.scikit-learn.org/ and it seems to work now.

@rth

This comment has been minimized.

Copy link

rth commented Feb 25, 2019

Great work on making nightly wheels available!

@ogrisel

This comment has been minimized.

Copy link
Contributor

ogrisel commented Feb 25, 2019

I checked an unfortunately it's not possible to configure HTTPS on CNAME entries that point to rackspace cloud files / CDN.

@ogrisel

This comment has been minimized.

Copy link
Contributor

ogrisel commented Feb 25, 2019

I tried:

pip install --pre -f http://nightly.scikit-learn.org scikit-learn

and it works (with a warning).

@amueller

This comment has been minimized.

Copy link
Contributor

amueller commented Feb 25, 2019

@ogrisel the https is a rackspace issue? Should we look into finding an alternative? Or replace the issue on the sklearn issue tracker with one that asks for https hosting?

@ogrisel

This comment has been minimized.

Copy link
Contributor

ogrisel commented Feb 25, 2019

The fact that we cannot setup https for CNAME -based URLs (e.g. nightly.scikit-learn.org in our case) is a limitation of rackspace CDN.

If we use the raw CDN URL, it should work with HTTPS but apparently the certificate is not valid:

$ pip install -U --pre -f https://68bea3902784d69c0aa8-ef5ddc96fef712c596c71d053dc2ad71.r77.cf2.rackcdn.com scikit-learn
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Looking in links: https://68bea3902784d69c0aa8-ef5ddc96fef712c596c71d053dc2ad71.r77.cf2.rackcdn.com
Certificate did not match expected hostname: 68bea3902784d69c0aa8-ef5ddc96fef712c596c71d053dc2ad71.r77.cf2.rackcdn.com. Certificate: {'crlDistributionPoints': (u'http://crl3.digicert.com/ssca-ecc-g1.crl', u'http://crl4.digicert.com/ssca-ecc-g1.crl'), 'subjectAltName': (('DNS', 'a248.e.akamai.net'), ('DNS', '*.akamaihd.net'), ('DNS', '*.akamaized-staging.net'), ('DNS', '*.akamaihd-staging.net'), ('DNS', '*.akamaized.net')), 'notBefore': u'Oct 18 00:00:00 2018 GMT', 'caIssuers': (u'http://cacerts.digicert.com/DigiCertECCSecureServerCA.crt',), 'OCSP': (u'http://ocsp.digicert.com',), 'serialNumber': u'04E528087A9D1BCDC05BECFC68C569D5', 'notAfter': 'Oct 18 12:00:00 2019 GMT', 'version': 3L, 'subject': ((('countryName', u'US'),), (('stateOrProvinceName', u'Massachusetts'),), (('localityName', u'Cambridge'),), (('organizationName', u'Akamai Technologies, Inc.'),), (('commonName', u'a248.e.akamai.net'),)), 'issuer': ((('countryName', u'US'),), (('organizationName', u'DigiCert Inc'),), (('commonName', u'DigiCert ECC Secure Server CA'),))}
Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(CertificateError("hostname '68bea3902784d69c0aa8-ef5ddc96fef712c596c71d053dc2ad71.r77.cf2.rackcdn.com' doesn't match either of 'a248.e.akamai.net', '*.akamaihd.net', '*.akamaized-staging.net', '*.akamaihd-staging.net', '*.akamaized.net'",),)': /

so indeed we could try to find an alternative solution...

@thomasjpfan

This comment has been minimized.

Copy link
Contributor Author

thomasjpfan commented Feb 25, 2019

The $0.10/GB is the standard CDN charge, for the first 10TB, so we should be able to turn on https without much more fees: https://support.rackspace.com/how-to/rackspace-cdn-secure-delivery-options/ (Using the SAN certificate)

(There is an additional $0.010 per 10,000 requests for HTTPS)

@thomasjpfan

This comment has been minimized.

Copy link
Contributor Author

thomasjpfan commented Feb 26, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.