New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Have this generate binary wheels with latest amalgamation and ship them to PyPI #2

Open
simonw opened this Issue Aug 15, 2018 · 7 comments

Comments

Projects
None yet
2 participants
@simonw
Contributor

simonw commented Aug 15, 2018

I'm really interested in this project at the moment.

SQLite 3.25.0 is due to ship shortly and will add support for window functions, which is really exciting - but the actual version of SQLite that is available in the standard library sqlite3 module is tied to the underlying operating system and is often very out of date (the Travis CI default build environment still exposes SQLite 3.8 for example!)

I would love to see this project published to PyPI with the latest stable version of the SQLite amalgamation compiled into it, in a binary wheel format such that it can be "pip install"ed on as many platforms as posssible.

I haven't worked with binary compiled wheels myself, but I would be happy to dig in and help figure out how best to do this.
biopython/biopython#1296 has some useful notes and
https://github.com/pypa/manylinux looks relevant. I've done a fair amount of work with Travis CI recently and it can provide build environments for both Linux and OS X (I have no idea how windows would work).

@simonw

This comment has been minimized.

Show comment
Hide comment
@simonw
Contributor

simonw commented Aug 15, 2018

@coleifer

This comment has been minimized.

Show comment
Hide comment
@coleifer

coleifer Aug 15, 2018

Owner

Yes, I posted it to HN a while back but it didn't receive much traction -- glad to see it on the front page today.

I've gotten pretty good at working with travis -- the peewee build has gotten pretty nice (including running against the latest sqlite for one round of tests). Unfortunately, I'm not at all sure how to manage "wheels". Do you need to build separate wheels for x64, x86, arm, darwin, windows? Does it make sense to have users of this library default to running a particular version of sqlite with such-and-such features compiled by default?

Honestly, it's very little work to compile your own sqlite with exactly the features you want, and it's just a tiny bit more effort to create a pysqlite3 that is statically-linked. Info here: http://charlesleifer.com/blog/compiling-sqlite-for-use-with-python-applications/

Owner

coleifer commented Aug 15, 2018

Yes, I posted it to HN a while back but it didn't receive much traction -- glad to see it on the front page today.

I've gotten pretty good at working with travis -- the peewee build has gotten pretty nice (including running against the latest sqlite for one round of tests). Unfortunately, I'm not at all sure how to manage "wheels". Do you need to build separate wheels for x64, x86, arm, darwin, windows? Does it make sense to have users of this library default to running a particular version of sqlite with such-and-such features compiled by default?

Honestly, it's very little work to compile your own sqlite with exactly the features you want, and it's just a tiny bit more effort to create a pysqlite3 that is statically-linked. Info here: http://charlesleifer.com/blog/compiling-sqlite-for-use-with-python-applications/

@simonw

This comment has been minimized.

Show comment
Hide comment
@simonw

simonw Aug 15, 2018

Contributor

It looks like you need to produce a few different wheels for different platforms. The best example I've found so far is this biopython repo which handles all of their wheel generation https://github.com/biopython/biopython-wheels

You can see their results on PyPI - https://pypi.org/project/biopython/

Selfishly, my principle goal is that I'd like to be able to run pip install pysqlite3 (or maybe pip install pysqlite3==3.25.0 or some other package name if you don't want the default package to include the amalgamation) on the platforms I care about (OS X and Linux) and get a working module. Then I can give tutorials on how to use the latest SQLite3 features without having to coach people through a build and compilation step first.

Contributor

simonw commented Aug 15, 2018

It looks like you need to produce a few different wheels for different platforms. The best example I've found so far is this biopython repo which handles all of their wheel generation https://github.com/biopython/biopython-wheels

You can see their results on PyPI - https://pypi.org/project/biopython/

Selfishly, my principle goal is that I'd like to be able to run pip install pysqlite3 (or maybe pip install pysqlite3==3.25.0 or some other package name if you don't want the default package to include the amalgamation) on the platforms I care about (OS X and Linux) and get a working module. Then I can give tutorials on how to use the latest SQLite3 features without having to coach people through a build and compilation step first.

@simonw

This comment has been minimized.

Show comment
Hide comment
@simonw

simonw Aug 15, 2018

Contributor

http://charlesleifer.com/blog/compiling-sqlite-for-use-with-python-applications/ is fantastic, I hadn't seen that - it's exactly the documentation I was needing. Thanks!

Contributor

simonw commented Aug 15, 2018

http://charlesleifer.com/blog/compiling-sqlite-for-use-with-python-applications/ is fantastic, I hadn't seen that - it's exactly the documentation I was needing. Thanks!

@simonw

This comment has been minimized.

Show comment
Hide comment
@simonw

simonw Aug 15, 2018

Contributor

The documentation for https://github.com/matthew-brett/multibuild looks pretty comprehensive. It includes scripts for building wheels on Travis CI (for OS X and Linux) and AppVeyor (for Windows).

Contributor

simonw commented Aug 15, 2018

The documentation for https://github.com/matthew-brett/multibuild looks pretty comprehensive. It includes scripts for building wheels on Travis CI (for OS X and Linux) and AppVeyor (for Windows).

@simonw

This comment has been minimized.

Show comment
Hide comment
@simonw

simonw Aug 15, 2018

Contributor

I just successfully installed pysqlite3 with 3.25.0 into a fresh virtual environment using the fork at https://github.com/karlb/pysqlite3

$ python3 -mvirtualenv venv
$ source venv/bin/activate
$ pip install git+git://github.com/karlb/pysqlite3
Collecting git+git://github.com/karlb/pysqlite3
...
Installing collected packages: pysqlite3
Successfully installed pysqlite3-0.2.0
$ python
Python 3.6.5 (default, Mar 30 2018, 06:41:53) 
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pysqlite3
>>> pysqlite3.connect(":memory:").execute("select sqlite_version()").fetchall()
[('3.25.0',)]
Contributor

simonw commented Aug 15, 2018

I just successfully installed pysqlite3 with 3.25.0 into a fresh virtual environment using the fork at https://github.com/karlb/pysqlite3

$ python3 -mvirtualenv venv
$ source venv/bin/activate
$ pip install git+git://github.com/karlb/pysqlite3
Collecting git+git://github.com/karlb/pysqlite3
...
Installing collected packages: pysqlite3
Successfully installed pysqlite3-0.2.0
$ python
Python 3.6.5 (default, Mar 30 2018, 06:41:53) 
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pysqlite3
>>> pysqlite3.connect(":memory:").execute("select sqlite_version()").fetchall()
[('3.25.0',)]
@simonw

This comment has been minimized.

Show comment
Hide comment
@simonw
Contributor

simonw commented Aug 16, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment