Update SQLite to 3.28 in Windows and macOS installer builds

[vstinner]

BPO	35360
Nosy	@pfmoore, @ronaldoussoren, @tjguk, @ned-deily, @ceronman, @berkerpeksag, @zware, @zooba, @animalize, @Mariatta, @miss-islington
PRs	bpo-35360: Update Windows builds to use SQLite 3.28.0 #14179 bpo-35360: Update macOS installer to use SQLite 3.28.0 #14180 [2.7] bpo-35360: Update Windows builds to use SQLite 3.28.0 #14182 [2.7] bpo-35360: Update macOS installer to use SQLite 3.28.0 #14183 [3.8] bpo-35360: Update macOS installer to use SQLite 3.28.0 (GH-14180) #14195 [3.7] bpo-35360: Update macOS installer to use SQLite 3.28.0 (GH-14180) #14196 [3.8] bpo-35360: Update Windows builds to use SQLite 3.28.0 (GH-14179) #14352 [3.7] bpo-35360: Update Windows builds to use SQLite 3.28.0 (GH-14179) #14353 [2.7] bpo-35360: Update Windows builds to use SQLite 3.28.0 (GH-14179) #14354

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-06-24.23:44:30.521>
created_at = <Date 2018-11-30.12:14:30.437>
labels = ['OS-mac', '3.8', 'build', '3.7', 'OS-windows']
title = 'Update SQLite to 3.28 in Windows and macOS installer builds'
updated_at = <Date 2019-09-30.14:52:45.917>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2019-09-30.14:52:45.917>
actor = 'steve.dower'
assignee = 'none'
closed = True
closed_date = <Date 2019-06-24.23:44:30.521>
closer = 'steve.dower'
components = ['Build', 'macOS', 'Windows']
creation = <Date 2018-11-30.12:14:30.437>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 35360
keywords = ['patch']
message_count = 25.0
messages = ['330779', '330783', '331947', '332014', '332261', '340978', '343456', '343531', '345708', '345865', '345890', '345901', '345939', '345941', '345970', '345974', '345977', '345979', '346448', '346449', '346453', '346455', '347161', '353441', '353583']
nosy_count = 14.0
nosy_names = ['paul.moore', 'ghaering', 'ronaldoussoren', 'tim.golden', 'ned.deily', 'ceronman', 'berker.peksag', 'zach.ware', 'steve.dower', 'malin', 'Big Stone', 'Mariatta', 'miss-islington', 'Scott Stevens']
pr_nums = ['14179', '14180', '14182', '14183', '14195', '14196', '14352', '14353', '14354']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue35360'
versions = ['Python 2.7', 'Python 3.7', 'Python 3.8']

[vstinner]

Windows and macOS installers require SQLite, but they require different versions! Windows uses 3.21 or 3.14, but macOS uses 3.22.

I'm talking about the following line in PCbuild\get_externals.bat:

set libraries=%libraries% sqlite-3.21.0.0

• 3.6, 3.7 and master branches:

SQLite[Windows]: 3.21.0.0
SQLite[macOS]: 3.22.0

• 2.7 branch:

SQLite[Windows]: 3.14.2.0
SQLite[macOS]: 3.22.0

Note: I wrote a script to get external dependencies:

https://github.com/vstinner/misc/blob/master/cpython/external_versions.py

[vstinner]

First the sqlite branch should be updated in:
https://github.com/python/cpython-source-deps

Then a new tag should be created in this repository.

I tried to update cpython-source-deps, but "git push" never completed. I will retry next week ;-)

[ScottStevens]

With the discovery of the SQLite "Magellan" bug, could the version be upgraded to 3.26 for all Python versions? As far as I know, the security case is restricted to where the user is allowing aribitrary SQL execution without arbitrary Python execution, but in that case I do believe remote code execution is possible.

https://blade.tencent.com/magellan/index_en.html

[zooba]

SQLite updates and changes for us almost always get stuck on someone being willing to verify that nothing has broken (and stand by their analysis).

Without an active expert (I just nosied ghaering in case they are around), I'm not confident to make this change in any version prior to 3.8.

[ned-deily]

We should look at doing this for the next set of maintenance updates. This doesn't appear to be critical enough to delay current releases unless someone can show how this exploit could be used in a typical Python application.

[BigStone]

sqlite-3.28.0 now available, with extended window functio support: EXCLUDE clause, GROUPS frame types, window chaining, and support for "<expr> PRECEDING" and "<expr> FOLLOWING" boundaries in RANGE frames.

[BigStone]

any hope to have a SQLite refresh in Python-3.8.0b1 for Windows/Mac ?

It's generally the ideal / less annoying moment to do so.

[animalize]

@Mariatta Wijaya, would you update SQLite?

I want to do it myself, by following your patch in bpo-28791.
But I find I have to commit SQLite's source code to https://github.com/python/cpython-source-deps, so I think this should be done by a core developer.

[BigStone]

any hope for beta2 ?

[zooba]

Has anyone tried building with the newer SQLite and confirmed that the tests and any scenarios they use still work?

We don't currently have a SQLite expert to do these kinds of things.

[BigStone]

on my test cases, sqlite_bro and baresql, db.py , it works without problems.

[zooba]

I just pushed sources with tag sqlite-3.28.0.0, so now anybody should be able to submit a CPython PR for it.

[animalize]

PR 14179 is for Windows build
PR 14180 is for Mac OS X build

Both update to Sqlite 3.28.0

[animalize]

2.7 branch:

PR 14182 is for Windows build
PR 14183 is for Mac OS X build

[ned-deily]

New changeset d8f336f by Ned Deily (animalize) in branch 'master':
bpo-35360: Update macOS installer to use SQLite 3.28.0 (GH-14180)
d8f336f

[ned-deily]

New changeset 373dace by Ned Deily (animalize) in branch '2.7':
[2.7] bpo-35360: Update macOS installer to use SQLite 3.28.0 (GH-14183)
373dace

[miss-islington]

New changeset a7072ff by Miss Islington (bot) in branch '3.8':
bpo-35360: Update macOS installer to use SQLite 3.28.0 (GH-14180)
a7072ff

[miss-islington]

New changeset 624c9a2 by Miss Islington (bot) in branch '3.7':
bpo-35360: Update macOS installer to use SQLite 3.28.0 (GH-14180)
624c9a2

[zooba]

New changeset 7fd2ba3 by Steve Dower (animalize) in branch 'master':
bpo-35360: Update Windows builds to use SQLite 3.28.0 (GH-14179)
7fd2ba3

[zooba]

New changeset 0fc14b3 by Steve Dower (animalize) in branch '2.7':
bpo-35360: Update Windows builds to use SQLite 3.28.0 (GH-14182)
0fc14b3

[miss-islington]

New changeset dad8f79 by Miss Islington (bot) in branch '3.8':
bpo-35360: Update Windows builds to use SQLite 3.28.0 (GH-14179)
dad8f79

[miss-islington]

New changeset 14c179f by Miss Islington (bot) in branch '3.7':
bpo-35360: Update Windows builds to use SQLite 3.28.0 (GH-14179)
14c179f

[ned-deily]

New changeset c58fc3a by Ned Deily (Miss Islington (bot)) in branch '3.7':
bpo-35360: Update Windows builds to use SQLite 3.28.0 (GH-14179)
c58fc3a

[BigStone]

there will be a security fix in sqlite-3.30 around October 10th.

https://nvd.nist.gov/vuln/detail/CVE-2019-16168#VulnChangeHistorySection

https://www.sqlite.org/draft/releaselog/3_30_0.html

[zooba]

Please file a new issue