Update SQLite to 3.31.1 in Windows and macOS installer builds

[BigStone]

BPO	38380
Nosy	@pfmoore, @ronaldoussoren, @tjguk, @benjaminp, @ned-deily, @zware, @zooba, @miss-islington, @erlend-aasland
PRs	bpo-38380: Update macOS & Windows builds to SQLite v3.31.1 #18678 [3.8] bpo-38380: Update macOS & Windows builds to SQLite v3.31.1 (GH-18678) #18740 [3.7] bpo-38380: Update macOS & Windows builds to SQLite v3.31.1 (GH-18678) #18741
Files	0001-bpo-38380-Update-macOS-installer-to-use-SQLite-3.30..patch: bpo-38380: Update macOS installer to use SQLite 3.30.0 0002-bpo-38380-Update-Windows-builds-to-use-SQLite-3.30.0.patch: bpo-38380: Update Windows builds to use SQLite 3.30.0 0001-bpo-38380-Update-macOS-installer-to-use-SQLite-3.30.1.patch 0002-bpo-38380-Update-Windows-builds-to-use-SQLite-3.30.1.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-03-04.02:45:36.925>
created_at = <Date 2019-10-05.22:12:20.397>
labels = ['OS-mac', '3.7', '3.8', '3.9', 'build', 'OS-windows']
title = 'Update SQLite to 3.31.1 in Windows and macOS installer builds'
updated_at = <Date 2020-03-04.02:45:36.924>
user = 'https://bugs.python.org/BigStone'

bugs.python.org fields:

activity = <Date 2020-03-04.02:45:36.924>
actor = 'benjamin.peterson'
assignee = 'none'
closed = True
closed_date = <Date 2020-03-04.02:45:36.925>
closer = 'benjamin.peterson'
components = ['Build', 'macOS', 'Windows']
creation = <Date 2019-10-05.22:12:20.397>
creator = 'Big Stone'
dependencies = []
files = ['48651', '48652', '48921', '48922']
hgrepos = []
issue_num = 38380
keywords = ['patch']
message_count = 30.0
messages = ['354023', '354025', '354280', '354281', '354282', '354284', '354300', '362803', '362923', '362928', '362929', '362930', '362933', '362935', '362938', '362963', '362970', '362973', '363144', '363156', '363158', '363160', '363161', '363162', '363166', '363178', '363179', '363180', '363182', '363323']
nosy_count = 10.0
nosy_names = ['paul.moore', 'ronaldoussoren', 'tim.golden', 'benjamin.peterson', 'ned.deily', 'zach.ware', 'steve.dower', 'Big Stone', 'miss-islington', 'erlendaasland']
pr_nums = ['18678', '18740', '18741']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue38380'
versions = ['Python 2.7', 'Python 3.7', 'Python 3.8', 'Python 3.9']

[BigStone]

there is a security fix in sqlite-3.30

https://nvd.nist.gov/vuln/detail/CVE-2019-16168#VulnChangeHistorySection

https://www.sqlite.org/releaselog/3_30_0.html

[ned-deily]

I verified it is exploitable via the sqlite3 module by adapting the test case from the SQLite ticket (https://www.sqlite.org/src/info/e4598ecbdd18bd82). But since it requires the exploiter to be able to specify raw SQL statements, it doesn't sound like it needs to be treated as a Python security issue. We should plan to to update to the latest SQLite but it doesn't need to be a release blocker.

[erlend-aasland]

I've prepared a PR for https://github.com/python/cpython-source-deps at https://github.com/erlend-aasland/cpython-source-deps/tree/upgrade-sqlite.

Patches for Windows and macOS installer builds on 3.9 prepared at https://github.com/erlend-aasland/cpython/tree/fix-issue-38380.

If it looks ok, I'll create PR's for CPython and cpython-source-deps. I guess this should be backported to 3.8, 3.7 and 2.7.

[erlend-aasland]

bpo-38380: Update macOS installer to use SQLite 3.30.0
erlend-aasland@aa7d7b1

[erlend-aasland]

bpo-38380: Update Windows builds to use SQLite 3.30.0
erlend-aasland@e25214e

[erlend-aasland]

FYI: Compiled cpython 3.9 with sqlite-3.30 on macOS 10.14.6. Make test completes without errors.

[erlend-aasland]

Update: Tested on macOS 10.14.6 with make test on 2.7.17rc1, 3.7.5rc1+, 3.8.0rc1+.

[erlend-aasland]

I've updated these patches to sqlite3-3.30.1 (https://www.sqlite.org/releaselog/3_30_1.html).

cpython-source-deps is updated (python/cpython-source-deps#17).

CPython commits are updated and rebased onto current 3.9 master: https://github.com/erlend-aasland/cpython/tree/fix-issue-38380

(Since Oct 2019, SQLite 3.31 have been released, but that would probably require opening a new issue.)

[zooba]

We still need the tag added to the cpython-source-deps repo, and I still can't complete a clone right now (something is strange with SSL to GitHub on my (temporary) internet connection).

Zach - can you tag it? python/cpython-source-deps#17

[erlend-aasland]

Yes, we need the tag for the Windows build, so the PR currently fails the Windows checks. (Tagging must be done explicitly by the maintainers, IIRC.) I also forgot to add a NEWS entry, so I'll do another push to add those (and kick off the CI) when the tag arrives.

[ned-deily]

Thanks for the PRs. If we're going to update now as we should, why not to 3.31.1 which is current?

[erlend-aasland]

You're welcome. If you are ok with that, I'd be happy to prepare a PR for the source deps for sqlite3 v3.31.1, and update #62878 as soon as it is tagged.

[ned-deily]

I would prefer to go to 3.31.1 at this point particularly given the track record of the SQLite project. It's been released for a month now. Any objections, Steve?

[erlend-aasland]

I agree. I've updated the branches for source deps and cpython. I'll wait for Steve's approval before I open a new PR over at cpython-source-deps and update #62878.

[zooba]

Isn't that what we ended up merging? (Goes to check). Ah, that was 3.30.1.

Sure, go for it. We'll have RCs of everything before the next final releases go out, so provided someone double checks that it's all good before then I'm okay with it.

[erlend-aasland]

Great, thanks folks! I've pushed an update to #62878.

(BTW, is it kosher to force push to PRs like this?)

[zooba]

(BTW, is it kosher to force push to PRs like this?)

We always squash merge, so you don't need to do it to hide your history. Personally, I do it occasionally when the history is very long and nobody has started reviewing.

Force pushing will reset any reviews, so you'll be forcing reviewers to start again instead of just looking at your updates. Some reviewers (e.g. me) will often defer a full rereview if we don't have a lot of time and someone just force pushed. So in general, avoid it.

[erlend-aasland]

Understood; I’ll remember it for future PRs. Thanks for clarifying.

[erlend-aasland]

Would you prefer that I close GH_18678 and split it up in separate RPs for macOS and Windows?

[zooba]

No, that one is fine.

I just need someone else to jump in and tag the commit in cpython-source-deps for me (because I can't clone it successfully on my current internet connection).

[erlend-aasland]

Ah, but you're all set @steve.dower, someone at cpython-source-deps already pulled my tags: https://github.com/python/cpython-source-deps/tags :)

But I guess that if I were to do the same maneouver again, it would probably be preferable if I made one PR for macOS and another for Windows, right?

[zooba]

Nope, same PR and bug is totally fine.

Putting the NEWS items in their platform-specific sections is how we normally do it (though to be fair, I had to go look it up).

[erlend-aasland]

Ok, thanks for your patience and time. I'll fix the NEWS entries right away.

[zooba]

I marked the PR to backport to 3.7 and 3.8.

Up to Benjamin whether 2.7 gets it, but unless there's a specific and impactful CVE that's been fixed, I doubt it (the one linked at the start of this issue seems to require direct modification of the SQL statement, which would be a bug in itself if permitted, so I think it's outside of our threat model for CPython).

[miss-islington]

New changeset 1382c32 by Erlend Egeberg Aasland in branch 'master':
bpo-38380: Update macOS & Windows builds to SQLite v3.31.1 (GH-18678)
1382c32

[miss-islington]

New changeset 7ad9982 by Miss Islington (bot) in branch '3.8':
bpo-38380: Update macOS & Windows builds to SQLite v3.31.1 (GH-18678)
7ad9982

[miss-islington]

New changeset 7ca251b by Miss Islington (bot) in branch '3.7':
bpo-38380: Update macOS & Windows builds to SQLite v3.31.1 (GH-18678)
7ca251b

[zooba]

Leaving open until someone (cough Benjamin) confirms whether this has to go into 2.7 or not.

[erlend-aasland]

Cool. Should I remove Python-3.* from the Versions label, so it doesn't show up in searches for 3.* issues?

[benjaminp]

Traditionally, we've taken these dependency updates in 2.7, but there doesn't seem to be a compelling reason to take this one.