Memory overallocation crash and keyboard interrupt stops working

[pabstersac]

BPO	27560
Nosy	@vadmium, @ztane, @zhangyangyu
Superseder	bpo-27559: Crash On bytearray()
Files	crash.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-07-18.10:44:06.482>
created_at = <Date 2016-07-18.09:08:54.570>
labels = []
title = 'Memory overallocation crash and keyboard interrupt stops working'
updated_at = <Date 2016-07-18.10:44:06.480>
user = 'https://bugs.python.org/pabstersac'

bugs.python.org fields:

activity = <Date 2016-07-18.10:44:06.480>
actor = 'martin.panter'
assignee = 'none'
closed = True
closed_date = <Date 2016-07-18.10:44:06.482>
closer = 'martin.panter'
components = []
creation = <Date 2016-07-18.09:08:54.570>
creator = 'pabstersac'
dependencies = []
files = ['43774']
hgrepos = []
issue_num = 27560
keywords = []
message_count = 6.0
messages = ['270732', '270738', '270739', '270740', '270741', '270747']
nosy_count = 4.0
nosy_names = ['martin.panter', 'ztane', 'xiang.zhang', 'pabstersac']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = None
status = 'closed'
superseder = '27559'
type = None
url = 'https://bugs.python.org/issue27560'
versions = ['Python 3.5', 'Python 3.6']

[pabstersac]

zlib.compress crashes when you put a lot of stuff into it and even when you try keyboard interrupts it doesn't work for some reason, and Python gets killed because trying to compress so much data gets the Python process get killed by the kernel. If you put a large amount of data in it will get killed quicker. A problem is that keyboard interrupts stop working while the program is running, which is usually around 5 - 15 seconds during which python stops working (including interrupts) until it gets killed by the kernel. This may be used to crash python remotely if a server takes user input and runs it through zlib.compress(b'variable' * 2**32) which is not very likely. You probably should set a limit for what you compress and how much it is.
The output is:
Killed: 9
Even though you try doing keyboard interrupt it doesn't work.

[vadmium]

There is a good chance crashes and related 2 GiB and 4 GiB limits in zlib will be solved by bpo-27130. Do you want to see if the patch improves the situation? (I cannot test it directly because I don’t have enough memory.)

[pabstersac]

I'd like to but I don't have enough memory either, sorry.

[zhangyangyu]

Compressing large data is born to be slow and it's highly possible to cause the kernel halt then due to resource limit.

This may be used to crash python remotely if a server takes user input and runs it through zlib.compress(b'variable' * 2**32) which is not very likely. You probably should set a limit for what you compress and how much it is.

This is the responsibility of application code not library code.

[ztane]

I am pretty sure **it never calls zlib.compress**. I get memory error from that argument alone, on Linux with overcommit memory enabled, 16G ram, swap.

[vadmium]

Yes you are probably right Antti. The input to compress would need 31 TB of memory, and is a text string, while Python 3’s zlib.compress() requires bytes. In other words, this will be the same problem of overallocating memory as bpo-27559.

Pablo: I suggest you carefully look at the code, try to understand what it is doing, eliminating the unnecessary parts, etc. Especially when so many of your reports today involve similar techniques.