raw debug allocators to not return malloc alignment

[jtaylor]

BPO	30150
Nosy	@vstinner, @methane, @mapreri

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-04-23.22:01:08.321>
labels = ['interpreter-core', '3.7']
title = 'raw debug allocators to not return malloc alignment'
updated_at = <Date 2017-05-24.07:31:37.509>
user = 'https://bugs.python.org/jtaylor'

bugs.python.org fields:

activity = <Date 2017-05-24.07:31:37.509>
actor = 'jtaylor'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2017-04-23.22:01:08.321>
creator = 'jtaylor'
dependencies = []
files = []
hgrepos = []
issue_num = 30150
keywords = []
message_count = 7.0
messages = ['292187', '292256', '292257', '292258', '294243', '294314', '294333']
nosy_count = 4.0
nosy_names = ['vstinner', 'methane', 'jtaylor', 'mapreri']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue30150'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7']

[jtaylor]

The debug raw allocator do not return the same alignment as malloc. See _PyMem_DebugRawAlloc:
https://github.com/python/cpython/blob/master/Objects/obmalloc.c#L1873

The line
return p + 2*SST

adds 2 * sizeof(size_t) to the pointer returned by malloc.
On for example x32 malloc returns 16 byte aligned memory but size_t is 4 bytes.
This makes all memory returned by the debug allocators not aligned the what the system assumes on such platforms.

[methane]

How it cause problem?
I think you should use malloc() instead of PyMem_Malloc() or other Python memory allocator when you need strict malloc() alignment.

[vstinner]

On for example x32 malloc returns 16 byte aligned memory but size_t is 4 bytes.

x32 is a strange platform :-( Does numpy support it? I'm not sure that Python works on such platform.

I suggest to hardcode 16 or 32 bytes in _PyMem_DebugRawAlloc instead of relying on sizeof(size_t). pymalloc aligns memory allocations to 8 bytes if I recall correctly.

How it cause problem?

numpy uses SIMD instructions which require strict memory alignement.

Note: There was also an issue bpo-18835 to "Add aligned memory variants to the suite of PyMem functions/macros", but it was rejected.

[vstinner]

Is this issue related to this numpy issue: "ENH: add support for python3.6 memory tracing"?
numpy/numpy#8885

[jtaylor]

no in numpy it is just a case of using the wrong allocator in a certain spot, an issue that can be fixed in numpy.
But it is also minor bug/documentation issue in Python itself.

Alignment isn't very important for SIMD any more but there are architectures where alignment is still mandatory so numpy is sprinkled with asserts checking alignment which triggered on x32.
It is a very minor issue as to my knowledge none of the platforms with alignment requirement has the properties of x32 and x32 doesn't actually care about alignment either.

[vstinner]

Maybe we can use Py_MAX (sizeof (size_t), 8) for SST? If I recall correctly, pymalloc uses 8 bytes for the alignement.

Somewhere I read that malloc uses sizeof (double), the largest C type.

Well, what do you suggest Julian? Do you want to write a PR?

[jtaylor]

The largest type is usually the long double. Its alignment ranges from 4 bytes (i386) to 16 bytes (sparc).
So Py_MAX (sizeof (size_t), 8) should indeed do it.