place_pages is not idempotent

[joaomlneto]

Calling place_pages with the same arguments results in extra pages being moved into the local node.

We wanted to improve it anyways -- rewrite!!

[joaomlneto]

Found the issue.

Whenever we call mbind() on memory sub-segment, the kernel will split the mapping into two separate ones.

Example

• We have a 100MB segment A that is interleaved (addresses 0x00000 to `0xFFFFF
• We call place_pages(A, 0.25) results in the mapping splitting into two:
  • A1 from 0x00000 to 0xBFFFF with interleaved policy
  • A2 from 0xC0000 to 0xFFFFF with local policy
• When we call place_all_pages(0.50) it will call place_pages on each of the segments.
• This results in place_pages(A1, 0.50) and place_pages(A2, 0.50) which results in the following mappings:
  • A11 from 0x00000 to 0x5FFFF with interleaved policy
  • A12 from 0x60000 to 0xBFFFF with local policy
  • A21 from 0xC0000 to 0xDFFFF with interleaved policy (although memory is already in the local node and the kernel won't move it)
  • A22 from 0xE0000 to 0xFFFFF with local policy

After the final call, which should have placed half of the memory in the local node and half interleaved in the remaining nodes, the actual proportion is higher: in this specific example we get around 62.5% in the local node, and only 37.5% interleaved in the remaining ones.

Even if we call the function twice with a ratio larger than 1/#numNUMAnodes, the proportion of local pages will always increase.

Solution (simple but with a big assumption)

We can either assume the mapping doesn't change (therefore we only read the mappings once and always move the same addresses)

Solution (not so simple but complete)

We need to be aware of the changes in the mapping, and we need to be aware of the memory policies of each, and what sets of sub-mappings represent the initial mappings.

[joaomlneto]

The bug is fixed -- we will just not update the memory mapping once we start the tuning process.

The dynamic mapping feature can be tracked in #11