[Python] [Plasma] Huge pages test failure

[asfimport]

This is a new failure here (Ubuntu 16.04, x86-64):

_____________________________ test_use_huge_pages _____________________________
Traceback (most recent call last):
  File "/home/antoine/arrow/python/pyarrow/tests/test_plasma.py", line 779, in test_use_huge_pages
    create_object(plasma_client, 100000000)
  File "/home/antoine/arrow/python/pyarrow/tests/test_plasma.py", line 80, in create_object
    seal=seal)
  File "/home/antoine/arrow/python/pyarrow/tests/test_plasma.py", line 69, in create_object_with_id
    memory_buffer = client.create(object_id, data_size, metadata)
  File "plasma.pyx", line 302, in pyarrow.plasma.PlasmaClient.create
  File "error.pxi", line 79, in pyarrow.lib.check_status
pyarrow.lib.ArrowIOError: /home/antoine/arrow/cpp/src/plasma/client.cc:192 code: PlasmaReceive(store_conn_, MessageType_PlasmaCreateReply, &buffer)
/home/antoine/arrow/cpp/src/plasma/protocol.cc:46 code: ReadMessage(sock, &type, buffer)
Encountered unexpected EOF
---------------------------- Captured stderr call -----------------------------
Allowing the Plasma store to use up to 0.1GB of memory.
Starting object store with directory /mnt/hugepages and huge page support enabled
create_buffer failed to open file /mnt/hugepages/plasmapSNc0X

Reporter: Antoine Pitrou / @pitrou

_{Note: This issue was originally created as ARROW-2237. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
@pcmoritz

[asfimport]

Philipp Moritz / @pcmoritz:
Was this on Travis or on your local machine?

[asfimport]

Wes McKinney / @wesm:
This looks like a local failure

[asfimport]

Robert Nishihara / @robertnishihara:
Interesting, does /mnt/hugepages exist locally? If not, the test should be skipped. If yes, then maybe there is some permission error or something.

[asfimport]

Philipp Moritz / @pcmoritz:
Which commands did you use to create  /mnt/hugepages? (The test is skipped if it doesn't exist)

I can try to reproduce this on a fresh image, but steps how to reproduce on say an Ubuntu image would be appreciated!

[asfimport]

Philipp Moritz / @pcmoritz:
Creating the /mnt/hugepages with

sudo mkdir -p /mnt/hugepages
 sudo mount -t hugetlbfs -o uid=`id -u` -o gid=`id -g` none /mnt/hugepages
 sudo bash -c "echo `id -g` > /proc/sys/vm/hugetlb_shm_group"
 sudo bash -c "echo 20000 > /proc/sys/vm/nr_hugepages"

I can't reproduce the test failure on Ubuntu.

[asfimport]

Antoine Pitrou / @pitrou:
/mnt/hugepages exists by default here. Though there's something weird: it's /dev/hugepages that's mounted if I understand correctly:

$ mount | \grep hugepages
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)

[asfimport]

Antoine Pitrou / @pitrou:
By the way, setting up 20000 huge pages seems excessive. Here, echo 20000 > /proc/sys/vm/nr_hugepages swaps wildly as it seems to evict existing normal pages to make place for huge pages (am I analyzing this right?). This is using a 4.13.0 kernel on a 16GB RAM machine.

Otherwise, yes, configuring /proc/sys/vm/hugetlb_shm_group and /proc/sys/vm/nr_hugepages and fixing the test to use /dev/hugepages makes it pass here. I wonder if there's a way to skip it or otherwise give better directions if hugepages support isn't usable on the system.

[asfimport]

Antoine Pitrou / @pitrou:
(I understand plasma is using huge pages by creating files directly inside the mount point, instead of using the MAP_HUGETLB flag to mmap()? out of curiosity, is there a rationale for that?)

[asfimport]

Wes McKinney / @wesm:
Is this still occurring ever?

[asfimport]

Antoine Pitrou / @pitrou:
I don't know, it's marked xfail :)

[asfimport]

Antoine Pitrou / @pitrou:
Since noone is actively working on it, and it is pretty minor, closing.