PLFS with N-1 opens perform horribly unless DVS_MAXNODES=1

[brettkettering]

This is a problem we've noticed on Cray XE-6 systems with a lanes-based parallel file system by Panasas.

I used fs_test to model the 512 pe N-1 job Jeff was doing. I ran 512 pes where each wrote and read 16864 bytes per write/read 934 times. This results in a total file size of 8,064,499,712 bytes (7.51 GB).

Cielo, N-1, MPI/IO w/ PLFS via ADIO ("plfs:/"), DVS_MAXNODES=9:

12/21/11, 19:44:42
Write Eff Bandwidth: 21.9 MB/s
Write Raw Bandwidth: 3490 MB/s
Write Avg Open Time: 347 sec.
Write Avg Close Time: 0.692 sec.

Read Eff Bandwidth: 1840 MB/s
Read Raw Bandwidth: 4150 MB/s
Read Avg Open Time: 2.21 sec.
Read Avg Close Time: 0.0663 sec.

Cielo, N-1, MPI/IO (350 wide, 4 MB), DVS_MAXNODES=9:

12/21/11, 15:41:29
Write Eff Bandwidth: 240 MB/s
Write Raw Bandwidth: 245 MB/s
Write Avg Open Time: 0.381 sec.
Write Avg Close Time: 0.122 sec.

Read Eff Bandwidth: 1460 MB/s
Read Raw Bandwidth: 1650 MB/s
Read Avg Open Time: 0.548 sec.
Read Avg Close Time: 0.0519 sec.

Cielo, N-1, MPI/IO w/ PLFS via ADIO ("plfs:/"), DVS_MAXNODES=1:

12/22/11, 10:16:29
Write Eff Bandwidth: 1370 MB/s
Write Raw Bandwidth: 3720 MB/s
Write Avg Open Time: 2.73 sec.
Write Avg Close Time: 0.473 sec.

Read Eff Bandwidth: 2330 MB/s
Read Raw Bandwidth: 3980 MB/s
Read Avg Open Time: 1.28 sec.
Read Avg Close Time: 0.065 sec.

Observations:

1. Using PLFS with DVS_MAXNODES=9 showed horrible write performance, primarily due to the extreme overhead doing all the opens, most of which are not necessary.

2. Using PLFS and restricting DVS_MAXNODES to 1 made a huge difference in lowering the file open overhead and produced much better effective bandwidth. While there is still significant overhead due to the number of files being used versus the very small amount of data, it indicates that we might see an improvement on the real problem that would make the performance attractive to users.

3. A straight N-1 MPI/IO job striped widely, as true N-1 jobs should be (note PLFS turns the N-1 into N-N so we don't want PLFS jobs striped widely), was still much lower bandwidth than Jeff's BulkIO.

Then, I wanted to see what would happen if we repeated this experiment with an N-N I/O model instead of N-1. Since PLFS converts N-1 to N-N we didn't expect to see any difference, but we did.

Cielo, N-N, MPI/IO w/ PLFS via ADIO ("plfs:/"), DVS_MAXNODES=9:

1/3/12
Write Eff Bandwidth: 2490 MB/s
Write Raw Bandwidth: 3730 MB/s
Write Avg Open Time: 0.425 sec.
Write Avg Close Time: 0.139 sec.

Read Eff Bandwidth: 3680 MB/s
Read Raw Bandwidth: 3950 MB/s
Read Avg Open Time: 0.0657 sec.
Read Avg Close Time: 0.0266 sec.

Cielo, N-N, MPI/IO w/ PLFS via ADIO ("plfs:/"), DVS_MAXNODES=1:

1/3/12
Write Eff Bandwidth: 2550 MB/s
Write Raw Bandwidth: 3770 MB/s
Write Avg Open Time: 0.41 sec.
Write Avg Close Time: 0.152 sec.

Read Eff Bandwidth: 3660 MB/s
Read Raw Bandwidth: 3920 MB/s
Read Avg Open Time: 0.0652 sec.
Read Avg Close Time: 0.0277 sec.

Cielo, N-N, MPI/IO (2 wide, 4 MB), DVS_MAXNODES=9:

1/3/12
Write Eff Bandwidth: 2310 MB/s
Write Raw Bandwidth: 3810 MB/s
Write Avg Open Time: 0.513 sec.
Write Avg Close Time: 0.159 sec.

Read Eff Bandwidth: 3310 MB/s
Read Raw Bandwidth: 3920 MB/s
Read Avg Open Time: 0.235 sec.
Read Avg Close Time: 0.004 sec.

Cielo, N-N, MPI/IO (2 wide, 4 MB), DVS_MAXNODES=1:

1/3/12
Write Eff Bandwidth: 2420 MB/s
Write Raw Bandwidth: 3740 MB/s
Write Avg Open Time: 0.469 sec.
Write Avg Close Time: 0.158 sec.

Read Eff Bandwidth: 3370 MB/s
Read Raw Bandwidth: 3980 MB/s
Read Avg Open Time: 0.225 sec.
Read Avg Close Time: 0.00405 sec.

Observations:

1. Using DVS_MAXNODES=1 improved effective write bandwidth and open time, but for the numbers are so close it's hard to say much about it at this level. It didn't do anything for reads.

2. PLFS improved the write and read performance by distributing the metadata load (presumably) and lowering the open overhead.

3. There was not huge open time penalty, like with N-1.

So, the question is, what is going on in opens with PLFS and N-1 that is not going on with PLFS and N-N?

[brettkettering]

This might be another good candidate to work with Aaron Torres as an opportunity for Aaron to become more familiar with the PLFS code base.

[brettkettering]

We are replacing the DVS I/O Forwarding Panasas file system with a non-I/O forwarding Lustre file system. This is no longer relevant.