prov/gni: fix a problem with progress

[hppritcha]

Turns out that the VC refactor had another impact,
it causes all data progress to be delayed until
the app starts calling fi_cq_read, etc. This
brings out all kind of issues with one-sided
program models.

Fix verified by SOS developers.

Signed-off-by: Howard Pritchard howardp@lanl.gov

[sungeunchoi]

Is this for manual or our pseudo-auto progress?

[chuckfossen]

Are you saying that we used to progress before the vc refactor even in manual progress mode? Or is this auto progress? What happens to the SOS app? Does it fail before it starts calling cq read?

[hppritcha]

I've attached the SOS app. Note it uses extensions to OpenSHMEM but you can see what its trying to do, namely post a whole bunch of non blocking AMOs, then go into a shmem quiet.

#include <shmem.h>
#include <shmemx.h>
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include <sys/time.h>
#include <time.h>

double get_wtime(void)
{
    double wtime = 0.0;

#ifdef CLOCK_MONOTONIC
    struct timespec tv;
    clock_gettime(CLOCK_MONOTONIC, &tv);
    wtime = tv.tv_sec * 1e6;
    wtime += (double)tv.tv_nsec / 1000.0;
#else
    struct timeval tv;
    gettimeofday(&tv, NULL);
    wtime = tv.tv_sec * 1e6;
    wtime += (double)tv.tv_usec;
#endif
    return wtime;
}

int main(int argc, char **argv) {
    if (argc != 2) {
        fprintf(stderr, "usage: %s natomics-per-pe\n", argv[0]);
        return 1;
    }

    const int natomics_per_pe = atoi(argv[1]);

    shmem_init();

    const int target_pe = (shmem_my_pe() + 1) % shmem_n_pes();
    unsigned long long *mem = (unsigned long long *)shmem_malloc(sizeof(*mem));
    assert(mem);

    double start_time, mid_time, start_quiet_time, end_time;

    // Warm up
    start_time = get_wtime();
    for (int i = 0; i < natomics_per_pe; i++) {
        shmemx_ulonglong_atomic_or(mem, 0xff, target_pe);
    }
    mid_time = get_wtime();
    shmem_quiet();
    end_time = get_wtime();


    start_time = get_wtime();
    for (int i = 0; i < natomics_per_pe; i++) {
        shmemx_ulonglong_atomic_or(mem, 0xff, target_pe);
    }
    mid_time = get_wtime();
    while (get_wtime() - mid_time < 2.0) ;
    start_quiet_time = get_wtime();
    shmem_quiet();
    end_time = get_wtime();
    printf("PE %d took %f seconds to issue, slept for %f seconds, %f seconds "
            "to complete\n", shmem_my_pe(), mid_time - start_time,
            start_quiet_time - mid_time, end_time - start_quiet_time);

    shmem_barrier_all();

    if (shmem_my_pe() == 0) {
        printf("\n");
    }
    shmem_barrier_all();

    start_time = get_wtime();
    for (int i = 0; i < natomics_per_pe; i++) {
        shmemx_ulonglong_atomic_or(mem, 0xff, target_pe);
    }
    mid_time = get_wtime();
    shmem_quiet();
    end_time = get_wtime();
    printf("PE %d took %f seconds to issue, did not sleep, %f seconds "
            "to complete\n", shmem_my_pe(), mid_time - start_time,
            end_time - mid_time);

    shmem_finalize();

    return 0;
}

sung: this is for either progress model.

If one tries to post many millions of atomic ops, OOM's are observed.

[chuckfossen]

Normally the progress happens in a separate thread, true? Is it ok to call gnix_nic_progress in the sender's thread?

[hppritcha]

no progress for small atomics doesn't happen from the progress thread unless you turn on zti's irq for every op option - then you get high latency and a nose-dive in message throughput rates.

[chuckfossen]

So, I'm hearing this won't affect latency.

[chuckfossen]

Should we have a similar addition to _gnix_sendv as well?

[hppritcha]

I'll add one for _gnix_sendv too. good catch.

[hppritcha]

add similar code in sendv code.