Not able to set stripe and stripe sie using mpi_info_set #10132

monsieuralok · 2022-03-17T09:14:13Z

We are using PnetCDF in CESM and usually, it sets automatic stripes on files using following call:
call mpi_info_set(info,"striping_factor",stripestr,ierr)
call mpi_info_set(info,"striping_unit",stripestr2,ierr)

while using cray mpich and intel MPI we are not having any problem. We are having a lustre file system and it is OpenMPI 4.0.3 compiled using intel/2020a compiler.
Please could you look over it.
Thanks a lot Alok

ggouaillardet · 2022-03-17T10:05:24Z

It is possible striping_factor and striping_unit keys are ignored by our io/ompio component (but used by the io/romio... component.

Are striping parameters used if you

mpirun --mca io ^ompio ...

monsieuralok · 2022-03-17T10:14:15Z

Hi,
Thanks a lot for reply and I have tried now mpirun -np 4 --mca io ^ompio ./a.out still stripe counts are not set. I guess on our system they have only installed ompio not romio as when I tried:
mpirun -np 4 --mca io romio ./a.out
I get message "A requested component was not found, or was unable to be opened. This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded). Note that
Open MPI stopped checking at the first component that it did not find."
Thanks a lot,
I will ask my sysadmin to install it.

ggouaillardet · 2022-03-17T10:33:45Z

With Open MPI 4.0.3, the ROMIO component is io/romio321, so you should try

mpirun -np 4 --mca io romio321 ./a.out

monsieuralok · 2022-03-17T10:39:08Z

mpirun -np 4 --mca io romio321 ./a.out
node 0 : Hello world
node 2 : Hello world
node 3 : Hello world
node 1 : Hello world
[agu002@login-2.BETZY /cluster/work/users/agu002/pnetcdf]$ lfs getstripe test.nc
test.nc
lmm_stripe_count: 1
lmm_stripe_size: 1048576
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: 18
lmm_pool: projects
obdidx objid objid group
18 80733022 0x4cfe35e 0x6c0000402

still, it does not set the stripe size and stripe count which should be 8. Below are the results from IntelMPI:
lmm_stripe_count: 8
lmm_stripe_size: 2097152
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: 18
lmm_pool: projects
obdidx objid objid group
18 80733271 0x4cfe457 0x6c0000402
16 82127405 0x4e52a2d 0x640000402
15 82447408 0x4ea0c30 0x840000402
17 81287868 0x4d85abc 0x800000402
13 86116640 0x5220920 0x7c0000402
14 82058460 0x4e41cdc 0x680000402
12 80181979 0x4c77adb 0x880000402
22 83375768 0x4f83698 0x600000402

monsieuralok · 2022-03-17T10:55:51Z

Here is my test program in fortran:

      program stripe

include <pnetcdf.inc>
include <mpif.h>
integer4 ,save :: info=MPI_INFO_NULL
character(len=3) :: stripestr
character(len=9) :: stripestr2
integer ierr,status,ncid,size,rank
character(8) :: fnm
fnm="test.nc"
write(stripestr,('(i3)')) 8
write(stripestr2,('(i9)')) 20481024
call MPI_INIT(ierr)
call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierr)
call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr)
print, 'node', rank, ': Hello world'
call mpi_info_create(info,ierr)
call mpi_info_set(info,'romio_ds_read','disable',ierr)
call mpi_info_set(info,'romio_ds_write','disable',ierr)
call mpi_info_set(info,'striping_factor',stripestr,ierr)
call mpi_info_set(info,'striping_unit',stripestr2,ierr)
ierr = nfmpi_create(MPI_COMM_WORLD,fnm,
& IOR(nf_clobber,nf_64bit_offset),INFO,ncid)
CALL MPI_INFO_FREE(INFO, IERR)
ierr=nfmpi_close(ncid)
call MPI_FINALIZE(ierr)
stop
end

ggouaillardet · 2022-03-17T11:24:51Z

I briefly checked the io/romio321 component, and it should handle striping_factor correctly.

A possible explanation is lustre support was not detected at configure time.

What if you replace test.nc with lustre:test.nc?
it should force ROMIO to use the ad_lustre component and fail if it is not builtin.

monsieuralok · 2022-03-17T11:33:26Z

I replaced
fnm="test.nc" in program with fnm="lustre:test.nc"
and I get following error:
MPI error (MPI_File_open) : MPI_ERR_IO: input/output error

so you want to say that OpenMPI is not built/configured with lustre support. I will write to our support. Is there any other way to check it? Thanks a lot Alok

ggouaillardet · 2022-03-17T12:34:22Z

Right, the message means that the ROMIO component has no lustre support.

I am not sure how to verify Lustre support was built from the install tree, so better check the output of configure (or ompi/mca/io/romio321/romio/config.log if the admins kept it)

edgargabriel · 2022-03-17T12:44:12Z

just to clarify briefly an item, ompio does recognize on luster file systems the info objects "stripe_size", and "stripe_width"(We might have to adjust the names of the info objects to match the MPI spec, I actually missed this aspect).

However, in the 4.0.x series romio is the default io component on Lustre file systems, that changed starting from the 4.1.x release series. And second, if I recall my tests correctly, setting stripe size and stripe width only works if the file does not exist at the time file opening, lustre will not allow to change the settings of an existing, non-empty file.

ericch1 · 2022-11-23T11:42:55Z

Hi,
(I post here, since it seems I can't post to discussion list right now???)

In 2012 we wrote and tested our functions to use MPI I/O to have good performances while doing I/O on a Lustre filesystem. Everything was fine about "striping_factor" we passed to file creation.

Now I am trying to verify some performance degradation we observed and I am surprised because it looks like I am unable to create a new file with a given "striping_factor" with any mpi flavor.

I attached a simple example for file creation with hints, and tried it the following way with OpenMPI:

OpenMPI-4.0.3:

which mpicc
/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/openmpi/4.0.3/bin/mpicc

mpicc -o s simple_file_create_with_hint.c

rm -f foo && mpiexec -n 1 --mca io romio321 ./s foo && lfs getstripe foo

Creating the file by MPI_file_open : foo
Informations on file:
Key is 'striping_factor' and worth: '2'
 ...closing the file foo
Informations on file:
foo
lmm_stripe_count:  1
lmm_stripe_size:   1048576
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 49
       obdidx           objid           objid           group
           49       485863591     0x1cf5b0a7                0

Not forcing romio321:

rm -f foo && mpiexec -n 1 ./s foo && lfs getstripe foo  
               
Creating the file by MPI_file_open : foo
Informations on file:
Key is 'striping_factor' and worth: '2'
 ...closing the file foo
Informations on file:
foo
lmm_stripe_count:  1
lmm_stripe_size:   1048576
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 19
       obdidx           objid           objid           group
           19       482813449     0x1cc72609                0

First, as you can see, even if I ask for a striping_factor of 2, I only get one! I tried to write some data too, but it changed nothing...

ompi_info seems to tell lustre component is there (fs-lustre):

ompi_info |grep lustre
  Configure command line: '--prefix=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/openmpi/4.0.3' '--build=x86_64-pc-linux-gnu' '--host=x86_64-pc-linux-gnu' '--enable-shared' '--with-verbs' '--with-hwloc=external' '--without-usnic' '--disable-wrapper-runpath' '--disable-wrapper-rpath' '--with-munge' '--with-slurm' '--with-pmi=/opt/software/slurm' '--enable-mpi-cxx' '--with-hcoll' '--disable-show-load-errors-by-default' '--enable-mpi1-compatibility' '--enable-mca-dso=common-ucx,common-verbs,event-external,atomic-ucx,btl-openib,btl-uct,coll-hcoll,ess-tm,fs-lustre,mtl-ofi,mtl-psm,mtl-psm2,osc-ucx,plm-tm,pmix-ext3x,pmix-s1,pmix-s2,pml-ucx,pnet-opa,psec-munge,ras-tm,spml-ucx,sshmem-ucx,hwloc-external' '--enable-mpirun-prefix-by-default' '--with-ofi=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/libfabric/1.10.1' '--with-ucx=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/ucx/1.8.0'
                  MCA fs: lustre (MCA v2.1.0, API v2.0.0, Component v4.0.3)

Where am I wrong?

Second, I was expecting that when I re-open the file for read-only, I would have some information in "MPI_Info" but it is empty... is that normal?

For example, using mpich-3.2.1 I have the following output:

which mpicc
/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/gcc7.3/mpich/3.2.1/bin/mpicc

mpicc -o s simple_file_create_with_hint.c             

                 
rm -f foo && mpiexec -n 1 ./s foo && lfs getstripe foo 

Creating the file by MPI_file_open : foo
Informations on file:
Key is 'cb_buffer_size' and worth: '16777216'
Key is 'romio_cb_read' and worth: 'automatic'
Key is 'romio_cb_write' and worth: 'automatic'
Key is 'cb_nodes' and worth: '1'
Key is 'romio_no_indep_rw' and worth: 'false'
Key is 'romio_cb_pfr' and worth: 'disable'
Key is 'romio_cb_fr_types' and worth: 'aar'
Key is 'romio_cb_fr_alignment' and worth: '1'
Key is 'romio_cb_ds_threshold' and worth: '0'
Key is 'romio_cb_alltoall' and worth: 'automatic'
Key is 'ind_rd_buffer_size' and worth: '4194304'
Key is 'ind_wr_buffer_size' and worth: '524288'
Key is 'romio_ds_read' and worth: 'automatic'
Key is 'romio_ds_write' and worth: 'automatic'
Key is 'cb_config_list' and worth: '*:1'
Key is 'romio_filesystem_type' and worth: 'UFS: Generic ROMIO driver for all UNIX-like file systems'
Key is 'romio_aggregator_list' and worth: '0 '
 ...closing the file foo
Informations on file:
Key is 'cb_buffer_size' and worth: '16777216'
Key is 'romio_cb_read' and worth: 'automatic'
Key is 'romio_cb_write' and worth: 'automatic'
Key is 'cb_nodes' and worth: '1'
Key is 'romio_no_indep_rw' and worth: 'false'
Key is 'romio_cb_pfr' and worth: 'disable'
Key is 'romio_cb_fr_types' and worth: 'aar'
Key is 'romio_cb_fr_alignment' and worth: '1'
Key is 'romio_cb_ds_threshold' and worth: '0'
Key is 'romio_cb_alltoall' and worth: 'automatic'
Key is 'ind_rd_buffer_size' and worth: '4194304'
Key is 'ind_wr_buffer_size' and worth: '524288'
Key is 'romio_ds_read' and worth: 'automatic'
Key is 'romio_ds_write' and worth: 'automatic'
Key is 'cb_config_list' and worth: '*:1'
Key is 'romio_filesystem_type' and worth: 'UFS: Generic ROMIO driver for all UNIX-like file systems'
Key is 'romio_aggregator_list' and worth: '0 '
foo
lmm_stripe_count:  1
lmm_stripe_size:   1048576
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 67
       obdidx           objid           objid           group
           67       357367195     0x154cfd9b                0

but still have only a striping_factor of 1 on the created file... while lfs dh -h gives me 85 OSTs...

Thanks,

Eric

ericch1 · 2022-11-23T22:26:00Z

Oups, I forgot... here is the example I used:

#include "mpi.h"
#include <stdio.h>
#include <stdlib.h>

/*
 * Simple function to abort execution and print an error based on MPI return values:
*/
void abortOnError(int ierr) {
  if (ierr != MPI_SUCCESS) {
    printf("ERROR Returned by MPI: %d\n",ierr);
    char* lCharPtr = (char*) malloc(sizeof(char)*MPI_MAX_ERROR_STRING);
    int lLongueur = 0;
    MPI_Error_string(ierr,lCharPtr, &lLongueur);
    printf("ERROR_string Returned by MPI: %s\n",lCharPtr);
    free(lCharPtr);
    MPI_Abort( MPI_COMM_WORLD, ierr );
  }
}

/*
 * Here I use only the first hint for now, but you can try a few to see the difference:
*/
const int lNbHints = 1;
char *lHints      [] = {"striping_factor",
                        "striping_unit",
                        "access_style"};

char *lHintsValues[] = {"2",
                        "1048576",
                        "write_once,random"};

/*
 * Simple function to extract all information in MPI_Info:
*/
void displayFileInfo(MPI_File pFile)
{
  printf( "Informations on file:\n");fflush(stdout);

  char lKey[MPI_MAX_INFO_KEY];
  char lValue[MPI_MAX_INFO_VAL];
  int lFlag = 0;
  MPI_Info lInfo;
  int lNKeys = 0;
  abortOnError(MPI_File_get_info( pFile, &lInfo ));
  abortOnError(MPI_Info_get_nkeys( lInfo, &lNKeys ));

  /* Note that an implementation is allowed to ignore the set_info, so we'll accept either the original or the updated version */
  for (int i = 0 ; i < lNKeys; ++i ) {
    abortOnError(MPI_Info_get_nthkey( lInfo, i, lKey ));
    abortOnError(MPI_Info_get       ( lInfo, lKey, MPI_MAX_INFO_VAL, lValue, &lFlag ));
    if (lFlag) {
      printf( "Key is '%s' and worth: '%s'\n", lKey, lValue);fflush(stdout);
    }
    else {
      printf( "Error getting key is '%s' \n", lKey);fflush(stdout);
    }
  }

  MPI_Info_free( &lInfo );

}

/*
 * Function to create a file with *hints*:
*/
void createFileCollectivelyWithHint(char* pFileName)
{
  int size, rank;

  MPI_Comm_rank( MPI_COMM_WORLD, &rank );
  MPI_Comm_size( MPI_COMM_WORLD, &size );

  MPI_File lFile = 0;

  MPI_Info lInfoIn;
  abortOnError(MPI_Info_create( &lInfoIn ));

  for (int i = 0 ; i < lNbHints; ++i ) {
    abortOnError(MPI_Info_set( lInfoIn, lHints[i], lHintsValues[i] ));
  }

  printf("Creating the file by MPI_file_open : %s\n", pFileName);
  abortOnError(MPI_File_open( MPI_COMM_WORLD, pFileName, ( MPI_MODE_CREATE | MPI_MODE_WRONLY ), lInfoIn, &lFile ));

  displayFileInfo(lFile);

  if (lFile) {
    printf("  ...closing the file %s\n", pFileName);
    abortOnError(MPI_File_close(&lFile ));
  }
}

int main(int argc, char *argv[])
{
    MPI_Init(&argc, &argv);

    if (2 != argc) {
      printf("ERROR: you must specify a filename to create.\n");
      MPI_Finalize();
      return 1;
    }
    char* lFileName = argv[1];


    createFileCollectivelyWithHint(lFileName);

    /* Re-open the file in read-only mode just to see MPI_Info values */
    MPI_Info lInfo = MPI_INFO_NULL;
    MPI_File lFile = 0;
    abortOnError(MPI_Info_create( &lInfo ));
    abortOnError(MPI_File_open( MPI_COMM_WORLD, lFileName, ( MPI_MODE_RDONLY ), lInfo, &lFile ));
    displayFileInfo(lFile);
    abortOnError(MPI_File_close(&lFile ));

    MPI_Finalize();

    return 0;
}

ericch1 · 2022-11-25T07:24:54Z

Hi,

@edgargabriel @ggouaillardet I am really confused about hint names. Please help me understand.

Lustre document "stripe_count" as: "Indicates the number of OSTs that this file will be striped across."
(https://github.com/DDNStorage/lustre_manual_markdown/blob/master/06.05-Setting%20Lustre%20Properties%20in%20a%20C%20Program%20(llapi).md#llapi_file_create)

OpenMPI calls:

llapi_file_create(filename,
                              fs_lustre_stripe_size,     /*** Eric's comment: this is stripe_size in lustre api doc. ***/
                              -1, /* MSC need to change that */
                              fs_lustre_stripe_width, /*** Eric's comment: this is stripe_count in lustre api doc. ***/
                              0); /* MSC need to change that */

with fs_lustre_stripe_width taken from hint stripe_width if given into a MPI_Info.

so far so good.

But, in commit 1631f38, support for "striping_factor" and "striping_unit" has been introduced from MPI standard, which defines

"striping_factor" (integer) [SAME]: This hint specifies the number of I/O devices that
the file should be striped across, and is relevant only when the file is created.

(see https://www.mpi-forum.org/docs/mpi-4.0/mpi40-report.pdf, page 656)

So I understand that "striping_factor" in MPI is the "stripe_count" in Lustre.

However, the commit 1631f38 links striping_factor to fs_lustre_stripe_size as shown here:

  opal_info_get (info, "striping_factor", &stripe_str, &flag);
    if ( flag ) {
        sscanf ( stripe_str->string, "%d", &fs_lustre_stripe_size );
        OBJ_RELEASE(stripe_str);
    }

then striping_factor is given as the 4th parameter to lustre api which is strip_size instead of the second, which is strip_count...

Am I confused or this a bug?

Thanks,

Eric

edgargabriel · 2022-11-28T14:43:31Z

@ericch1 sorry for the late reply, I was out of office last week. I will have a look in a bit, there is a good chance that you are right and this is a bug.

ericch1 · 2022-11-28T20:06:31Z

No problem @edgargabriel , there is no hurry for me... Thanks for looking into this! :)

edgargabriel · 2022-12-04T17:28:37Z

@ericch1 would you have a chance to test https://github.com/edgargabriel/ompi/tree/topic/lustre-info-swap to see whether this fixes the info object assignment on lustre? I do not have currently access to a lustre file system to test this out, but I think you are right that I accidentally swapped the meaning of the two MPI info objects on lustre.

ericch1 · 2022-12-04T23:08:57Z

Ok, thanks @edgargabriel , I am trying to install it right now, but I looked at the commit and I am very confident that it will works since the hints "stripe_size" and "stripe_width" where working for me with OpenMPI 4.1.1...
I may need Compute Canada help to have Lustre configured correctly... To be continued tomorrow... (Monday)....

ericch1 · 2022-12-05T08:02:57Z

I am stucked here:

make[4]: Entering directory '/lustre03/project/6003115/ericc/MPI_File_open_with_hints/test_for_EG/ompi/3rd-party/openpmix/src/mca/prm/tm'
  CC       prm_tm_component.lo
  CC       prm_tm.lo
prm_tm.c: In function ‘tm_notify’:
prm_tm.c:54:46: error: unused parameter ‘status’ [-Werror=unused-parameter]
   54 | static pmix_status_t tm_notify(pmix_status_t status, const pmix_proc_t *source,
      |                                ~~~~~~~~~~~~~~^~~~~~
prm_tm.c:54:73: error: unused parameter ‘source’ [-Werror=unused-parameter]
   54 | static pmix_status_t tm_notify(pmix_status_t status, const pmix_proc_t *source,
      |                                                      ~~~~~~~~~~~~~~~~~~~^~~~~~
prm_tm.c:55:50: error: unused parameter ‘range’ [-Werror=unused-parameter]
   55 |                                pmix_data_range_t range, const pmix_info_t info[], size_t ninfo,
      |                                ~~~~~~~~~~~~~~~~~~^~~~~
prm_tm.c:55:75: error: unused parameter ‘info’ [-Werror=unused-parameter]
   55 |                                pmix_data_range_t range, const pmix_info_t info[], size_t ninfo,
      |                                                         ~~~~~~~~~~~~~~~~~~^~~~~~
prm_tm.c:55:90: error: unused parameter ‘ninfo’ [-Werror=unused-parameter]
   55 |                                pmix_data_range_t range, const pmix_info_t info[], size_t ninfo,
      |                                                                                   ~~~~~~~^~~~~
prm_tm.c:56:49: error: unused parameter ‘cbfunc’ [-Werror=unused-parameter]
   56 |                                pmix_op_cbfunc_t cbfunc, void *cbdata)
      |                                ~~~~~~~~~~~~~~~~~^~~~~~
prm_tm.c:56:63: error: unused parameter ‘cbdata’ [-Werror=unused-parameter]
   56 |                                pmix_op_cbfunc_t cbfunc, void *cbdata)
      |                                                         ~~~~~~^~~~~~
cc1: all warnings being treated as errors

Does it exists a quick fix?

edgargabriel · 2022-12-05T14:09:28Z

hm, I didn't get this error, can you maybe try setting --with-pmix=internal and --with-prrte=internal during configure time? I am wondering whether configure accidentally picks up an external pmix.

jsquyres · 2022-12-05T14:11:07Z

I'm guessing that these are legit warnings in the PMIx build; I notice that @ericch1 has "treat warnings as errors", and the compile therefore fails.

ericch1 · 2022-12-07T22:47:06Z

Still have the errors with --with-pmix=internal --with-prrte=internal options.

Il will try with an older compiler than gcc 10.3.0...

rhc54 · 2022-12-07T23:08:49Z

Are those from OMPI main? or OMPI v5? I'm wondering if the submodules are stale as I thought we fixed that quite a while ago.

rhc54 · 2022-12-07T23:21:42Z

Just took a quick peek and those are definitely fixed in HEAD of OMPI main. The "prm/tm" component does not exist in the HEAD of OMPI v5.0.x branch.

ericch1 · 2022-12-09T06:37:21Z

I do know why I am at that point, but looking at the logs, the version Edgard gave me is not so outdated:

commit 2778e26869955bfd90ab8cac64613cfb6ad380b5 (HEAD -> main, origin/topic/lustre-info-swap)
Author: Edgar Gabriel [<edgar.gabriel@amd.com>](mailto:edgar.gabriel@amd.com)
Date:   Sun Dec 4 11:23:59 2022 -0600

   fs/lustre: fix assignment of info objects to lustre args
    
   It seems that commit 1631f387704641dd7939cbce156a47ea8eda39eb
   accidentally reverted the two info objects setting stripe size and stripe
   width for lustre file systems.
    
   Signed-off-by: Edgar Gabriel [<edgar.gabriel@amd.com>](mailto:edgar.gabriel@amd.com)

commit 4dae1421f1c19fed941eca42cac8756debca3b7b
Merge: 0f54160621 f96d9b8e11
Author: Tommy Janjusic [<janjust@users.noreply.github.com>](mailto:janjust@users.noreply.github.com)
Date:   Thu Dec 1 15:08:14 2022 -0600

   Merge pull request #11071 from wckzhang/querycuda
    
   mpiext/cuda: Properly return CUDA status

Ok, I got it... I did a :

git reset --hard remotes/origin/topic/lustre-info-swap

but forgot to do a:

git submodule update --recursive

and submodules are now:

git submodule status --recursive
 250004266bc046c6303c8531ababdff4e1237525 3rd-party/openpmix (v1.1.3-3661-g25000426)
 9a9be81ba63a50e5b0c997e95358298b85400600 3rd-party/openpmix/config/oac (heads/main)
 ca2bf3aeab38261ae7c88cea64bc782c949bd76e 3rd-party/prrte (psrvr-v2.0.0rc1-4517-gca2bf3aeab)
 9a9be81ba63a50e5b0c997e95358298b85400600 3rd-party/prrte/config/oac (heads/main)
 5c8de3d97b763bf8981fb49cbedd36e201b8fc0a config/oac (5c8de3d)

But now that I did it, relauched "./autogen" and "configure", I have these new errors at make:

In file included from mtl_ofi_endpoint.c:11:
mtl_ofi.h: In function ‘ompi_mtl_ofi_register_buffer’:
mtl_ofi.h:328:26: error: ‘FI_HMEM_ROCR’ undeclared (first use in this function); did you mean ‘FI_HMEM_CUDA’?
 328 |             attr.iface = FI_HMEM_ROCR;
     |                          ^~~~~~~~~~~~
     |                          FI_HMEM_CUDA

mtl_ofi.h:328:26: note: each undeclared identifier is reported only once for each function it appears in
In file included from mtl_ofi_isend_opt.c:11:
mtl_ofi.h: In function ‘ompi_mtl_ofi_register_buffer’:
mtl_ofi.h:328:26: error: ‘FI_HMEM_ROCR’ undeclared (first use in this function); did you mean ‘FI_HMEM_CUDA’?
 328 |             attr.iface = FI_HMEM_ROCR;
     |                          ^~~~~~~~~~~~
     |                          FI_HMEM_CUDA
mtl_ofi.h:328:26: note: each undeclared identifier is reported only once for each function it appears in

...

It looks like I am unable to configure/compile correctly... :(

Where am I wrong?

@edgargabriel did you compiled successfully this branch/sha?

Eric

edgargabriel · 2022-12-09T15:16:36Z

@ericch1 yes, it did work for me. I think this last error is because your configure picked up an 'old' libfabric installation somewhere which does not have yet FI_HMEM_ROCR declared. You can prevent this by adding '--with-ofi=no' to your configure line (assuming you are not using the ofi components), or explicitely point Open MPI to a newer libfabric installation.

ericch1 · 2022-12-12T09:58:05Z

Ok, I have been able to test it!

It works:

./test_for_EG/ompi_EG/bin/mpiexec --mca smsc ^knem -n 2 ./s toto 
Creating the file by MPI_file_open : toto
Creating the file by MPI_file_open : toto
s: setstripe error for 'toto': stripe already set
Informations on file:
Key is 'striping_factor' and worth: '2'
  ...closing the file toto
Informations on file:
Key is 'striping_factor' and worth: '2'
  ...closing the file toto
Informations on file:
Informations on file:
[ericc@beluga2 MPI_File_open_with_hints]$ rm toto
[ericc@beluga2 MPI_File_open_with_hints]$ ./test_for_EG/ompi_EG/bin/mpiexec --mca smsc ^knem -n 2 ./s toto 
Creating the file by MPI_file_open : toto
Creating the file by MPI_file_open : toto
Informations on file:
Key is 'striping_factor' and worth: '2'
  ...closing the file toto
Informations on file:
Key is 'striping_factor' and worth: '2'
  ...closing the file toto
Informations on file:
Informations on file:
[ericc@beluga2 MPI_File_open_with_hints]$ lfs getstripe toto
toto
lmm_stripe_count:  2
lmm_stripe_size:   1048576
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 239
        obdidx           objid           objid           group
           239         6091454       0x5cf2be     0x3d00000400
           323         6062745       0x5c8299     0x5700000400

and when I try to set the striping_factor to an existing file I have a very nice and comprehensible error! :) Thanks for that!

Eric

edgargabriel · 2022-12-12T13:52:55Z

@ericch1 excellent, thank you very much! I will file a pr to bring this into master and 5.0.x

ax3l · 2023-05-23T01:34:25Z

This is awesome! Thank you all!

@edgargabriel Do you think this fix can also be backported to the 4.x releases? (Or was the swapped name only in the main branch and not yet released? 1631f38)

edgargabriel · 2023-05-23T13:30:22Z

@ax3l I think this should be possible. We will need to cherry-pick PR #10135 in addition to PR #11191 for that.

edgargabriel · 2023-05-23T14:37:40Z

@ax3l if you have a chance, could you please test whether PR #11713 works for your testcase?

ericch1 mentioned this issue Dec 5, 2022

Set I_MPI_EXTRA_FILESYSTEM=on in Intel MPI ComputeCanada/software-stack-config#44

Open

edgargabriel mentioned this issue Dec 12, 2022

fs/lustre: fix assignment of info objects to lustre args #11191

Merged

ax3l mentioned this issue May 23, 2023

MPI-I/O Hints openPMD/openPMD-api#1449

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Not able to set stripe and stripe sie using mpi_info_set #10132

Not able to set stripe and stripe sie using mpi_info_set #10132

monsieuralok commented Mar 17, 2022

ggouaillardet commented Mar 17, 2022

monsieuralok commented Mar 17, 2022 •

edited

Loading

ggouaillardet commented Mar 17, 2022

monsieuralok commented Mar 17, 2022 •

edited

Loading

monsieuralok commented Mar 17, 2022 •

edited

Loading

ggouaillardet commented Mar 17, 2022

monsieuralok commented Mar 17, 2022

ggouaillardet commented Mar 17, 2022

edgargabriel commented Mar 17, 2022

ericch1 commented Nov 23, 2022

ericch1 commented Nov 23, 2022

ericch1 commented Nov 25, 2022

edgargabriel commented Nov 28, 2022

ericch1 commented Nov 28, 2022

edgargabriel commented Dec 4, 2022

ericch1 commented Dec 4, 2022

ericch1 commented Dec 5, 2022 •

edited

Loading

edgargabriel commented Dec 5, 2022 •

edited

Loading

jsquyres commented Dec 5, 2022

ericch1 commented Dec 7, 2022

rhc54 commented Dec 7, 2022

rhc54 commented Dec 7, 2022

ericch1 commented Dec 9, 2022

edgargabriel commented Dec 9, 2022 •

edited

Loading

ericch1 commented Dec 12, 2022

edgargabriel commented Dec 12, 2022

ax3l commented May 23, 2023 •

edited

Loading

edgargabriel commented May 23, 2023

edgargabriel commented May 23, 2023

Not able to set stripe and stripe sie using mpi_info_set #10132

Not able to set stripe and stripe sie using mpi_info_set #10132

Comments

monsieuralok commented Mar 17, 2022

ggouaillardet commented Mar 17, 2022

monsieuralok commented Mar 17, 2022 • edited Loading

ggouaillardet commented Mar 17, 2022

monsieuralok commented Mar 17, 2022 • edited Loading

monsieuralok commented Mar 17, 2022 • edited Loading

ggouaillardet commented Mar 17, 2022

monsieuralok commented Mar 17, 2022

ggouaillardet commented Mar 17, 2022

edgargabriel commented Mar 17, 2022

ericch1 commented Nov 23, 2022

ericch1 commented Nov 23, 2022

ericch1 commented Nov 25, 2022

edgargabriel commented Nov 28, 2022

ericch1 commented Nov 28, 2022

edgargabriel commented Dec 4, 2022

ericch1 commented Dec 4, 2022

ericch1 commented Dec 5, 2022 • edited Loading

edgargabriel commented Dec 5, 2022 • edited Loading

jsquyres commented Dec 5, 2022

ericch1 commented Dec 7, 2022

rhc54 commented Dec 7, 2022

rhc54 commented Dec 7, 2022

ericch1 commented Dec 9, 2022

edgargabriel commented Dec 9, 2022 • edited Loading

ericch1 commented Dec 12, 2022

edgargabriel commented Dec 12, 2022

ax3l commented May 23, 2023 • edited Loading

edgargabriel commented May 23, 2023

edgargabriel commented May 23, 2023

monsieuralok commented Mar 17, 2022 •

edited

Loading

monsieuralok commented Mar 17, 2022 •

edited

Loading

monsieuralok commented Mar 17, 2022 •

edited

Loading

ericch1 commented Dec 5, 2022 •

edited

Loading

edgargabriel commented Dec 5, 2022 •

edited

Loading

edgargabriel commented Dec 9, 2022 •

edited

Loading

ax3l commented May 23, 2023 •

edited

Loading