UCX support for ompi and oshmem #1008

alex-mikheev · 2015-10-11T14:51:04Z

@miked-mellanox @shamisp @yosefe @jladd-mlnx
please take a look

shamisp · 2015-10-11T15:37:36Z

jladd-mlnx · 2015-10-12T14:52:10Z

Need to ensure this works with the new add procs behavior. We need to be able to support the direct modex. As far as I can tell, there is only support for bulk add procs.

jsquyres · 2015-10-12T15:00:23Z

ompi/datatype/ompi_datatype.h

@bosilca Are you ok with adding this member to the ompi_datatype_t?

This addition only affects the OMPI datatypes, everything below (and the real datatype engine is down in OPAL) remains unaffected.

shamisp · 2015-10-12T15:11:36Z

@jladd-mlnx UCX itself does not make any assumptions about wire up protocols used by the above layer. Upper layers may use direct/inderect/bulk wire ups.

jsquyres · 2015-10-12T18:47:18Z

@shamisp I think @jladd-mlnx was referring to the recent change in the PML add_procs() behavior to allow partial wireup. His question is not related to wire-line protocols.

shamisp · 2015-10-12T21:17:35Z

@jsquyres - I reviewed this code as well already existing code in the master. Our add_proc() behavior is aligned with the add_proc behavior in the master.

jsquyres · 2015-10-12T21:28:00Z

@shamisp Ok, sweet.

jsquyres · 2015-10-12T21:28:41Z

FWIW, the "OSHMEM/SPML/IKRIT: fixes typo in .h file" commit should be squashed down into the commit that it is fixing before this is merged in.

shamisp · 2015-10-13T16:57:45Z

@jsquyres - sure, we will squash it once we address all the comments.

lanl-ompi · 2015-10-14T14:49:22Z

Test FAILed.

hppritcha · 2015-10-14T16:28:13Z

Which version of ucx do I need to compile this code?

shamisp · 2015-10-14T19:17:16Z

@hppritcha https://github.com/yosefe/ucx/tree/topic/nb-tag-matching
(once Yossi back from travel it will go into master)

shamisp · 2015-10-14T19:47:23Z

I'm not sure if LANL-disable-dlopen-check is related to our code.

jsquyres · 2015-10-14T19:58:22Z

hello_c timed out after 3 minutes. Does that happen if you --disable-dlopen with ucx?

hppritcha · 2015-10-14T20:31:38Z

Hi Jeff. That may be spurious (the hang).

hppritcha · 2015-10-14T20:33:48Z

bot:retest

rolfv · 2015-10-19T19:54:52Z

Has anyone configured with --enable-picky? I get lots of warnings with it configured that way.

yosefe · 2015-10-20T10:51:33Z

@rolfv I've fixed the warning in OMPI code, some warnings come from UCX headers, and this is addressed by openucx/ucx#448

rolfv · 2015-10-20T13:24:05Z

Thanks, things look much better.

rolfv · 2015-10-20T13:36:23Z

Updated to latest ucx and then recompiled Open MPI and I know see this:

[rvandevaart@drossetti-ivy4 dbg]$ gmake -j 20 > make.log
../../../../../ompi/mca/pml/ucx/pml_ucx.c: In function 'mca_pml_ucx_iprobe':
../../../../../ompi/mca/pml/ucx/pml_ucx.c:512: warning: implicit declaration of function 'ucp_tag_probe_nb'
../../../../../ompi/mca/pml/ucx/pml_ucx.c: In function 'mca_pml_ucx_improbe':
../../../../../ompi/mca/pml/ucx/pml_ucx.c:564: warning: implicit declaration of function 'ucp_tag_msg_probe_nb'
../../../../../ompi/mca/pml/ucx/pml_ucx.c:565: warning: assignment makes pointer from integer without a cast
../../../../../ompi/mca/pml/ucx/pml_ucx.c: In function 'mca_pml_ucx_mprobe':
../../../../../ompi/mca/pml/ucx/pml_ucx.c:595: warning: assignment makes pointer from integer without a cast
../../../../../ompi/mca/pml/ucx/pml_ucx.c: In function 'mca_pml_ucx_imrecv':
../../../../../ompi/mca/pml/ucx/pml_ucx.c:617: warning: implicit declaration of function 'ucp_tag_msg_recv_nb'
../../../../../ompi/mca/pml/ucx/pml_ucx.c:620: warning: cast to pointer from integer of different size
../../../../../ompi/mca/pml/ucx/pml_ucx.c: In function 'mca_pml_ucx_mrecv':
../../../../../ompi/mca/pml/ucx/pml_ucx.c:643: warning: cast to pointer from integer of different size
[rvandevaart@drossetti-ivy4 dbg]$ 
``

yosefe · 2015-10-20T13:37:35Z

ucx probe is not merged yet, we just need to finalize the api

jsquyres · 2015-10-20T13:51:47Z

@yosefe Did you commit something to Open MPI that does not yet exist in the downstream UCX API? That does not seem like a good idea.

yosefe · 2015-10-20T14:21:18Z

@jsquyres Yes, i've opened this PR before all code was merged to UCX, to get comments and address them as early as possibly. Obviously, this will not be merged to Open MPI before all UCX code is in place.

jsquyres · 2015-10-20T14:25:00Z

@yosefe Ah, got it -- the code in question is just here on the PR, not already in master. Thanks for the clarification.

Fast path lookup is done in inline funcion.

PML UCX will use it to cache a handle for UCX datatype.

yosefe · 2015-10-20T17:01:33Z

squashed the various fixes into the commits.

shamisp · 2015-10-21T02:27:11Z

👍

mike-dubman · 2015-10-21T06:33:30Z

@yosefe - please integrate this PR into v2.x and v1.10.1

UCX support for ompi and oshmem

osc/pt2pt: do no use OPAL_THREAD_ADD64 for lock serial number

jsquyres reviewed Oct 12, 2015
View reviewed changes

alex-mikheev force-pushed the topic/ucx_support branch from b025251 to e6be425 Compare October 14, 2015 14:22

alex-mikheev and others added 2 commits October 20, 2015 19:36

spml_ikrit: fixes typo in .h file.

8fa1438

spml/memheap: add support for mkey unpack.

bd3f4c8

alex-mikheev and others added 4 commits October 20, 2015 19:45

oshmem/memheap: optimized mkey lookup.

b020b62

Fast path lookup is done in inline funcion.

oshmem: Add UCX spml.

f2b501a

add pml-specific field in OMPI datatype.

502dc8a

PML UCX will use it to cache a handle for UCX datatype.

ompi: Add UCX PML.

a313588

yosefe force-pushed the topic/ucx_support branch from a17394e to a313588 Compare October 20, 2015 17:00

bosilca mentioned this pull request Oct 21, 2015

Bring UCX PML and SPML in #982

Closed

mike-dubman added a commit that referenced this pull request Oct 21, 2015

Merge pull request #1008 from alex-mikheev/topic/ucx_support

4ea13f1

UCX support for ompi and oshmem

mike-dubman merged commit 4ea13f1 into open-mpi:master Oct 21, 2015

yosefe deleted the topic/ucx_support branch October 21, 2015 13:15

bosilca mentioned this pull request Oct 26, 2015

Add UCX support open-mpi/ompi-release#718

Merged

yosefe mentioned this pull request Nov 19, 2015

Add ucx support to ompi and oshmem. open-mpi/ompi-release#775

Merged

jsquyres pushed a commit to jsquyres/ompi that referenced this pull request Sep 19, 2016

Merge pull request open-mpi#1008 from hjelmn/32bit_fixes

a3b4735

osc/pt2pt: do no use OPAL_THREAD_ADD64 for lock serial number

UCX support for ompi and oshmem #1008

UCX support for ompi and oshmem #1008

Uh oh!

Conversation

alex-mikheev commented Oct 11, 2015

Uh oh!

shamisp commented Oct 11, 2015

Uh oh!

jladd-mlnx commented Oct 12, 2015

Uh oh!

jsquyres Oct 12, 2015

Choose a reason for hiding this comment

Uh oh!

bosilca Oct 14, 2015

Choose a reason for hiding this comment

Uh oh!

shamisp commented Oct 12, 2015

Uh oh!

jsquyres commented Oct 12, 2015

Uh oh!

shamisp commented Oct 12, 2015

Uh oh!

jsquyres commented Oct 12, 2015

Uh oh!

jsquyres commented Oct 12, 2015

Uh oh!

shamisp commented Oct 13, 2015

Uh oh!

lanl-ompi commented Oct 14, 2015

Uh oh!

hppritcha commented Oct 14, 2015

Uh oh!

shamisp commented Oct 14, 2015

Uh oh!

shamisp commented Oct 14, 2015

Uh oh!

jsquyres commented Oct 14, 2015

Uh oh!

hppritcha commented Oct 14, 2015

Uh oh!

hppritcha commented Oct 14, 2015

Uh oh!

rolfv commented Oct 19, 2015

Uh oh!

yosefe commented Oct 20, 2015

Uh oh!

rolfv commented Oct 20, 2015

Uh oh!

rolfv commented Oct 20, 2015

Uh oh!

yosefe commented Oct 20, 2015

Uh oh!

jsquyres commented Oct 20, 2015

Uh oh!

yosefe commented Oct 20, 2015

Uh oh!

jsquyres commented Oct 20, 2015

Uh oh!

yosefe commented Oct 20, 2015

Uh oh!

shamisp commented Oct 21, 2015

Uh oh!

mike-dubman commented Oct 21, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants