[test] Add MCH `alltoallv` check #347

ajocksch · 2018-07-05T10:01:51Z

Closes #253

ajocksch · 2018-07-05T10:03:44Z

I get this error message from the performance check

debug: Alltoallv_ on kesch:cn using PrgEnv-gnu: caught socket.gaierror: [Errno -2] Name or service not known

teojgo · 2018-07-06T14:43:18Z

@ajocksch the error you get is the one of issue #349

vkarak · 2018-07-10T16:10:45Z

@ajocksch Can you be a bit more precise on the title of your PR? Also the socket error is no more relevant; it was fixed in #352.

teojgo · 2018-07-11T11:13:08Z

After the merge of #352, the tests now complete successfully on kesch. I do not know if they are relevant to daint/dom since the module craype-network-infiniband is not available there.

vkarak · 2018-07-11T11:16:21Z

This PR is not ready to merge. I have several comments. I will review it soon.

vkarak · 2018-08-28T07:17:51Z

cscs-checks/mch/alltoallv.py

+
+    def setup(self, partition, environ, **job_opts):
+        if environ.name.startswith('PrgEnv-cray'):
+            environ.fflags = '-O2 -hacc -hnoomp'


Setting the environment flags is now deprecated. You should use the build systems for that.

vkarak · 2018-08-28T07:19:02Z

cscs-checks/mch/alltoallv.py

+        self.num_gpus_per_node = 16
+        self.num_tasks_per_node = 16
+        self.num_tasks_per_socket = 8
+        self.executable = 'src/comm_overlap_benchmark %s' % exec_parameter


Use self.executable_opts to pass the parameters.

vkarak · 2018-08-28T07:20:00Z

cscs-checks/mch/alltoallv.py

+        self.num_tasks_per_node = 16
+        self.num_tasks_per_socket = 8
+        self.executable = 'src/comm_overlap_benchmark %s' % exec_parameter
+        self.sourcesdir = ('https://github.com/cosunae/comm_overlap_bench')


Please remove the parentheses here.

vkarak · 2018-08-28T07:23:30Z

cscs-checks/mch/alltoallv.py

+                'kesch:cn': {
+                    'perf': (5.62155, None, 0.15)
+                },
+            }


There is too much code duplication in the above if/else. Rewrite it as follows:

if not exec_parameter: ref = 5.53777 elif exec_parameter == '--nocomm': ref = 5.7878 elif exec_parameter == '--nocomp': ref = 5.62155 self.reference = { 'kesch:cn': { 'perf': (ref, None, 0.15) } }

vkarak · 2018-08-28T07:24:16Z

cscs-checks/mch/alltoallv.py

+        self.sourcepath = 'src'
+        self.sanity_patterns = sn.assert_found(r'ELAPSED TIME:', self.stdout)
+        self.perf_patterns = {
+            'perf': sn.extractsingle(r'ELAPSED TIME:\s+(?P<perf>\S+)',


Please name the performance variable with a more meaningful name. What this metric is? Latency?

elapsed_time

vkarak · 2018-08-28T07:26:02Z

cscs-checks/mch/alltoallv.py

+                },
+            }
+
+        self.modules += ['craype-haswell', 'craype-network-infiniband', 'mvapich2gdr_gnu/2.2_cuda_8.0', 'cray-libsci_acc/17.03.1', 'cmake']


Please wrap this line.

vkarak · 2018-08-28T07:26:54Z

cscs-checks/mch/alltoallv.py

+             -DCUDA_COMPUTE_CAPABILITY="sm_37" \
+             -DCMAKE_BUILD_TYPE=Release \
+             -DENABLE_MPI_TIMER=ON'
+        ]


This test should be revised as soon as #299 is fixed.

vkarak · 2018-08-28T07:46:24Z

config/cscs.py

-                            '_rfm_gpu': ['--gres=gpu:{num_gpus_per_node}']
+                            '_rfm_gpu': ['--gres=gpu:{num_gpus_per_node}'],
+                            'distribution': ['--distribution=block:block'],
+                            'cpu_bind' : ['--cpu_bind=q']


The resource names must be more generic and allow tests to pass options:

'resources': { 'task_placement': ['--distribution={distribution}', '--cpu_bind={cpu_binding}'] }

Then you should define the extra_resources as follows:

self.extra_resources = { 'task_placement': { 'distribution': 'block:block', 'cpu_binding': 'q' }

But I don't think that the --cpu_bind option is needed: q is the default value.

this comment is outdated

Why this comment is outdated?

…cks/mch_comm_overlap_bench_barebones

vkarak

I have several comments. It seems you are mixing parts that are relevant for different systems (Daint/Dom and Kesch) and different MPI versions. Have you tested this on Daint/Dom? I'm sure it would have failed. I suggest focusing only on Kesch, remove Daint/Dom from the supported systems list, as well as all relevant bits from the building and running. Unless it's straightforward to support Daint/Dom.

vkarak · 2018-10-10T12:11:49Z

config/cscs.py

-                            '_rfm_gpu': ['--gres=gpu:{num_gpus_per_node}']
+                            '_rfm_gpu': ['--gres=gpu:{num_gpus_per_node}'],
+                            'distribution': ['--distribution=block:block'],
+                            'cpu_bind' : ['--cpu_bind=q']


Why this comment is outdated?

vkarak · 2018-10-10T12:14:09Z

cscs-checks/mch/alltoallv.py

+import reframe.utility.sanity as sn
+
+
+@rfm.parameterized_test([''], ['--nocomm'], ['--nocomp'])


I know it's handy, but I wouldn't pass the option here. I would rename the exec_parameter to variant and pass three distinct values: default, nocomm, nocomp. I would then set up the executable options accordingly. The way it is now will produce ugly test names, e.g., Alltoallv___nocomm.

vkarak · 2018-10-10T12:14:28Z

cscs-checks/mch/alltoallv.py

+        if self.current_system.name in ['daint', 'dom']:
+            self.modules = ['craype-accel-nvidia60']
+            self._pgi_flags = ['-acc', '-ta=tesla:cc60', '-Mnorpath']
+        elif self.current_system.name in ['kesch']:


== 'kesch'

vkarak · 2018-10-10T12:16:22Z

cscs-checks/mch/alltoallv.py

+        if environ.name.startswith('PrgEnv-cray'):
+            self.build_system.fflags = ['-O2', '-hacc', '-hnoomp']
+        elif environ.name.startswith('PrgEnv-pgi'):
+            self.build_system.fflags = [self._pgi_flags]


I wonder how this is allowed. It should have been:

self.build_system.fflags = self._pgi_flags

Ok, this simply works because we are not running this test for the PrgEnv-pgi.

vkarak · 2018-10-10T12:17:03Z

cscs-checks/mch/alltoallv.py

+        self.num_tasks_per_node = 16
+        self.num_tasks_per_socket = 8
+        self.executable = ('build/src/comm_overlap_benchmark '
+                           '%s' % exec_parameter)


The exec_parameter should be part of the self.executable_opts list.

vkarak · 2018-10-10T12:19:50Z

cscs-checks/mch/alltoallv.py

+
+        self.modules += [
+            'craype-haswell', 'craype-network-infiniband',
+            'mvapich2gdr_gnu/2.2_cuda_8.0', 'cray-libsci_acc/17.03.1', 'cmake'


mvapich2gdr_gnu/2.2_cuda_8.0 is supposed to be loaded already by ReFrame's PrgEnv-cray. Also verify that the other modules are still needed.

vkarak · 2018-10-10T12:20:50Z

cscs-checks/mch/alltoallv.py

+
+        self.variables = {
+            'G2G': '1',
+            'jobs': '144',


This seems not to be used.

vkarak · 2018-10-10T12:24:04Z

cscs-checks/mch/alltoallv.py

+
+        self.pre_run = [
+            'export BOOST_LIBRARY_PATH=/apps/escha/UES/PrgEnv-gnu-17.02'\
+            '/modulefiles/boost/1.63.0-gmvolf-17.02-python-2.7.13/lib',


Do not split the lines with \. I would suggest to put each path in a single long line, even if it is beyond 80 columns.

vkarak · 2018-10-10T12:24:33Z

cscs-checks/mch/alltoallv.py

+            'export XXX_LIBRARY_PATH=/apps/escha/UES/RH7.3_experimental/pgi'\
+            '/17.10/linux86-64/17.10/REDIST',
+            'export LD_LIBRARY_PATH=$XXX_LIBRARY_PATH:$LD_LIBRARY_PATH',
+            'export LD_PRELOAD=/opt/mvapich2/gdr/2.3a/mcast/no-openacc'\


This seems to contradict with the mvapich2gdr_gnu/2.2_cuda_8.0 loaded above.

vkarak · 2018-10-10T12:24:50Z

cscs-checks/mch/alltoallv.py

+        }
+
+        self.pre_run = [
+            'export BOOST_LIBRARY_PATH=/apps/escha/UES/PrgEnv-gnu-17.02'\


Don't we have a module for Boost?

Is this needed on Kesch or just for Daint/Dom.

vkarak · 2018-10-10T14:57:45Z

@ajocksch Indeed it does not work on Dom:

[==========] Running 3 check(s)
[==========] Started on Wed Oct 10 16:56:49 2018

[----------] started processing Alltoallv_ (Alltoallv_)
[     FAIL ] Alltoallv_ on dom:gpu using PrgEnv-gnu
[----------] finished processing Alltoallv_ (Alltoallv_)

[----------] started processing Alltoallv___nocomm (Alltoallv___nocomm)
[     FAIL ] Alltoallv___nocomm on dom:gpu using PrgEnv-gnu
[----------] finished processing Alltoallv___nocomm (Alltoallv___nocomm)

[----------] started processing Alltoallv___nocomp (Alltoallv___nocomp)
[     FAIL ] Alltoallv___nocomp on dom:gpu using PrgEnv-gnu
[----------] finished processing Alltoallv___nocomp (Alltoallv___nocomp)

[----------] waiting for spawned checks to finish
[----------] all spawned checks have finished

[  FAILED  ] Ran 3 test case(s) from 3 check(s) (3 failure(s))
[==========] Finished on Wed Oct 10 16:56:51 2018

==============================================================================
SUMMARY OF FAILURES
------------------------------------------------------------------------------
FAILURE INFO for Alltoallv_
  * System partition: dom:gpu
  * Environment: PrgEnv-gnu
  * Stage directory: None
  * Job type: batch job (id=-1)
  * Maintainers: ['AJ', 'VK']
  * Failing phase: setup
  * Reason: caught framework exception: could not load module craype-network-infiniband
------------------------------------------------------------------------------
FAILURE INFO for Alltoallv___nocomm
  * System partition: dom:gpu
  * Environment: PrgEnv-gnu
  * Stage directory: None
  * Job type: batch job (id=-1)
  * Maintainers: ['AJ', 'VK']
  * Failing phase: setup
  * Reason: caught framework exception: could not load module craype-network-infiniband
------------------------------------------------------------------------------
FAILURE INFO for Alltoallv___nocomp
  * System partition: dom:gpu
  * Environment: PrgEnv-gnu
  * Stage directory: None
  * Job type: batch job (id=-1)
  * Maintainers: ['AJ', 'VK']
  * Failing phase: setup
  * Reason: caught framework exception: could not load module craype-network-infiniband
------------------------------------------------------------------------------

vkarak · 2018-10-11T07:28:57Z

@ajocksch I have fixed and adapted the test for both Kesch and Dom/Daint. It turned out to be much easier. Their build scripts seem quite outdated. Please have a look so that we can merge by today.

vkarak · 2018-10-11T08:27:35Z

cscs-checks/mch/alltoallv.py

+            self.num_tasks_per_node = 1
+            self.modules = ['craype-accel-nvidia60', 'CMake']
+            self.variables['MPICH_RDMA_ENABLED_CUDA'] = '1'
+            self.build_system.config_opts += ['-DMPI_VENDOR=mvapich2',


This is not correct, I should adapt the CMake options for the test on Daint/Dom.

vkarak · 2018-10-11T09:38:20Z

@jenkins-cscs retry dom kesch

vkarak · 2018-10-11T11:21:11Z

@jenkins-cscs retry kesch

WIP: alltoallv check; socket.gaierror

5c90ce8

ajocksch requested a review from teojgo July 5, 2018 10:02

vkarak self-requested a review July 5, 2018 10:08

vkarak assigned ajocksch Jul 5, 2018

vkarak added regression test prio: important labels Jul 5, 2018

vkarak added this to the ReFrame sprint 2018w26 milestone Jul 5, 2018

vkarak added the request for enhancement label Jul 5, 2018

vkarak changed the title ~~WIP: alltoallv check; socket.gaierror~~ [WIP] alltoallv check; socket.gaierror Jul 5, 2018

vkarak modified the milestones: ReFrame sprint 2018w26, Upcoming sprint Jul 12, 2018

vkarak added prio: normal and removed prio: important labels Jul 18, 2018

vkarak modified the milestones: Summer sprint, ReFrame sprint 2018w35 Aug 24, 2018

vkarak requested changes Aug 28, 2018

View reviewed changes

vkarak removed this from the ReFrame sprint 2018w35 milestone Sep 4, 2018

vkarak removed the request for enhancement label Sep 27, 2018

ajocksch added 3 commits October 3, 2018 15:17

Merge branch 'master' of https://github.com/eth-cscs/reframe into che…

68c30b3

…cks/mch_comm_overlap_bench_barebones

revised check; cmake; build_system

fc46d6d

deleted boost variable

ec86472

ajocksch changed the title ~~[WIP] alltoallv check; socket.gaierror~~ alltoallv check; socket.gaierror Oct 4, 2018

vkarak changed the title ~~alltoallv check; socket.gaierror~~ [test] Add MCH alltoallv check Oct 4, 2018

vkarak added this to the Upcoming sprint milestone Oct 9, 2018

made changes requested

0370035

2nd attempt; made changes required

90cf263

vkarak requested changes Oct 10, 2018

View reviewed changes

vkarak self-assigned this Oct 10, 2018

Vasileios Karakasis added 2 commits October 10, 2018 17:52

Merge branch 'master' into checks/mch_comm_overlap_bench_barebones

0da3858

Adapt MCH Alltoallv test for Kesch/Dom/Daint

0de854d

vkarak approved these changes Oct 11, 2018

View reviewed changes

vkarak reviewed Oct 11, 2018

View reviewed changes

Remove unnecessary CMake flag from Alltoallv test

f05934a

Merge branch 'master' into checks/mch_comm_overlap_bench_barebones

a5b33e0

vkarak merged commit e0d608a into master Oct 11, 2018

vkarak deleted the checks/mch_comm_overlap_bench_barebones branch October 11, 2018 11:37

		import reframe.utility.sanity as sn


		@rfm.parameterized_test([''], ['--nocomm'], ['--nocomp'])

[test] Add MCH alltoallv check #347

[test] Add MCH alltoallv check #347

Uh oh!

Conversation

ajocksch commented Jul 5, 2018 • edited by vkarak Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ajocksch commented Jul 5, 2018

Uh oh!

teojgo commented Jul 6, 2018

Uh oh!

vkarak commented Jul 10, 2018

Uh oh!

teojgo commented Jul 11, 2018

Uh oh!

vkarak commented Jul 11, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vkarak left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vkarak Oct 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vkarak commented Oct 10, 2018

Uh oh!

[test] Add MCH `alltoallv` check #347

[test] Add MCH `alltoallv` check #347

ajocksch commented Jul 5, 2018 •

edited by vkarak

Loading

vkarak Oct 10, 2018 •

edited

Loading

vkarak Oct 11, 2018 •

edited

Loading

vkarak commented Oct 11, 2018 •

edited

Loading