enable tests in ga.spec #5

marcindulak · 2020-02-15T20:12:11Z

As mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1709933#c15 we should execute tests during the build of ga, but it seems like the test is specific to mpich

fedpkg/ga/ga.spec

Lines 214 to 220 in 7cf8461

    
           %if %{?do_test}0 
        
           %{_mpich_load} 
        
           cd %{name}-%{version}-mpich 
        
           make check 
        
           cd .. 
        
           %{_mpich_unload} 
        
           %endif

marcindulak · 2020-02-15T20:17:10Z

The mpi modules set the MPI_SUFFIX variable, which includes underscore

grep MPI_SUFFIX /etc/modulefiles/mpi/mpich-x86_64 /usr/share/modulefiles/mpi/openmpi-x86_64 
/etc/modulefiles/mpi/mpich-x86_64:setenv        MPI_SUFFIX    _mpich
/usr/share/modulefiles/mpi/openmpi-x86_64:setenv			MPI_SUFFIX	_openmpi

We should use that in the spec instead of manually use a dash (-) in

fedpkg/ga/ga.spec

Lines 147 to 151 in 7cf8461

    
           pushd %{name}-%{ga_version} 
        
           popd 
        
           for i in mpich openmpi; do 
        
             cp -a %{name}-%{ga_version} %{name}-%{version}-$i 
        
           done

edoapra · 2020-02-16T00:42:08Z

There is an "chicken and egg" issue here (likely to affect in GlobalArrays/ga#154, too)
If you try to run the GA tests with the current ga.spec that uses the --enable-peigs option ,
you end up with test binaries that need the Peigs library to run (you will get an undefined pdspev_ symbol).
The obvious solution is to eliminate the --enable-peigs option, but this will make the GA RPMs useless for NWChem unless NWChem has been built with ScaLapack, instead of Peigs. Luckily,
this should be the case, but I need to test this.

…rpms #5

edoapra · 2020-02-18T17:42:13Z

@marcindulak I have updated my ga and nwchem trees to enable the ga tests.
Successful koji rpm logs for epel7, f30, rawhide
https://koji.fedoraproject.org/koji/tasks?owner=edoapra&state=all

edoapra · 2020-02-28T21:36:32Z

@marcindulak ga 5.7.2 is out
https://github.com/GlobalArrays/ga/releases/tag/v5.7.2
I have updated ga.spec and elempatch.patch accordingly.
2c11609

marcindulak · 2020-03-02T21:33:49Z

It seems like tests on el6/el7/el8 are failing:
https://koji.fedoraproject.org/koji/taskinfo?taskID=42111382
https://koji.fedoraproject.org/koji/taskinfo?taskID=42111340
https://koji.fedoraproject.org/koji/taskinfo?taskID=42113554

edoapra · 2020-03-02T21:46:18Z

I will have a look at it. In the meantime, I have just partially removed the last commit
6ba1a3e

edoapra · 2020-03-02T22:49:22Z

After adding verbose output for tests 6c92d39,
the log shows that gethostbyname is failing.
This could be due to the network on the koji build machines not been correctly configured ... can we check this?
Bye the way, the build works on my RHEL6 box
https://koji.fedoraproject.org/koji/getfile?taskID=42116109&volume=DEFAULT&name=build.log&offset=-4000

===========================================================================
   Communication Runtime for Extreme Scale (comex) 1.1: ./test-suite.log   
===========================================================================
6 of 6 tests failed.  
.. contents:: :depth: 2
FAIL: testing/perf (exit: 1)
============================
Fatal error in MPI_Init: Other MPI error, error stack:
MPIR_Init_thread(467)..............: 
MPID_Init(177).....................: channel initialization failed
MPIDI_CH3_Init(70).................: 
MPID_nem_init(319).................: 
MPID_nem_tcp_init(171).............: 
MPID_nem_tcp_get_business_card(418): 
MPID_nem_tcp_init(377).............: gethostbyname failed, buildvm-10.phx2.fedoraproject.org  (errno 2)

edoapra · 2020-03-02T22:57:20Z

errno=2 should correspond to

TRY_AGAIN
              A temporary error occurred on an authoritative name server.  Try again later.

edoapra · 2020-03-02T23:09:28Z

This is a recent MPICH bug report that could be related
https://src.fedoraproject.org/rpms/mpich/pull-request/2

edoapra · 2020-03-02T23:22:36Z

openmpi tests do work, instead

marcindulak · 2020-03-04T01:38:42Z

I believe I've seen already el6 "gethostbyname failed". Maybe we should exclude mpich from el6 tests? Can you look into el7/el8 failures?

After 6ba1a3e and 6c92d39

	mpich	openmpi
el6	gethostbyname failed	? (slow/hangs)	https://koji.fedoraproject.org/koji/taskinfo?taskID=42159052
el7	? (slow/hangs)	Comparison failed	https://koji.fedoraproject.org/koji/taskinfo?taskID=42117101
el8	OK	Segmentation fault	https://koji.fedoraproject.org/koji/taskinfo?taskID=42159069

Another thing to check/fix - how do we limit the number of cores used by the tests? I would set it to 2.

edoapra · 2020-03-04T02:04:49Z

7fb103a sets NPROCS=2 for tests
Are you sure that all your builds were using commit 6c92d39? I do not see the verbose log in your failed tests for epel7

marcindulak · 2020-03-04T02:17:45Z

epel7 https://koji.fedoraproject.org/koji/taskinfo?taskID=42159068 is the hanging build and it uses make VERBOSE=1 check (that line is in the log). The el6 hanging build is here https://koji.fedoraproject.org/koji/taskinfo?taskID=42160273

I've updated the table above to the right epel7 failing with "Comparison failed" with openmpi.

edoapra · 2020-03-04T02:28:20Z

Thanks for the prompt update
The openmpi version for el7 looks odd (or old) to me. From root.log

openmpi-devel             x86_64   1.10.7-5.el7

The seg fault for el8 could be an issue that was reported for openmpi 4.1.0
open-mpi/ompi#6789
I am trying the workaround

edoapra · 2020-03-04T02:53:46Z

I will have to get rid of the NPROCS=2 commit since some tests do require 4 processes

marcindulak · 2020-03-04T03:27:16Z

I will have to get rid of the NPROCS=2 commit since some tests do require 4 processes

Can we disable those tests? We don't know what cpu resources we have on the koji build servers.

Apparently el7 provides a newer openmpi, but called openmpi3 https://bugzilla.redhat.com/show_bug.cgi?id=1709933#c3 and scalapack.spec uses special logic to handle this exception in naming https://bugzilla.redhat.com/show_bug.cgi?id=1709933#c18

edoapra · 2020-03-04T03:42:44Z

Pushed ed797e7 to limit number of tests and restore NPROC=2

edoapra · 2020-03-04T05:13:01Z

Green light now on epel8 and epel7

edoapra · 2020-03-04T09:02:24Z

@marcindulak please try the latest commit c495ba3

marcindulak · 2020-03-04T20:40:20Z

el6/el7/el8/f33 build now. Should I package this ga for all of them or are we waiting still for some changes in ga needed by nwchem?

edoapra · 2020-03-04T20:43:50Z

I would go ahead an package this ga since it contains all the changes needed by nwchem

marcindulak · 2020-03-04T23:29:37Z

Can you provide build feedback?

https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-86ad2256db
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-f15be59ce7
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-e117e9c57b

edoapra · 2020-03-04T23:57:39Z

Can you provide build feedback?

https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-86ad2256db
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-f15be59ce7
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-e117e9c57b

Done

edoapra · 2020-03-05T01:27:49Z

small ga.spec changes with f978657

marcindulak · 2020-03-05T03:32:18Z

How critical is this change? Could you merge the fedora spec https://src.fedoraproject.org/rpms/ga/tree/master into your repo? We diverged in terms of the changelog, since I'm not merging your github changes into the fedora git repository (I had to release for example 5.6.5-8). We could agree on how we synchronize work between this github and fedora repository.

edoapra · 2020-03-05T15:57:25Z

These changes are not critical at all. They bear zero impact on functionality

edoapra · 2020-03-05T17:50:21Z

Could you merge the fedora spec https://src.fedoraproject.org/rpms/ga/tree/master into your repo? We diverged in terms of the changelog, since I'm not merging your github changes into the fedora git repository (I had to release for example 5.6.5-8). We could agree on how we synchronize work between this github and fedora repository.

8354f3a

marcindulak · 2020-03-05T21:01:47Z

Thanks.

dereferencing_fix.patch seems to be missing. If the patches are not critical, can we hold them back, and rely on them being part of the source of the next ga release instead?

edoapra · 2020-03-05T21:05:36Z

Thanks for spotting this 6dd7868

Yes, we can hold them back and wait for next ga release

marcindulak · 2020-03-05T21:12:42Z

Would you be willing then to go back to 5.7.2-2 in your repo? Fedora will keep making auto commits to the spec master branch and we'll diverge in the changelog.

edoapra · 2020-03-05T21:30:10Z

Back to 5.7.2-2
c3019a0

marcindulak · 2020-03-06T01:13:41Z

I meant rather to sync to what's currently in https://src.fedoraproject.org/rpms/ga/blob/master/f/ga.spec, without additions to the changelog or

fedpkg/ga/ga.spec

Lines 16 to 17 in c3019a0

    
           Patch1:        ga572_version.patch 
        
           Patch2:        dereferencing_fix.patch

Making another build of all epel and fedora's will require 1-2 hours of attention, and I prefer to not spend time on those now.

I think the way to go forward could be to have a branch in your repo called e.g. develop, and keeping your master branch always synced with the fedora master. We could prepare new builds in the develop branch and merge only to your master when things work, then I take your master and write into the fedora master.

marcindulak · 2020-03-07T09:58:30Z

I think this issue can be closed as fixed

edoapra added a commit that referenced this issue Feb 18, 2020

enabled tests; patch to disable failing steps in elempatch.x #5

9807cd1

edoapra added a commit that referenced this issue Feb 18, 2020

switch to latest 7.0.0 commit for compatibility with latest ga 5.7.1 …

b51cf09

…rpms #5

edoapra mentioned this issue Feb 28, 2020

ga.spec 5.7.1 needs to generate configure #6

Closed

edoapra added a commit that referenced this issue Mar 6, 2020

syncing #5

5765626

edoapra added a commit that referenced this issue Mar 6, 2020

syncing #5

4887553

marcindulak closed this as completed Mar 7, 2020

marcindulak mentioned this issue Mar 11, 2020

Please clarify what is "peigs" library GlobalArrays/ga#96

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enable tests in ga.spec #5

enable tests in ga.spec #5

marcindulak commented Feb 15, 2020

marcindulak commented Feb 15, 2020 •

edited

edoapra commented Feb 16, 2020

edoapra commented Feb 18, 2020

edoapra commented Feb 28, 2020

marcindulak commented Mar 2, 2020

edoapra commented Mar 2, 2020

edoapra commented Mar 2, 2020 •

edited

edoapra commented Mar 2, 2020

edoapra commented Mar 2, 2020

edoapra commented Mar 2, 2020

marcindulak commented Mar 4, 2020 •

edited

edoapra commented Mar 4, 2020

marcindulak commented Mar 4, 2020 •

edited

edoapra commented Mar 4, 2020 •

edited

edoapra commented Mar 4, 2020

marcindulak commented Mar 4, 2020

edoapra commented Mar 4, 2020 •

edited

edoapra commented Mar 4, 2020

edoapra commented Mar 4, 2020

marcindulak commented Mar 4, 2020

edoapra commented Mar 4, 2020

marcindulak commented Mar 4, 2020

edoapra commented Mar 4, 2020

edoapra commented Mar 5, 2020

marcindulak commented Mar 5, 2020

edoapra commented Mar 5, 2020

edoapra commented Mar 5, 2020

marcindulak commented Mar 5, 2020

edoapra commented Mar 5, 2020

marcindulak commented Mar 5, 2020

edoapra commented Mar 5, 2020

marcindulak commented Mar 6, 2020

marcindulak commented Mar 7, 2020

enable tests in ga.spec #5

enable tests in ga.spec #5

Comments

marcindulak commented Feb 15, 2020

marcindulak commented Feb 15, 2020 • edited

edoapra commented Feb 16, 2020

edoapra commented Feb 18, 2020

edoapra commented Feb 28, 2020

marcindulak commented Mar 2, 2020

edoapra commented Mar 2, 2020

edoapra commented Mar 2, 2020 • edited

edoapra commented Mar 2, 2020

edoapra commented Mar 2, 2020

edoapra commented Mar 2, 2020

marcindulak commented Mar 4, 2020 • edited

edoapra commented Mar 4, 2020

marcindulak commented Mar 4, 2020 • edited

edoapra commented Mar 4, 2020 • edited

edoapra commented Mar 4, 2020

marcindulak commented Mar 4, 2020

edoapra commented Mar 4, 2020 • edited

edoapra commented Mar 4, 2020

edoapra commented Mar 4, 2020

marcindulak commented Mar 4, 2020

edoapra commented Mar 4, 2020

marcindulak commented Mar 4, 2020

edoapra commented Mar 4, 2020

edoapra commented Mar 5, 2020

marcindulak commented Mar 5, 2020

edoapra commented Mar 5, 2020

edoapra commented Mar 5, 2020

marcindulak commented Mar 5, 2020

edoapra commented Mar 5, 2020

marcindulak commented Mar 5, 2020

edoapra commented Mar 5, 2020

marcindulak commented Mar 6, 2020

marcindulak commented Mar 7, 2020

marcindulak commented Feb 15, 2020 •

edited

edoapra commented Mar 2, 2020 •

edited

marcindulak commented Mar 4, 2020 •

edited

marcindulak commented Mar 4, 2020 •

edited

edoapra commented Mar 4, 2020 •

edited

edoapra commented Mar 4, 2020 •

edited