-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Substitution of NAG libraries #1
Comments
@jloizu can you identify which NAG routines are causing the problem? |
Here is an un-authoritative list of subroutines and the NAG routines inside them. Clearly we have our work cut out for us. I'm pushing a branch to the repo today for replacement of the NAG routines. |
I created a branch no_nag and replaced C05NDF and C05PDF in newton.h with hybrd and hybrj from MINPACK. I can compile it without any errors. Further debugs need to compare the two versions. I will talk with Stuart tomorrow. There are a lot of Fortran source code here. We can replace NAG subroutines with corresponding source codes. |
@StellaratorCoils I've also done similar on a branch called NAG_REPLACE. Which I've pushed. Specifically the routines in casing and dforce. We should merge and choose one branch name to develop under. |
The routine C06FUF is doing an FFT. Usually if you want to do FFT's with free software, I'd suggest going with FFTW which is available on the PPPL cluster. I would suspect IPP has this as well since the GENE code also takes advantage of this library. FFTW is VERY fast compared to other libraries so I don't really see any downside to using it. Thoughts? |
I'd be happy to use FFTW for this. Would you recommend I start a new
branch to make the change, or work within an existing one?
…______________________________________________________________________
Joshua Breslau
Princeton Plasma Physics Laboratory Tel.: (609)
243-2677
Office: A-129
E-mail: jbreslau@pppl.gov
Princeton, NJ 08543
Fax: (609) 243-2662
On Mon, Jul 24, 2017 at 2:59 PM, Samuel Lazerson ***@***.***> wrote:
The routine C06FUF is doing an FFT. Usually if you want to do FFT's with
free software, I'd suggest going with FFTW <http://www.fftw.org> which is
available on the PPPL cluster. I would suspect IPP has this as well since
the GENE code also takes advantage of this library. FFTW is VERY fast
compared to other libraries so I don't really see any downside to using it.
Thoughts?
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/Ac620BFk2feeNrU_SFLFDWS742P97NSlks5sROmNgaJpZM4OeLaa>
.
|
@jbreslau Go ahead and work in the NAG_REPLACE branch. I think preset.h is the first place to attempt to implement it. For sanity sake I'd probably use the legacy interface. |
I'm getting build errors for this branch because mpif90 can't find the hdf5
module (I loaded the default hdf5-serial in the intel environment). What
module environment do you use to build the code?
Thanks,
-Josh
…______________________________________________________________________
Joshua Breslau
Princeton Plasma Physics Laboratory Tel.: (609)
243-2677
Office: A-129
E-mail: jbreslau@pppl.gov
Princeton, NJ 08543
Fax: (609) 243-2662
On Mon, Jul 24, 2017 at 3:23 PM, Samuel Lazerson ***@***.***> wrote:
@jbreslau <https://github.com/jbreslau> Go ahead and work in the
NAG_REPLACE branch. I think preset.h is the first place to attempt to
implement it. For sanity sake I'd probably use the legacy interface.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/Ac620AD7vxxEVGScaKg1KD9tss3QsfQtks5sRO8lgaJpZM4OeLaa>
.
|
I've committed a revision to the NAG_REPLACE branch that replaces the C06FUF calls in numrec.h with the fftw/3.3.4 equivalents. I haven't completely eliminated the NAG call (it's still in preset) because I first need to make sure the trig coefficients computed there aren't used anywhere else. The revised version runs slightly faster and produces what appear to me to be very slightly different results on the test case l2stell.sp. Someone with more experience with SPEC might want to take a look at the output and verify that the difference is acceptable... |
I am currently writing routines to make precise (quantitative) tests to, for example, check that two outputs are exactly the same in the case they should be the same, or to give a degree of reproducibility in the case where small differences are expected to arise. This should take me a few more days. I will keep you up to date. |
I have finished removing all references to C06FUF from my working copy of NAG_REPLACE, but the system won't let me commit the changes; I get this message: % svn commit -m '...' I have tried updating, but the message persists. Does anyone know how to get around this issue? |
@jbreslau You may need to use git rather than svn. Something like
|
I can't seem to use git commands for my working version; it isn't recognized as a git repository. |
@jbreslau You need to module load git for it to work properly. How did you get the code in the first place? |
I believe I followed the instructions on Github for accessing repositories
using svn.
…______________________________________________________________________
Joshua Breslau
Princeton Plasma Physics Laboratory Tel.: (609)
243-2677
Office: A-129
E-mail: jbreslau@pppl.gov
Princeton, NJ 08543
Fax: (609) 243-2662
On Wed, Jul 26, 2017 at 10:37 AM, Samuel Lazerson ***@***.***> wrote:
@jbreslau <https://github.com/jbreslau> You need to module load git for
it to work properly. How did you get the code in the first place?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/Ac620BEGcVPbU7eO3pKkl98G5R1OZvA2ks5sR086gaJpZM4OeLaa>
.
|
I've found a workaround and commit the changes to the NAG_REPLACE branch. It looks like there's just a handful of C06*** NAG routines remaining, all in numrec.h subroutines (slowft and invft) that never get called. Do these NAG calls need to be replaced, or can we just delete (or at least comment out) the subroutines? |
@jbreslau For the intel compiler I did have to module load fftw/3.3.4 as the fftw module was not .f03 compatible. The code runs for the l2stel example but for some reason the iteration numbers and timing isn't being displayed correctly.
|
Yes, I debugged the changes using fftw/3.3.4 under intel. I'm aware of the formatting bug; I'll fix it today if I get a chance. |
I've looked into the formatting issue. The offending lines are being written by subroutine fcn2() in newton.h. The way that subroutine is written, the values of nFcalls, nDcalls, and lastcpu will always be undefined (and apparently initialized to zero) when the write statement is reached. I think this should be fixed by someone who knows what this subroutine is supposed to be doing. |
Is this formatting issue still a problem? Please let me know what branch. |
Can anyone find a routine for solving a system of real symmetric linear equations? The routine mp00ac calls the NAG routine F04AEF, which does not exploit symmetry. Note that the matrix is NOT positive definite. Otherwise I could use NAG F04ABF. At present, most of the computation time (according to my way of computing the timing) is taken by mp00ac. I would think that a routine that exploits symmetry would be faster. Josh: could you, when you have time, please make it a priority to replace F04AEF with a routine for solving real, symmetric linear equations with multiple right hand sides? |
It's not a formatting issue. It's a bug in fcn2. The time elapsed is not
computed correctly because the variable storing the starting time is never
initialized.
…______________________________________________________________________
Joshua Breslau
Princeton Plasma Physics Laboratory Tel.: (609)
243-2677
Office: A-129
E-mail: jbreslau@pppl.gov
Princeton, NJ 08543
Fax: (609) 243-2662
On Tue, Aug 1, 2017 at 12:53 PM, Dr. S.R. Hudson ***@***.***> wrote:
Is this formatting issue still a problem? Please let me know what branch.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/Ac620GMd-8LVo54nP1KnfxOHWFpoP4Foks5sT1f2gaJpZM4OeLaa>
.
|
There should be something in LAPACK. I'll check it out.
…______________________________________________________________________
Joshua Breslau
Princeton Plasma Physics Laboratory Tel.: (609)
243-2677
Office: A-129
E-mail: jbreslau@pppl.gov
Princeton, NJ 08543
Fax: (609) 243-2662
On Tue, Aug 1, 2017 at 12:58 PM, Dr. S.R. Hudson ***@***.***> wrote:
Can anyone find a routine for solving a system of real symmetric linear
equations? The routine mp00ac calls the NAG routine F04AEF, which does not
exploit symmetry. Note that the matrix is NOT positive definite. Otherwise
I could use NAG F04ABF.
At present, most of the computation time (according to my way of computing
the timing) is taken by mp00ac. I would think that a routine that exploits
symmetry would be faster.
Josh: could you, when you have time, please make it a priority to replace
F04AEF with a routine for solving real, symmetric linear equations with
multiple right hand sides?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/Ac620IXD2kOYd-Qkwazx0ZwM_WwNErNoks5sT1k9gaJpZM4OeLaa>
.
|
I just commited version 109, which replaces both calls to F04AEF from mp00ac.h in the NAG_REPLACE branch with calls to LAPACK routine DSYSVX, the expert driver for iterative linear solves of systems with general real symmetric matrices. For the l2stell test case, this speeds up execution by over 30%. There is an equivalent LAPACK routine for symmetric positive definite matrices, which should probably be substituted for F04ABF in the same routine in the interest of de-NAG-ification, but I wouldn't expect much speedup from that. |
I am having trouble checking out NAG_REPLACE. I get the message |
I think it's because that you made a lot changes recently and some of them were committed to your local branch NAG_REPLACE. You can try this to compare your local branch with the public branch.
git diff NAG_REPLACE origin/NAG_REPLACE
Then you will see the differences and decide whether you want to delete your local stuffs.
If you want to, you can try
git pull --hard origin/NAG_REPLACE
… On 1 Aug 2017, at 5:30 PM, Dr. S.R. Hudson ***@***.***> wrote:
I am having trouble checking out NAG_REPLACE. I get the message
"Your branch is ahead of 'origin/NAG_REPLACE' by 15 commits.
(use "git push" to publish your local commits)"
and I don't see any of the changes that have been made. I have made no changes, at least not intentionally. Perhaps I need to delete my local branch and re-checkout NAG_REPLACE?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
I THINK I HAVE MESSED UP NAG_REPLACE!! |
I can have a look. And even it's messed, we can simply revert. Don't worry! |
Problem solved (thanks to Caoxiang and Josh). |
I just tried to compile xspec on NAG_REPLACE, and I get the following error: |
you could type
|
Ok, module load fftw/3.3.4 works. What about |
Josh: I fixed the counters and timing variables in newton, fcn1 and fcn2. Please pull NAG_REPLACE. |
Everything works fine for me when I typed |
Thanks, that fixed the problem.
I pushed a fix to the Makefile to link the fftw library when building
dspec; it should work now.
…______________________________________________________________________
Joshua Breslau
Princeton Plasma Physics Laboratory Tel.: (609)
243-2677
Office: A-129
E-mail: jbreslau@pppl.gov
Princeton, NJ 08543
Fax: (609) 243-2662
On Wed, Aug 2, 2017 at 6:20 PM, Dr. S.R. Hudson ***@***.***> wrote:
Josh: I fixed the counters and timing variables in newton, fcn1 and fcn2.
Please pull NAG_REPLACE.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/Ac620G-fB_wDk1f6jL4fbDvGOCNbj0Lrks5sUPY4gaJpZM4OeLaa>
.
|
I notice that there are some more f04aef calls in tr00ab.h. Can any assumptions be made about the symmetry or lack thereof of the matrix they are factoring? The linear solver could be replaced with DGESVX, DSYSVX, or DPOSVX depending on its properties... |
Also, do we have a test case where Lposdef.eq.1, so I can test the replacement of F04ABF in mp00ac.h with the equivalent LAPACK call? |
|
Josh: I pushed my change to tr00ab that comments out (using ifdef LSPARSE) all the code related to the real-space calculation of the transformation to straight fieldline coordinates (which provides the rotational-transform). Please pull . . . |
Please point me to the input file that's giving you problems with the
gaussian abscissae, and I'll try to debug.
…______________________________________________________________________
Joshua Breslau
Princeton Plasma Physics Laboratory Tel.: (609)
243-2677
Office: A-129
E-mail: jbreslau@pppl.gov
Princeton, NJ 08543
Fax: (609) 243-2662
On Thu, Aug 3, 2017 at 5:12 PM, Dr. S.R. Hudson ***@***.***> wrote:
Josh: I pushed my change to tr00ab that comments out (using ifdef LSPARSE)
all the code related to the real-space calculation of the transformation to
straight fieldline coordinates (which provides the rotational-transform).
Please pull . . .
Also, I am getting lots of errors with the gaussian abscissae construction
. . . Please check and/or let's discuss tomorrow.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/Ac620MgWDxIgrmN_xgh_VUw7-SrCANkCks5sUjfTgaJpZM4OeLaa>
.
|
Josh: please replace the eigenvalue / eigenvector routine in hesian, F02EBF, at your earliest convenience. After doing so, Joaquim should merge the NAG_REPLACE branch with master so that we can use the same code to examine the eigenvalues of the "hessian". |
I had already started looking into this, but found that the routine was not
called for the test case. Am I right in thinking I just need to set Lcheck
to 5 in the input file to activate it?
…______________________________________________________________________
Joshua Breslau
Princeton Plasma Physics Laboratory Tel.: (609)
243-2677
Office: A-129
E-mail: jbreslau@pppl.gov
Princeton, NJ 08543
Fax: (609) 243-2662
On Thu, Aug 10, 2017 at 8:56 AM, Dr. S.R. Hudson ***@***.***> wrote:
Josh: please replace the eigenvalue / eigenvector routine in hesian,
F02EBF, at your earliest convenience. After doing so, Joaquim should merge
the NAG_REPLACE branch with master so that we can use the same code to
examine the eigenvalues of the "hessian".
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/Ac620JNrg-haKwuVtTy5oTvLENZXQixSks5sWv4agaJpZM4OeLaa>
.
|
No, it is not to do with Lcheck. Lcheck=5 will test the analytic construction of the hessian with a finite difference estimate. To call F02EBF, choose either LHevalues = T or LHevectors = T. |
It looks like Sam already replaced F02EBF in the NAG_REPLACE branch with
the LAPACK routine DGEEV on July 24th.
…______________________________________________________________________
Joshua Breslau
Princeton Plasma Physics Laboratory Tel.: (609)
243-2677
Office: A-129
E-mail: jbreslau@pppl.gov
Princeton, NJ 08543
Fax: (609) 243-2662
On Thu, Aug 10, 2017 at 9:39 AM, Dr. S.R. Hudson ***@***.***> wrote:
No, it is not to do with Lcheck. Lcheck=5 will test the analytic
construction of the hessian with a finite difference estimate.
To call F02EBF, choose either LHevalues = T or LHevectors = T.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/Ac620GmvntWvrAR72gEN-6FkxE_ggC7jks5sWwgqgaJpZM4OeLaa>
.
|
That was fast. Well done Josh! I knew I could depend on you to get this done quickly! Joaquim: would it be appropriate to merge NAG_REPLACE with master? |
I have not worked on that branch (NAG_REPLACE). This means, if you guys think that it is a stable branch, and that it is a good moment to merge it into master, then make a "pull and merge request". Then, I will pull that branch, and make some tests to be sure that it compiles, runs testcases, and produces "the same" results as the current master branch. Only then I would merge it. I can do this today, once you open the pull request (which is essentially a sign that you are waiting for people to pull, test, and merge). |
OK, I had to modify the Makefile to be able to compile in IPP (so far I could only make it work on the local workstation, somehow the fftw gives me problems on the cluster - I need to explore this further). I tested the l2stell.sp case with both branches (master and NAG_REPLACE). Applying my comparison routine to both outputs, I get
and thus I consider that in principle the two branches can be merged after I solve the compilation issue in the IPP cluster and the conflicts between the two branches are resolved (has to be done by hand, but that should be quite easy). I am going tomorrow morning on holiday, so I am not sure I can solve the compilation issue. Since the master versus NAG_REPLACE comparison was positive, I think you can go on if you whish to merge and not wait for 3 weeks. Just one comment: the l2stell.sp run becomes slower with the NAG_REPLACE branch. I used the timing option of @SRHudson and got total of 23sec (master) versus 35 (nag), the difference being in ~10 extra seconds spent in ma00aa. |
I made what i thought were several speed improvements to ma00aa, which later was merged with master. So it makes sense that master is faster than the NAG_REPLACE branch. |
I see what you mean. It may be more tricky than I thought, indeed. Unfortunately in this case GIT says that it cannot do it automatically. We have to do it by hand, I think. @zhucaoxiang , @jbreslau , @lazersos, what do you think? |
I just saw (in the pull-request analysis) that ma00aa is NOT among the "conflicting files". So we only need to take care of the other files. |
The NAG libraries often cause problems with portability of code. NAG is not freely available. Simple changes in release number (R23 vs. R24) can break codes. It is proposed that we fork a branch of SPEC where the NAG libraries will be replaced one by one. Once this code has been sufficiently tested it can be merged with the master branch. Where possible routines will be replaced by BLAS or LAPACK libraries as highly optimized versions of these libraries are available and we can free include unoptimized versions with SPEC so that at least basic functionality is always possible.
The text was updated successfully, but these errors were encountered: