Help with exclude_dir

[mathomp4]

I'm a newbie to FORD, but I'm trying to apply it to a library I work on, MAPL. I grabbed FORD from Brew and it is 6.1.11

I decided to go for broke and just try it out. I cloned MAPL and made a directory, Ford/ just to keep things safe and then made a basic project.md:

src_dir: ..
graph: false
include: ../include
include: /Users/mathomp4/installed/MPI/gcc-gfortran-11.3.0/openmpi-4.1.3/Baselibs/6.2.13/Darwin/GFTL-1.5/include/v1
include: /Users/mathomp4/installed/MPI/gcc-gfortran-11.3.0/openmpi-4.1.3/Baselibs/6.2.13/Darwin/GFTL-1.5/include/v2
include: /Users/mathomp4/installed/Compiler/gcc-gfortran-11.3.0/openmpi/4.1.3/include
include: /Users/mathomp4/installed/MPI/gcc-gfortran-11.3.0/openmpi-4.1.3/Baselibs/6.2.13/Darwin/include/netcdf

When I did that all was well but then it failed with:

Processing documentation comments...
Correlating information from different parts of your project...
Traceback (most recent call last):
  File "/Users/mathomp4/.homebrew/brew/bin/ford", line 8, in <module>
    sys.exit(run())
  File "/Users/mathomp4/.homebrew/brew/Cellar/ford/6.1.11/libexec/lib/python3.9/site-packages/ford/__init__.py", line 637, in run
    main(proj_data, proj_docs, md)
  File "/Users/mathomp4/.homebrew/brew/Cellar/ford/6.1.11/libexec/lib/python3.9/site-packages/ford/__init__.py", line 584, in main
    project.correlate()
  File "/Users/mathomp4/.homebrew/brew/Cellar/ford/6.1.11/libexec/lib/python3.9/site-packages/ford/fortran_project.py", line 263, in correlate
    container.correlate(self)
  File "/Users/mathomp4/.homebrew/brew/Cellar/ford/6.1.11/libexec/lib/python3.9/site-packages/ford/sourceform.py", line 1256, in correlate
    interface.correlate(project)
  File "/Users/mathomp4/.homebrew/brew/Cellar/ford/6.1.11/libexec/lib/python3.9/site-packages/ford/sourceform.py", line 2090, in correlate
    raise RuntimeError(
RuntimeError: Could not find interface procedure 'strTemplate_' in 'ESMF_CFIOMod'. 
Known procedures are: ['esmf_cfiofilecreate', 'esmf_cfiovarread3d_', 'esmf_cfiovarread2d_',
 'esmf_cfiovarread1d_', 'esmf_cfiovarwrite3d_', 'esmf_cfiovarwrite2d_', 'esmf_cfiovarwrite1d_', 
'esmf_cfiofileopen', 'esmf_cfiofileclose', 'esmf_cfiodownbit3d_', 'esmf_cfiodownbit2d_', 
'esmf_cfiovarreadt3d_', 'esmf_cfiovarreadt3d__', 'esmf_cfiovarreadt2d_', 'esmf_cfiovarreadt2d__', 
'esmf_cfiovarwrite', 'esmf_cfiovarread', 'esmf_cfiovarreadt', 'esmf_cfiovarreadt2', 
'esmf_cfiostrtemplate', 'esmf_cfiodownbit', 'IdentifyDim', 'julday', 'DiffDate', 'err', 
'CFIO_GetMissing', 'addList', 'getList', 'getMaxLenCnt', 'CFIO_DimInquire', 'GetDateTimeVec', 
'GetBegDateTime', 'CFIO_parseIntTime', 'strToInt', 'CFIO_Open', 'CFIO_Close', 'CFIO_PutIntAtt', 
'CFIO_PutRealAtt', 'CFIO_PutCharAtt', 'CFIO_GetAttNames', 'CFIO_AttInquire', 'CFIO_GetIntAtt', 
'CFIO_GetRealAtt', 'CFIO_GetCharAtt', 'CALDAT', 'ParseTimeUnits', 'CFIO_SPutVar', 'CFIO_SGetVar', 
'CFIO_GetVar', 'CFIO_PutVar', 'GetDateInt8', 'GetDateInt4', 'strTemplate_', 'GX_', 'GetDate', 
'ESMF_CFIOGridCreate', 'ESMF_CFIOGridSet', 'ESMF_CFIOGridGet', 'ESMF_CFIOGridDestroy', 
'ESMF_CFIOVarInfoCreate', 'ESMF_CFIOVarInfoSet', 'ESMF_CFIOVarInfoGet', 'ESMF_CFIOVarInfoDestroy', 
'iNodeDestroy', 'rNodeDestroy', 'cNodeDestroy', 'addlist', 'ESMF_CFIOCreate', 'ESMF_CFIOSet', 
'ESMF_CFIOGet', 'ESMF_CFIODestroy', 'ESMF_CFIOSdfFileCreate', 'ESMF_CFIOSdfFileOpen', 
'ESMF_CFIOSdfFileClose', 'ESMF_CFIOSdfVarWrite', 'ESMF_CFIOSdfVarRead', 'ESMF_CFIOSdfVarReadT']

This is a bit weird since, well, strTemplate_ is in that list of known procedures! (It's next to GX_ 6 lines up from the end.)

But, fine, this is a bit of the library (MAPL_cfio) I don't care much about, so I'm fine just ignoring it. The issue is, I can't seem to exclude it. If I add to project.md:

exclude_dir: ../MAPL_cfio

it doesn't work and errors out the same way. I then tried:

exclude_dir: MAPL_cfio

But nope. I even tried a full path (even though the docs say "Don't do that.") but nope.

The only way I can get past this error is to just move the directory out of the source entirely.

So what is the right way to use exclude_dir?

[mathomp4]

Know how I said it worked fine after I moved away MAPL_cfio, well...it didn't quite:

...
jinja2.exceptions.UndefinedError: 'str object' has no attribute 'meta'

So...more for me to learn/debug! But I'm still interested in how to use exclude_dir.

[mathomp4]

Also, I did have to fix a few programs in MAPL (they had an end at the end but no program foo at the top). But it seemed to be happy with everything else!

[ZedThree]

You're not doing anything wrong really, you've just hit a bit of a sharp edge: the Ford directory is below your src_dir, which means the Ford/doc/src directory, which contains copies of your project's source code, gets processed by ford. So if you run ford once, then decide to exclude a directory or file and run it again, the copy gets picked up, and that hits your original error.
I thought that we delete the output directory so we've got a clean environment, but either that's not working correctly or I've misremembered.

So for now, I suggest deleting Ford/doc, and then adding the following to project.md:

exclude_dir:
    ../Ford
    ../MAPL_cfio

to exclude the ford output directory.

Really, we should be a bit more clever ourselves and exclude the ford output directory automatically! I'll see if I can make that happen.

The error with strTemplate_ is absolutely a bug, and I've now found and fixed that. I'll try and get that sorted and released tomorrow.
You've also pointed me towards some error messages that could be improved, so I'll also make those a bit better too!

I've not yet got to the bottom of the jinja2 error. I think I'll have to cut down the file by hand to debug it. My suspicion is that we're not processing a type-bound procedure correctly and just leaving it as a string somehow.

[mathomp4]

@ZedThree Well, I never even got to the Ford/doc stage...but now I can with a couple things:

1. I added ../Ford to exclude_dir
2. I stopped using blank lines to make my project.md pretty (that wasted a lot of time trying to figure out why things weren't working)
3. I moved to use NAG Fortran's fpp to preprocess.

This last one is fun. I'm running on a MacBook and without any module loaded, cpp is evaluating to clang cpp. That goes boom because I guess the default cpp options aren't right for clang cpp.

If I explicitly point to GNU cpp with:

preprocessor: /Users/mathomp4/installed/Core/gcc-gfortran/11.3.0/bin/cpp -traditional-cpp -E

Then I get the jinja2 error. But with NAG fpp it seems to work. 🎉 The weird thing is, @tclune and I looked at the preprocessed output from both cpp and fpp and they were nigh-identical (ignoring whitespace and comments). Not sure why GNU cpp is having a bad day.

So for now I'll work with NAG fpp but I can help with anything you'd like to run. Now to make another issue just because I want a feature I see in other docs.

[ZedThree]

I've tracked down the jinja2 error now too. cpp is choking on a redefinition of _UNUSED_DUMMY which results in ford trying to parse the un-preprocessed version, and tries to interpret procedure :: ___SUB(get_var,int32,0) as ordinary Fortran.

fpp is probably more happy with the repeated macro definition which is why it works.

Actually looking into this further, if preprocessing reports anything to stderr then we don't use the preprocessed result, which is maybe not exactly what we want.

[mathomp4]

I've tracked down the jinja2 error now too. cpp is choking on a redefinition of _UNUSED_DUMMY which results in ford trying to parse the un-preprocessed version, and tries to interpret procedure :: ___SUB(get_var,int32,0) as ordinary Fortran.

fpp is probably more happy with the repeated macro definition which is why it works.

Actually looking into this further, if preprocessing reports anything to stderr then we don't use the preprocessed result, which is maybe not exactly what we want.

Ohh. Yeah. We probably do repeatedly re-define it and since things still worked, we never took a hard look. I'll take a look on our end and see if we can't just clean that up in MAPL with more careful #ifdef/#undef/#endif and do things cleaner.

[mathomp4]

@ZedThree I added a handful of fixes to fix the redefinitions and I can now run FORD with no warnings and no jinja2 errors (as long as I exclude the MAPL_cfio and Ford directories).

So I suppose I can now close this and start looking at #422 more closely..

[ZedThree]

I think I've now fixed the issues that prevented ford working on the MAPL_cfio directory. You can try out the master branch if you don't mind using the development version. There's a couple more PRs to merge before we release the next version, which should be within the next week or so.