[Fortran fpm] Add support for command-line specification of compiler options

[urbanjost]

The h-fpm version allows compiler options on the command line. I would propose allowing the same capability but using the
syntax

fpm build [NAME(S)] [-release] -- ARGS

where ARGS could be multiple compiler options.

• Any feedback on use cases for the feature, and comments on the h-fpm option would be informative as to whether this option should be added or not.
• should the options replace all the options sans the ones required for the build (ie. -J, -I, ...) or be in addition to the default options?
• I think the documentation should indicate that this potentially breaks a package being self-describing, as it would be problematic to have required options on external dependencies. It would be useful for building a top-level package with different options. Options for profiling come to mind, as well as external system libraries. Other options are being discussed to handle the compile and load options as something provided in the manifest file but there will probably always be a case for supplying various options while developing a package.
• The h-fpm version hashes the options to create a unique pathname for each combination of switches. That is an option here ; as well as requiring a user-supplied name ;or still being built in the standard location. Not sure what the general sentiment is about that. Using the hash resolves all kinds of collision problems but could be hard to remember and duplicate unless the hash is reversible and tagged with a name for reuse.
• should the options only apply to an application and not external dependencies and files in src/? Should there be an option to control what it applies to? Would a rebuild of all package components be required?
• should the syntax be the same in both the f-fpm and h-fpm packages or is it an advantage to allow for two syntaxes (and underlying implementation differences) for exploring more possibilities?

[everythingfunctional]

Just a note, if you tie the build directory to the compiler options, as I've done for h-fpm, then you have to pass the same options to fpm run, or it will (re)build and run a different version.

I don't know that the CLI for h-fpm and f-fpm must be identical. I think a little bit of experimentation in this area can help us find a more optimal solution. Ideally they should be similar and converge to the same, but don't have to be identical initially.

[awvwgk]

We were discussing using another command for this over at #220. There is cargo-rustc for Rust, which essentially allows to build using the syntax you proposed. A similar command for fpm could be fpm-compile.

To solve the issue with the arguments for the other build modes, fpm-compile could allow to build and run targets just like the fpm-run and fpm-test:

fpm compile [--run|--test target] [--compiler name] [-- [args]...]

In contrast fpm-build would require a defined profile in the package manifest.

[everythingfunctional]

I like it, but it's missing two aspects.

1. Specifying the runner (like Test and executable runner options #221)
2. Specifying the target arguments

For 1, I would just add the --runner argument. For 2, I think using the [-- [args]] for that, and then adding a --flags argument that takes a string with all the flags might be the better way to go. In fact, I may switch to that for #220 (or a future PR, I haven't decided).

[urbanjost]

I have a version ready that currently assumes
#238 and #243 are implemented. It would be trivial to remove
the dependencies but it logically follows #238 and the Haskell
version of --flag; and becomes much cleaner if the --dir option
discussed in #244 (but not implemented in my current implementation)
is accepted.

The version I have ready just applies the options to the debug version
unless --release is specified in which case it applies to the release
version. I do not consider that ideal but it is useful and allows for
builds that would otherwise require the support of options in the fpm.toml
file or custom build scripts, such as allowing for X11 libraries or other
external libraries.

Lean towards there being two options, one for compiles and one for
loads but for now matching the current proposal to some extent (but that
automatically builds in an alternate directory) seems needed.

[urbanjost]

lternate compilers. This is an alpha version for furthering discussion
on how to implement these features as discussed here so I do not think
it is appropriate yet for an actual pull request, but I would like some
trials and feedback on the interface. The code itself might need some
refactoring depending on how that goes. So, if interested

There are three new switches

   --build_name NAME, -B          has to be Fortran name
   --compiler COMPILER_NAME, -C   has to be Fortran name
   --flags ARGUMENTS, -F

Essentially, if you use --compiler and/or --flags you have to specify a
build name with the --build_name NAME switch.

The names release and debug are reserved.

You can only specify the custom build flags on the "build" subcommand.
You then use the --compiler and --build_name on "run" and "test" to match
the build.

So this is just a single file you have to build with gfortran:

   # get the single-file build file
   wget http://www.urbanjost.altervista.org/REMOVE/ffpm.f90
   # compile and name "ffpm" somewhere in your path
   gfortran ffpm.f90 -o $HOME/.local/bin/ffpm
get an fpm project. For example
```bash
git clone https://github.com/urbanjost/M_CLI2

cd M_CLI2

# assuming `ffpm` is in your path from above:

# build the package with the name "ONE" with the nvfortran compiler:
ffpm build --compiler nvfortran --flags '-Mbackslash' --build_name ONE

# build it again with the ifort compiler:
ffpm build --compiler ifort

# build it again normally
ffpm build

# and again as a release version
ffpm build --release

# build with gfortran but with no user-specified compiler switches
ffpm build  --build_name DEFAULT
# to run that
ffpm run --build_name DEFAULT

# profile with gfortran
ffpm build --build_name PROFILE --flags '-pg'
ffpm run   --build_name PROFILE
ffpm run   --build_name PROFILE  --runner gprof

# build with ifort(1) compiler and profiling options
ffpm build --compiler ifort --build_name PROFILE --flags '-p'     # build
ffpm run   --compiler ifort --build_name PROFILE                  # run
ffpm run   --compiler ifort --build_name PROFILE  --runner gprof  # profile

Ideally I would like to see this functionality available via the TOML file as discussed in other issues but this gives a working model for getting an idea of what some of the issues will be. RIght now for compilers other than gfortran it is assuming the modules are placed in the directory specified via -module DIRNAME which is an issue that needs a more generic solution. Hopefully this will further the discussion a bit more and help lead to a real pull request.

[urbanjost]

So after trying a lot of variants, I think:
There are two special build suffix names (assuming the compiler name is the prefix) that are used as build directories that are reserved. They can only be used if the compiler is supported and take no user-supplied link options. They are only allowed if the compiler is supported. The one is debug, which is the default. The other is release, which is used if you do a --release.
There is a compiler switch --compiler whose default is set with the environment variable FPM_COMPILER else it is gfortran.

If you supply any user-specified compiler options with --flags you must use the build command to build and you must specify a user-supplied build name via --build_name. You can then use --compiler and --build_name with a matching name on the run and test commands to run executables built with those options. On build options after -- can be options equivalent to --flags or maybe should be "loader only" options. If your compiler name is not known and the switch for specifying where output files to be placed is not "-module $NAME" then you must supply and environment variable called FPM_BUILD_OUTPUT which has a space placed between it and the build directory name unless it ends in an = in which case the filename is appended directory to it.

So current usage would be unchanged.
setting FPM_COMPILER to a supported compiler name would work just like the default except put files in build/${COMPILER}_debug/ and _release. Or, on build, run, and test you could add --compiler $FPM_COMPILER. If you want to specify your own compiler switches (assuming internally used option names are "universal" like -o, which is generally but not completely true) you can add a -flags option and a build name. This ignores specifying anything in the TOML file which I support. These switches (--compiler --build_name --flags, -- ARGS) and environment variables would be marked as provisional and documented completely only in "fpm help compiler" with a prominent note saying they are intended primarily to handle build requirements not yet internalized in fpm and are subject to change. The names are so long I think short names are appropriate, all capitalized to emphasize they are provisional (--compiler,-C; --build_name,-B, --flags,-F). If no strong opinions not to after waiting for this week I will put in a pull request for a prototype that meets that description after a week that should follow PR #239.

[awvwgk]

There is a compiler switch --compiler whose default is set with the environment variable FPM_COMPILER else it is gfortran.

I would suggest to stay as consistent with other build systems. The established default for the compiler environment variable is FC across several build systems and meta build systems.

If your compiler name is not known and the switch for specifying where output files to be placed is not "-module $NAME" then you must supply and environment variable called FPM_BUILD_OUTPUT which has a space placed between it and the build directory name unless it ends in an = in which case the filename is appended directory to it.

The environment variable FPM_BUILD_OUTPUT doesn't capture this purpose in my opinion. I have seen get_module_outdir_args in the meson implementation of the Fortran compiler. CMake uses Fortran_MODULE_DIRECTORY to set the directory path in the build file. From the name FPM_BUILD_OUTPUT I wouldn't expect to set the module output directory for the Fortran compiler.

The names are so long I think short names are appropriate, all capitalized to emphasize they are provisional (--compiler,-C; --build_name,-B, --flags,-F).

-C is expected in many build systems (make, ninja, meson, ...) and other programs as well (git, ...) to switch the directory, I would discourage using this short flag for the compiler. -B is the CMake flag to select the build directory output, so this might work as expected. I wonder if we should prefer underscores or hyphens in flags? Or might there a better synonym for build name we could use here instead?

[urbanjost]

For whatever reason the browser is not letting me edit the post. I wanted to update it with some of the changes as a working document. Using FC instead of making up a "new" name seems like a particularly good one. Since hyphens are part of the "space-hyphen value" of POSIX commands I find the underscore useful for names, and then the names can be used directly in the code as well as hyphen is allowed in Fortran variable names; but that is probably as much personal taste as anything else as that argument has been going on at least as long as which text editor is "best" and which programming language is "best". When I get to a machine that works I will try to capture this all into a second take. Thanks for the quick response. I strongly want some of this to go into the TOML/configuration files so packages can be shared more readily but there will always be some need for directly specifying the switches and it lets us more experience with the issues of multiple compiler support On that note, I THINK your input on M_CLI2 plus working through several compiler bugs has the latest version of M_CLI2 passing its tests with nvfortran, ifort, and gfortran8.3+. Each compiler has it's pros and cons and bugs/features; but considering all that I think it is quite portable to modern Fortran compilers. It's predecessor was F77 and can on everything from SCOPE 2/UNICOS/SunOS/Solaris/HP-UX/AIX/NetBSD/Linux/Tru64/... so it was interesting that some of the modernizing made that a bit harder than I expected. It got cluttered by making a stand-alone version of; I hope to streamline it and add a bit of OOP to it in the future, but not in a hurry as long as it functions as intended.

[urbanjost]

Anyone have any favorite compiler options for compilers other than gfortran that would be used as a seed for the default and --release compiler switches?

Any suggestions for the --build_name key name that seem better or shorter? --tag or --ID perhaps?

I added a number of other compiler names for testing based on the list in #223 but have no great way of testing the majority of them.

A lot of the vendors used to support on-line access to test machines (manufactures and compiler vendors) but I do not see any remaining ones. Something for the wish list would be web interfaces for various compilers where you could give the URL of an fpm package and it would do a build of it and show the resulting output. Probably asking to much for more than a build, but just the build would be nice. After all, legend says some unnamed companies have the policy "It compiled, ship it!".

[urbanjost]

still oscillating on a separate subcommand like compile or custom versus build, test, run allowing customization if a custom build name is provided. A actually like the switches better but the interface is cleaner (as is documentation) if it a separate command. Something in-between where the custom/compile command just does a custom build and the test and run subcommands remain the same except they have a -ID switch added? If the -ID switch is present run and test would not do an automatic build even if the executables are not built. Everyone would have the --compiler switch.

fpm compile --ID xxxxx  --compiler xxxx -flag xxx  .[regular_build_options] -- LOADER_OPTIONS

maybe no --flag option but options allowed after the compiler name, eg. "--compiler 'myf90 -traceback -pg' "
which might be easier for a user to use using environment variables , or if the build,run,test commands could have a tag added to the name which would act like the --ID switch?

fpm build,mytest=' -pg -O3' ...
fpm run,mytest ...
fpm test,mytest ...

If ID is not provided

[urbanjost]

Provisional Custom Compiles

So, after going through the suggestions and some testing with similar h-fpm features this is
what I am proposing. Looking for feedback and whether this is ready to
propose as a PR (Pull Request).

Add compiler switch --fc

run, test,build, and a new command called compile have a switch called
--fc that sets the Fortran compiler name. The default is to use the
value of the environment variable FC. If it is not set the name gfortran
is used.

FC is apparently a commonly used environment variable for the compiler, is
short, and allows for specifying compilers for other languages like C (ie. -cc gcc)
That is part of the reasons to use --fc instead of --compiler.

new command "compile"

The compile command is like the build command except it does not have
the --release parameter. In addition it has the new options

--ID NAME               user-supplied name for the custom build. Required
--flags ARGUMENTS, -F   custom compiler flags

-ID switch added to test and run to match name on compile command

The ID names release and debug are reserved and cannot be used.
--ID is also added to run and test. If present on those commands no
automatic builds occur and the matching build created with compile is used.
The name custom is currently an alias for compile. I prefer custom and
am looking for feedback on it.

preliminary support for alternate compilers for testing

A skeleton was started for standard debug and release builds that allows
for compilers other than gfortran. I now have access to ifort
and nvfortran and it works with at least simple cases for those
compilers. Looking for someone with access to other compilers to help
flesh that out.

SUMMARY

So default usage is unchanged from the previous version. By simply setting FC you can use it like the
previous version with other compilers for default builds. Other than the compiler
switch which I suspect would be used infrequently versus the FC compiler the
only change to test and run is the --ID switch. The ID must be a name
suitable as a Fortran variable. It allows for simple custom builds and loads.

How does that sound?

Instructions for testing before PR is submitted

# single-file program for testing
wget http://www.urbanjost.altervista.org/REMOVE/ffpm.f90
# or pull from webpage of same name

# build and call ffpm. Example. Change output name as appropriate to put
# into your search path
gfortran ffpm.f90 -o $HOME/.local/bin/ffpm

sample testing

# get a test package or use your own
if [ ! -d M_CLI2 ]
then
   git clone https://github.com/urbanjost/M_CLI2
fi
cd M_CLI2

set -x

# simple custom build
ffpm compile --fc nvfortran --flags '-Mbackslash' --ID ONE
ffpm run -ID ONE

# default build using ifort
ffpm build --fc ifort

# default build
ffpm build
# default release build
ffpm build --release

# not sure if this should do a default build or build with gfortran with
# no other switches
ffpm build --fc gfortran
# will not allow reserved names debug and reserve even for this case
ffpm compile --fc gfortran --ID debug

# profile with gfortran
unset FC
ffpm compile --ID PROFILE  --flags '-pg'
ffpm run     --ID PROFILE
ffpm run     --ID PROFILE  --runner gprof

# profile with --fc switch and ifort
ffpm compile --fc ifort --ID PROFILE --flags '-p'   # build
ffpm run     --fc ifort --ID PROFILE                  # run
ffpm run     --fc ifort --ID PROFILE  --runner gprof  # profile

# profile using FC variable and appropriate flags for that compiler
export FC=ifort
ffpm compile --ID PROFILE --flags '-p'   # build
ffpm run     --ID PROFILE                  # run
ffpm run     --ID PROFILE  --runner gprof  # profile

[LKedward]

Thanks @urbanjost; I've had a quick play around with your proposal demo — it's exciting to be able to use other compilers with fpm!

I've been using the --flag feature since it was introduced in bootstrap version and unfortunately I find it very clunky for a number of reasons. Your addition of compile/custom and ID is certainly an improvement on the compile+run workflow, but I still have to specify the flags correctly at the command line every time I run compile (which is often).
It's also worth noting that unlike the bootstrap method of hashing compiler flags for the output directory, this approach may end up with unintended effects if the compiler flags change but the ID does not during a partial rebuild (coming in #248).

Once flags can be specified in the manifest as different configs, I would be unlikely to use the compile/custom command or --flags flag ever again.
In essence, I don't think the command line is the right place for specifying compiler flags because it inevitably puts the burden on the user to manually 'load' and 'store' flags correctly at each invocation — I'd much rather store various flag combinations digitally in a toml file than in my brain.

@urbanjost, would you be happy opening separate PRs for the specification of compilers and specification of flags? The former seems ready to go from a specification standpoint, whereas the latter may need more discussion in it's own PR.

[urbanjost]

I will split them. My own comments in several other discussions is of the same sentiment. The prototype is just to aid the discussion. Everyone is different but I find it useful to actually try something if the discussion is stalled or has hit a roadblock.

I separated this out as a way to actually test using other compilers and a --flags switch like h-fpm because of some of those reasons. Since a TOML file is not the most intuitive feature I found a prototype useful for exploring other alternatives like a command similiar to "custom" that would write a TOML entry; using aliases and environment variables from the shell to create alternatives; whether a single name or ID should imply a compiler ... If the configuration parameters are in a file they tend to be so specific to a compiler that I want something like "fpm custom profile" and the name "profile" can be used with various compilers.

As-is I found
I was not using the --flags directly but was buiding up a list of aliases such as "fpm-profile" that had the flags set for the build; but once you combine that with multiple compilers then it needs to be a script instead of an alias, and so on.

So this let me actually test some of those combinations. It also matches the flag of the same name in h-fpm, and so on; but a lack of compiler and load options seemed to be the most commonly mentioned reason people could not use the earlier fpm versions. Thanks for actually trying it. There are pros and cons to any approach that involves supporting multiple compilers.

So my sentiment is similar but there is nothing like trying it. I have been finding that with three compilers now available on my machine that I usually use the environment variable FC and start three terminal sessions in screen(1) or tmux(1) with one set up for each compiler and toggle between them. That has made me wonder if the compiler switch should allow multiple names like
`fpm build -fc "nvfortran gfortran intel" . That might seem useful to me because I am testing. In normal use I would guess most people have a specific compiler they primarily work in and they would just set "FC". Since other products apparently use FC if that was implemented with a variable I think it would have to be one with a unique name like FPM_FC that would be looked for.

Still looking for other feedback on --flags including experiences with h-fpm or other source package managers.

[urbanjost]

On the contrarian side of this; because of the problems with remembering the compiler switches I was considering that the first build of a directory cache some data. A TOML file would see appropriate but at least for the first cut I was just going to use NAMELIST. The first time you built a (compiler-specific) directory it would create a file with the compiler and load flags from the custom command, so that it would actually be more like new in that it would initialize a directory $FC/ID.
Then build as well as run and test would have the ID switch; and build would not build unless the directory already existed and custom would always do a full build. That way the switches are remembered and are consistent and the previous versions of build, run, and test have nothing added by one switch something like release. Maybe even change the name from --ID to --custom, and the custom commands' first parameter would be the ID name. That would be
the next prototype if everyone agrees this one does not cut it.

[urbanjost]

fpm custom -fc ifort -fflags '-g -C -CB -CU -fpe8 -ftrapuv -fp-model=precise' -ID my_debug
fpm run -ID my_debug
fpm build -ID mydebug
fpm test -ID mydebug

or fpm build --custom mydebug ...
Assuming we allow a short name for --ID/--custom and you name with short names this could be
fpm build -i d

[urbanjost]

Currently build and custom do not use the "unnamed" fields so The --ID switch would not be required. TO allow the same for f-fpm there would have to be a change to do the same. Currently names following the subcommand are assumed to be targets (the name of the executables. To remove the --ID switch from all commands that would have to be changed so the same as h-fpm where you specify that with --target NAME. It depends on which would be used more commonly. Alternatively there could be runc and testc which would assume the first parameter is the custom build name instead of a target and test and run would be left as-is. With no pre-view or revision of these messages with an old version of firefox these are getting hard to compose. I will clean up the typos when I can access a more up-to-date machine.

It would be a non-compatible change to f-fpm to change to --target but would keep f-fpm and h-fpm more aligned; but this is still alpha. I think with these changes the changes would be useful even with the TOML interface for development. For producing a real package I think having the configuration in a shippable format in the TOML file will be the dominant format for build options that are required for a package to be functional (ie. coarray, openMP, external packages like -X11 and netCFD, .ncurses, MPI, ...).

[urbanjost]

So then the format could be

fpm custom myopts --fflags 'compiler args' -- load_args
fpm run myopts
fpm run myopts -target demo1 -- options_for_demo1

But then to run multiple programs instead of

fpm test test1 test2 test3

it would have to be

fpm run --target 'test1 test2 test3'

[urbanjost]

If anyone is interested I renamed the ffpm.f90 file to ffpm-custom.f90 and put ffpm.f90 there, which is a preliminary version that just specifies a compiler. So new, test, build have a --fc COMPILER_NAME switch. If not specified it looks for names specified with environment variable name FPM_FC, then FC.

• should --compiler be an alias for --fc?
• any suggestions for options for particular compilers for a default and release build?
• should an unknown compiler name be an error or should fpm(1) warn and try with no compiler options?
• allowing more than one compiler name on a single build would require some refactoring. Is that desirable?
• should the --fc switch name be optional on build? build ifort |build -fc ifort? If so should new and test do the same and have to have application names specified via --target NAME(s) instead of being able to be names specified after the run subcommand?
• other?

After some further testing I will start a new issue for just the compiler switch and probably close the one for the custom command unlesss there is interest. If there is interest I will start a new discussion for that. In either case I plan on closing this one. Thoughts and comments welcome. It will probably be a few days so in the meantime if anyone is interested in trying the single-file prototype please do.