Investigate/write alternative unit test framework

[sentientmonkey]

Right now, I've started with funit.

Pros:

• Fortran-like syntax
• From what I've seen, cleanest/easiest to write tests in
• codebase is ruby, easy to follow

Cons:

• Not actually Fortran
• Requires ruby and rubygems to be installed
• Weird bugs (can't use strings with commas in assertions?)
• Invalid Fortran test ends up with
• Has to run for a folder vs file

Given the list of other options, I'm not sure I'm seeing anything right now that meets what I would like:
http://fortranwiki.org/fortran/show/Unit+testing+frameworks

Ideally, I want something that allows you to write valid Fortran (so it can by syntax checked first), and requires as minimal setup for a user as possible (i.e. no new languages).

So I want to write this...

module hello_test
  use hello
  character(20) :: expected_greeting

  function setup
    expected_greeting = 'Hello, World!'
  end function setup
  
  function test_hello
    assert_equals( expected_greeting, greet() )
  end function test_hello

end module hello_test

and be able run it like...

  $ xfunit hello_test.f90
  .
  1 test passed, 1 assertion

This may require parsing the test Fortran module to scan for methods like setup/teardown & test_*.

[kytrinyx]

Those are some pretty significant cons! Thanks for investigating this.

[sentientmonkey]

Thanks - going to keep implementing exercises today in funit for now until a better alternative is found.

This seems to be a good overview of the problems and options in Fortran unit testing frameworks. Ultimately, because Fortran doesn't have reflection, another language is typically used to generate a Fortran test runner, which can be implemented in Fortran.

Given the goals of exercism and not wanting to install additional languages to work on the exercises (which is the right approach), the existing options don't seem great. I might prototype out something similar to the existing options, but implemented as a simple binary that can easily be packaged and distributed to users.

That leaves the following language options IMHO:

• Fortran
• C / C++
• mruby
• Go
• Rust

Given that I still don't know Fortran, implementing in the language itself might be rough. C/C++ is fine but not exactly something I'm excited about. Mruby could be cool, but last I tried the built toolchain is still kinda gnarly. I'm still a bit too table-flippy in Rust to be productive, so I think that leaves Go (which is a good fit for exercism anyway). The test driver will still be implemented Fortran, but the driver generator and test file parser will be in Go.

I'm open to any thoughts/feedback from anyone interested. 😄

[kytrinyx]

I think either Go or Rust would be the best bet. We have people who are actively involved in Exercism who know one or the other or both—both contributors and maintainers.

[zbeekman]

Hi @sentientmonkey and @kytrinyx!

I don't know Go or Rust, but I DO know Fortran, and this project has peaked my interest. I am one of the developers of OpenCoarrays which is the official distributed shared memory runtime for GFortran. I also started the informal, and poorly documented Fortran F/OSS Programmers group.

I used to use FUnit in the past, but now I typically use CMake/CTest for my testing needs, but this is far less than ideal in most cases. One of the biggest PITA with Fortran is the compilation order for source files due to the non-standard binary module file format. (.mod files are kinda/sorta like precompiled headers.... but unfortunately, you can't even perform a successful syntax check unless you process the files in the correct order.)

In my opinion, this means that a build system with autmomatic dependency resolution is a must. In my opinion, the most robust and cross platform solution is CMake. It has first class Fortran support, but it is a complicated beast. You could potentially use the test harness/automation capabilities for CMake if you use ISO_C_BINDING and/or write wrappers in Fortran that maps arguments to/from ISO_C_BINDING types. I have done this in the past, but it's been a while, so I'll need to refresh my memory.

Another potential is FoBiS.py. Stefano, is a very nice and responsive guy. (Although I think he needs to rephrase/fix some of his talk about "Fortran poor people"... but anyway, I digress) FoBiS.py has automatic dependency resolution, and has added the ability to write "doc-tests" in the most recent release.

As far as languages go, I think the largest overlap with Fortran programmers and some other language is likely to be one of C,C++, or Python.

Anyway, I'm going to take a look at what you have now, and think about what needs to happen to get this up and running. No promises, but, would love to see more mainstream Fortran resources. We do have a burgeoning community of open source tools, libraries and utilities on GH, and modern Fortran is 1) A parallel, PGAS language thanks to Coarrays and 2) fully object oriented if you choose to use the new language features.

[kytrinyx]

@zbeekman Thank you so much for jumping in here, your experience and background are much needed and very welcome!

I think the largest overlap with Fortran programmers and some other language is likely to be one of C,C++, or Python.

This is the best argument, as far as I can tell, for using C, C++, or Python. If Fortran programmers are interested in helping out with the track, I'd love to have tooling in a language that they're familiar with so that the barrier to jumping in and fixing it is low.

No promises, but, would love to see more mainstream Fortran resources.

Likewise!

modern Fortran is 1) A parallel, PGAS language thanks to Coarrays and 2) fully object oriented if you choose to use the new language features.

I'd love to have this emphasized in the "about" documentation for the track.

Did you check the weather today before you got dressed? If so then you've already used Fortran, and it's had a very real, immediate impact on your day to day life.

You said this in another issue, and it's such a great statement. This might be the best opening (ever) for the about documentation.

[zbeekman]

For anyone who wants to have a play with modern Fortran & Coarrays, I have a Jupyter kernel, that you can use in the cloud with Binder: https://bit.ly/CAF-Binder. Anyway I know you've got a lot on your plate, @kytrinyx, as do I so I'll probably refrain from posting again today, but I hope to find some time to help you guys get this off the ground.

FYI, I found exercism thanks to the GH post about Open Source Friday

[kytrinyx]

I have a Jupyter kernel, that you can use in the cloud with Binder

Very cool! I wonder if that would make it easier for folks to get started—not sure how easy it is to get the exercises running in the cloud, though, so we'd have to figure that out.

[pclausen]

Hi @kytrinyx , Hi @sentientmonkey

Sorry, for reviving an old thread....

I have been working with Fortran for 13 years and I would like to help out. I cloned the repo and Funit was a bit of a stumble block for me. I know C++, Python, Perl, ... but Ruby is not one of my go-to tools so it took some time figuring out how to even install funit on Windows. For anyone interested I used chocolatey and then ran choco install ruby and then gem install funit which seems to have done the job. I do have IntelFortran, but I thought I would use GFortran for this as it is accessible to all.

Setting FC, CXX and CC to the mingw compiler paths and then running funit I get this error:

:\> c:\tools\ruby25\bin\funit.bat hamming
Traceback (most recent call last):
        8: from c:/tools/ruby25/bin/funit:23:in `<main>'
        7: from c:/tools/ruby25/bin/funit:23:in `load'
        6: from c:/tools/ruby25/lib/ruby/gems/2.5.0/gems/funit-0.12.4/bin/funit:61:in `<top (required)>'
        5: from c:/tools/ruby25/lib/ruby/gems/2.5.0/gems/funit-0.12.4/lib/funit.rb:26:in `run_tests'
        4: from c:/tools/ruby25/lib/ruby/gems/2.5.0/gems/funit-0.12.4/lib/funit.rb:26:in `each'
        3: from c:/tools/ruby25/lib/ruby/gems/2.5.0/gems/funit-0.12.4/lib/funit.rb:28:in `block in run_tests'
        2: from c:/tools/ruby25/lib/ruby/gems/2.5.0/gems/funit-0.12.4/lib/funit.rb:28:in `each'
        1: from c:/tools/ruby25/lib/ruby/gems/2.5.0/gems/funit-0.12.4/lib/funit.rb:33:in `block (2 levels) in run_tests'
c:/tools/ruby25/lib/ruby/gems/2.5.0/gems/funit-0.12.4/lib/funit.rb:33:in `read': no implicit conversion of nil into String (TypeError)

Maybe Windows is not supported(?)

I would second @zbeekman that CMake and CTest seem to be the main place where most Fortran projects go, because Fortran is mostly only used for some inner numerical operations wrapped within C, C++, Python, R, ... or whatever. I did use CMake/CTest some years ago and in a mixed environment (Fortran, C++ on Windows and Linux) CMake/CTest was a very stable and usable tool. The exercism/cpp-guys also use CMake so we would be in good company.

If you are interested I would try to convert some tests into CTests in and see how it goes. It might take some months because I don't have too much time, but i would like to contribute.

It would be good if you could give some feedback if you think CMake/CTest makes sense or if you want to continue with Funit(?)

[kytrinyx]

I have been working with Fortran for 13 years and I would like to help out.

That is wonderful!

I thought I would use GFortran for this as it is accessible to all

That's definitely the direction we're hoping to take things in.

Ruby is not one of my go-to tools so it took some time figuring out how to even install funit on Windows.

Yeah, in my opinion Ruby should only be used for the Ruby track. It's a pain to get installed (particularly on Windows), so it raises the bar for contributions in a way that is entirely unhelpful.

I would second @zbeekman that CMake and CTest seem to be the main place where most Fortran projects go

That makes two knowledgable votes. I'd say go for it. I've no experience with Fortran, so should not really be allowed to have an opinion, much less a vote :-) My general guideline for all the tracks is to use the tooling that is common among the people who use the language.

From @sentientmonkey's original post:

Ideally, I want something that allows you to write valid Fortran (so it can by syntax checked first), and requires as minimal setup for a user as possible (i.e. no new languages).

So it sounds like Funit was just one of several possible choices (and with plenty of cons). I'd say let's ditch Funit and go with CMake and CTest.

[sentientmonkey]

I am all for something other than funit! Mostly I went with it first because I'm pretty new to Fortran and not really familiar with what's best to use to unit test it. If CMake/CTest is what's used commonly, I'm down for that route. Do you have any good examples of Fortran unit testing in a project?

Sorry for falling behind on this one - I haven't been doing Fortran at work recently, so this fell a bit on the wayside.

[zbeekman]

Sorry I haven't been able to (and probably won't be able to) help out with this. Between having joined on as a Homebrew maintainer and all of my other commitments, this is unlikely to float to the top of my priority list any time soon. However, feel free to ping me with questions or if you want me to help out with a piece of the CMake puzzle.

Your comment about 13 years caused me to do some math, and, good lord, I've been writing Fortran since sophmore year of undergrad, since 2004, so for 😬 ~14 years. Where does time go?!

[kytrinyx]

having joined on as a Homebrew maintainer

Congrats! I work with a couple of Homebrew maintainers and the pace of that project is staggering! Good luck with your work there :)

[zbeekman]

Ha thanks. It's like drinking from a fire hose.

…

On Tue, Aug 21, 2018 at 3:43 PM Katrina Owen ***@***.***> wrote: having joined on as a Homebrew maintainer Congrats! I work with a couple of Homebrew maintainers and the pace of that project is staggering! Good luck with your work there :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAREPPXx-5ObzOIYsgmnQu5C0x_ysEZxks5uTGLDgaJpZM4NtKyL> .

[pclausen]

Change to use CMake and CTest has been done so I will close this issue.