I/O issues

[jabl]

Hello,

I recently became aware of this project and took a look, and I have to say it's a really nice example of well written modern Fortran. One thing that raised my eyebrow was the handling of I/O, which seems a bit convoluted. Part of this might be due to the need to interface with matlab (a topic on which I'm not familiar with myself), but some issues are, I think, fixable.

1. The use of a hard-coded unit number (OUTUNIT = 42). This can lead to issues if an application that uses PRIMA accidentally uses the same unit number for it's own use. Luckily modern Fortran has a solution for this, namely the newunit= specifier for the open statement. By using newunit= the Fortran runtime library assigns a unit number which is guaranteed to not be in use. Similar to fopen in MATLAB and C, or open in POSIX.
2. Reopening and closing the file for every message. It would be more efficient to open the file once when writing the first message, and then only close it at the end of the optimization.
3. For integrating into other applications or higher level environments like scipy, I wonder if it would be more flexible if the optimization routines would take an additional optional callback procedure argument for handling messages. Then the user could decide what to do with them, and e.g. printing them to the screen would be integrated with the I/O buffering of the calling application, or it could process the messages somehow and log them somewhere etc.

[zaikunzhang]

Hi @jabl ,

Thank you for your encouraging and worm words, and for raising the point about the IO operations.

You are right. The interfacing with MATLAB (and maybe with other languages in the future) complicates the IO. For example, the MEX of MATLAB hijacks the standard output, so that anything printed to the standard output using write will not be shown at all.

I will improve the code according to your suggestions, especially the one concerning newunit. I will let you know when the revisions are made.

Many thanks again!

Best regards,
Zaikun

[zaikunzhang]

Hi @jabl ,

1. The use of a hard-coded unit number (OUTUNIT = 42). This can lead to issues if an application that uses PRIMA accidentally uses the same unit number for it's own use. Luckily modern Fortran has a solution for this, namely the newunit= specifier for the open statement. By using newunit= the Fortran runtime library assigns a unit number which is guaranteed to not be in use. Similar to fopen in MATLAB and C, or open in POSIX.

This is absolutely right. I have made revisions accordingly. See

https://github.com/libprima/prima/blob/f6f2aef8b973393e16a1ad5f5eaaeb7144c9b9a1/fortran/common/fprint.f90

Thank you for telling me if there are still problems with my implementation.

2. Reopening and closing the file for every message. It would be more efficient to open the file once when writing the first message, and then only close it at the end of the optimization.

I agree. However, I refrain from implementing file opening/closing in this way for the moment. The reason is as follows.

1. The solvers in PRIMA are designed for derivative-free / black-box / simulation-based optimization problems. For such problems, the major cost is the function evaluation, which is supposed to be provided by the user via a subroutine/function. Very often, each function evaluation takes minutes or hours (months in some extreme cases). Therefore, the cost of the solver itself is negligible, even if heavy I/O is involved.

2. Since I am developing a reference implementation, I hope to hide language-specific operations as much as possible. Thus I chose to concentrate all I/O operations at a single place, namely fprint.f90. Consequently, the file has to be opened and closed for each message.

3. For integrating into other applications or higher level environments like scipy, I wonder if it would be more flexible if the optimization routines would take an additional optional callback procedure argument for handling messages. Then the user could decide what to do with them, and e.g. printing them to the screen would be integrated with the I/O buffering of the calling application, or it could process the messages somehow and log them somewhere etc.

A callback procedure is a good idea. I will discuss it with the SciPy community.

Many thanks again!

Best regards,
Zaikun