Rawtherapee accepts only regular files but not named pipes

[SarenT]

Hi,

This is related to my previous issue on rawtherapee not being able to read from standard input.
As a workaround I have tried to use a fifo named pipe. The problem seems to be that this code in rtgui/main.cc prevents pipes being used:
if (Glib::file_test (argument, Glib::FILE_TEST_IS_REGULAR)) { inputFiles.emplace_back (argument); continue; }
Is Glib::file_test (argument, Glib::FILE_TEST_IS_REGULAR) really necessary? Why does rawtherapee care about the file being special, if raw data could also come from pipes, devices etc?

Version: 5.0-r1-gtk3
Branch: releases
Commit: 50114c1
Commit date: 2017-02-01
Compiler: x86_64-linux-gnu-gcc 5.4.0
Processor: x86_64
System: Linux
Bit depth: 64 bits
Gtkmm: V3.18.0
Build type: Release
Build flags: -O3 -std=gnu++11 -Werror=unused-label -fopenmp -Werror=unknown-pragmas
Link flags: -Wl,-Bsymbolic-functions -Wl,-z,relro
OpenMP support: ON
MMAP support: ON

[Floessie]

There are fewer options for Glib::file_test() than I expected. Could it be, your named pipe is executable? Maybe we could change it to

if (Glib::file_test(argument, Glib::FILE_EXISTS) && !Glib::file_test(argument, Glib::FILE_TEST_IS_DIR))

[SarenT]

Not really, but I can make it executable.
prw-rw-r-- 1 user group 0 Mar 9 23:07 raw.nef

I think this (your suggestion) is a better alternative. Some people may even concider device files (/dev/... stuff) such as drivers etc. to provide raw input.
I have now cloned the project. I was actually hoping to add standard input option to get raw input, which would simplify my code a lot. But that might require more expertise and my C/C++ knowledge is very limited.

[SarenT]

Update here... I have tried to replace it but obviously it didn't work. It seems to be that image IO relies on fseek to move to the end of the file and getting the current position to find the size of the file. All of which cannot be done with named pipes. But it seems to be that fopen call from gfopen in myfile.cc seems to fail. I am still investigating it. Simply allowing pipes is therefore not a good idea yet. One needs a dedicated handling of pipes, where all data can be loaded into memory before being used. But then implementing stdio seems to be easier or more straight forward than that. I will try to implement that...

PS: As for my purpose of not writing bunch of data onto my SSD, I have found ramfs as a workaround. ramfs in Linux basically mounts some space from memory, which is very fast and bypasses any disk IO. Rest is usual regular file IO, which rawtherapee can handle. But this requires root access, which is not very suitable for end users.

[heckflosse]

@SarenT Did you try building RT without MMAP support?

[Floessie]

@heckflosse The problem is a bit different: Pipes like stdin or FIFOs can (by their streaming nature) only be read once. RT on the other hand opens a file multiple times during startup. One example is to obtain the EXIF. This will hang. So it's not a problem of MMAP or not.

I've investigated a bit (and even made rtexif use myfile.h) and it looks as if a solution is possible. It will change myfile.h to return std::shared_ptr<IMFILE> (also for future caching, maybe) and is thus a little invasive. I think I'll come up with a proof of concept within a few days.

Best,
Flössie