Add support for Windows via Callback Filesystem?

[ljani]

Callback File System claims to support easy porting of FUSE modules to Windows with MinGW.

An evaluation version of Callback File System is available for download on their home page. The package contains the required libraries as well as samples. To actually run the binary, you'll need a free evaluation key, available on their site as well.

What do you think? Too much work?

[hasse69]

I can not even compile the simplest sample that is provided by this package :(

$ make
mingw32-g++ -g -c -I../../../../CPP -I../../../CPP/MinGW -I../../../CPP/FUSE -I.
./../../../FUSE_CPP -I.. -D_REENTRANT -D_THREAD_SAFE -D_UNICODE -DUNICODE -D_WIN
32_IE=0x0300 -D_FILE_OFFSET_BITS=64 -Wno-write-strings -Wno-conversion-null -fpermissive ../fusexmp.c
In file included from c:\mingw\include\unistd.h:95:0,
                 from ../fusexmp.c:28:
c:\mingw\include\parts\time.h:65:8: error: redefinition of 'struct timespec' struct timespec
        ^
In file included from ../../../CPP/FUSE/fuse.h:34:0,
                 from ../fusexmp.c:22:
../../../CPP/FUSE/fusecbfs.h:68:8: error: previous definition of 'struct timespec'
 struct timespec {
        ^
make: *** [../fusexmp.o] Error 1

I would not hold my breath on this one! The CBFS is a strict commercial product and, from what I can tell, has no intentions what so ever of supporting the Open Source community. Just the fact that the company behind this refuses me to even use a gmail account to request for the evaluation license makes me slightly uncomfortable to say the least.
For that reason alone I feel it is simply a waste of time and energy trying to get this to work. Sorry, but feel free to experiment with it yourself. If you manage to get past the simple step of compiling their so called "sample" program using Mingw let me know. I might reconsider, even if I doubt it.

[ljani]

EDIT: anyone reading this, please use msys2 instead of msys!

Thanks for taking interest! I kinda expected this, but there's no harm in asking.

I got the driver to compile by:

• Passing -D__MSVCRT_VERSION__=0x0601 to g++ (patch the makefile)
• Patching fusecbfs.h to use #define stat __stat64 instead of _stat64
  Google told me some discussion related to these changes in MinGW in general..

And the sample:

• Pass -D__MSVCRT_VERSION__=0x0601 -D_TIMESPEC_DEFINED to g++ (patch the makefile)
• Add #undef stat at the end of fusecbfs.h (before #endif //_CBFS_FUSE_USER_LIB_)

I tried to compile rar2fs under MinGW but didn't get very far.. Here's my progress so far:

• Compile the driver in CPP\FUSE (see notes above)
• Add #ifndef uid_t and #ifndef gid_t around the typedefs in fusecbfs.h, otherwise you'll get duplicate definitions
• configure.ac should read something like this:

case "$host" in
     *-*-mingw*)
            FUSE_LIBS="-lfusecbfs"
        ;;
     *)
AC_CHECK_LIB(fuse4x,fuse_new, [FUSE_LIBS="-lfuse4x"],
    [ AC_CHECK_LIB(fuse_ino64,fuse_new, [FUSE_LIBS="-lfuse_ino64"],
        [ AC_CHECK_LIB(fuse,fuse_new, [FUSE_LIBS="-lfuse"],
            [ AC_CHECK_LIB(osxfuse,fuse_new, [FUSE_LIBS="-losxfuse"],
                [AC_MSG_ERROR([
                    Can't find fuse library - add alternative library search path using
                    --with-fuse-lib and rerun configure. ])
            ],)
        ],)
    ],)
],)
        ;;
esac

• Then run autoreconf -f -i && ./configure --with-fuse=/home/Jani/CPP/FUSE/ CPPFLAGS="-x c++ -D_TIMESPEC_DEFINED -I/home/Jani/CPP/MinGW" --with-fuse-lib=/home/Jani/CPP/FUSE/ LDFLAGS="-L/home/Jani/CPP/MinGW/ -lcbfs"
• This fails to check the fuse version because fuse_version can't be linked(?):

configure:8443: error: 
    rar2fs requires FUSE 2.6 or newer. Please check config.log for errors.
    If you cannot determine the problem, mail the author of the package
    and include the config.log file

EDIT: yeah, I agree, this is kinda hacky.. Their MinGW support isn't the best :/

[hasse69]

Just remove the check for FUSE_XXX_VERSION in configure.ac or patch the CBFS/FUSE header files. It is not a problem with fuse_version(), it seems to be that this package does not define FUSE_MAJOR_VERSION and FUSE_MINOR_VERSION in fuse.h.

You seem to have got a lot of things working though. Feel free to continue your efforts :) I will assist in any way I can but I do not know how much time I will be able to spend on building this myself. Maybe if you succeed I can follow your "step by step". What I need to focus on is to make sure that any patches required to rar2fs to make this work is suitable for upstream. Also remeber you need the UnRAR source library.

The lack of a POSIX compliant thread library is going to be an issue here :(

[hasse69]

I managed to build the UnRAR source library after some struggle.
One of the most anoying facts seems to be that the swprintf() family of functions on Windows takes less arguments than the Unix variant!? Specifically it is the size argument that is not present on Windows! If simply removing the use of that argument is going to work will remain a mystery for a while.

The gcc in MinGW seems to enable a few more warnings by default than what seems to be standard. For that reason remove the Makefile needs to be slightly modified:

• Remove use of -fPIC, -DRAR_SMP and -lpthread
• Add creation of a static library using ar rcs libunrar.a *.o (this can be done manually after build)

The source code (.cpp files) need several changes, preferably using __MINGW32__ conditional:

• all swprintf() functions must remove the size argument.
• the strnlen() function does not seem to exist and need to be replaced by standard strlen() followed by a check for size > n.
• In win32lnk.cpp remove the typedef of struct _REPARSE_DATA_BUFFER and pointer declaration of PREPARSE_DATA_BUFFER since it is already defined by MinGW.

It does not build completely clean, but almost.
Now add --with-unrar and --with-unrar-lib when running ./configure.

Having problems though with RARGetDllVersion(), it seems it cannot resolve it.
But the function is there, but looks scrambled when it should not be.

$ nm -a unrar/libunrar.a | grep RARGetDll
000000c0 T _RARGetDllVersion@0

I might have missed something, but this is it for today!

[hasse69]

It is the use of the PASCAL data type that confuses autoconf. AC_CHECK_LIB is not made for finding C++ functions like this on Windows since PASCAL mangles the function prototype.
I will need to use AC_TRY_LINK instead, but until then simply bypass the check to get further.

[vadmium]

Hello, this could be a waste of time, but have you looked at using Dokan at all? https://web.archive.org/web/20150511010630/http://dokan-dev.net/en/, https://github.com/Maxhy/dokany? It is an open source project and apparently has a FUSE API, but I haven’t investigated it closely.

[ljani]

@hasse69 thanks for the pointers. Yeah, I had to run after reaching the version check. I'll try to catch up with your work now :)

You can get swprintf to work by #define swprintf _snwprintf in os.hpp. It's a little less labour that way.

Windows uses __stdcall instead of __cdecl indeed.. I too solved that issue with the libfusecbfs by bypassing the check altogether.

[ljani]

@vadmium Do you have any links to the FUSE API?

This seems to do the swprintf problem better (os.hpp, right after WIN32_LEAN_AND_MEAN):

#define swprintf _snwprintf
#define vswprintf _vsnwprintf
#define __STRICT_ANSI__

EDIT: be sure to issue make lib to get the exports. I messed up with that.

[ljani]

Patching away those rar checks, I get this:

checking for source revision in SVN... no
checking for commit hash in git repo... e08a828
checking for sched_setaffinity... no
checking scandir ARG3 => const struct dirent * ... not ok
checking scandir ARG3 => struct dirent * ... not ok
configure: error: scandir unable to find a working combination

The log says:

conftest.c:81:34: error: 'alphasort' was not declared in this scope
   scandir(dir, namelist, select, alphasort);
                                  ^
conftest.c:81:43: error: 'scandir' was not declared in this scope
   scandir(dir, namelist, select, alphasort);

I wonder where to find those on MinGW..

[hasse69]

From the scandir(3) man page:

alphasort() and scandir() are specified in POSIX.1-2008, and are widely available. versionsort() is a GNU extension.

So either MinGW does not provide it, or it is called something else.

[ljani]

Hmm, I just moved to msys2/mingw-w64-i686 and libfusecbfs.a seems to compile without modifications. Also libunrar can be installed via pacman.

Here's my progress with msys2:

• Install msys2 from here. I used msys2-base-x86_64-20150512.tar.xz
• Run mingw32_shell.bat and restart it after it has done the initial setup
• If your msys2 is old do this:

pacman -Sy
pacman --needed -S bash pacman pacman-mirrors msys2-runtime
RESTART THE SHELL HERE
pacman -Su

• Unzip the CBFS driver
• Continue here:

pacman -S base-devel git mingw-w64-i686-gcc
cd CPP/FUSE
make
cd
git clone https://github.com/hasse69/rar2fs.git
export LANG=en_US.UTF-8
cd rar2fs
autoreconf -f -i

EDIT: mingw-w64-i686 means 64-bit toolchain cross compiling to 32-bit. The provided driver is 32-bit.
EDIT: Also msys2 has a little different approach. Afaik they provide a posix compatibility layer via Cygwin whereas msys specifically avoids that.
EDIT: uid_t and gid_t still require patching. My current configure command reads:
$ ./configure --with-fuse=/home/Jani/CPP/FUSE/ --with-fuse-lib=/home/Jani/CPP/FUSE/ CPPFLAGS="-x c++ -I/home/Jani/CPP/MinGW"

[hasse69]

You still need the complete source of UnRAR to build rar2fs. If you install libunrar through pacman make sure you download a matching version of the source.

[ljani]

Yeah, well, to think about it, it won't work.. libunrar provided by pacman is 64-bit and we need 32-bit..

unrar also compiles fine without modifications, but there's lots of warnings: warning: -fPIC ignored for target (all code is position independent). I guess I should remove that flag again. I kinda skipped that part and just copypasted your results :)

[hasse69]

I still get issues with _stat64 when compiling the driver :(

[ljani]

Are you using the already patched files? Make sure you revert any changes done to stat. Ie the line must read #define stat _stat64

[ljani]

And here we are..

checking scandir ARG3 => const struct dirent * ... not ok
checking scandir ARG3 => struct dirent * ... not ok
configure: error: scandir unable to find a working combination

conftest.c:82:34: error: 'alphasort' was not declared in this scope
   scandir(dir, namelist, select, alphasort);
                                  ^
conftest.c:82:43: error: 'scandir' was not declared in this scope
   scandir(dir, namelist, select, alphasort);
                                           ^

[hasse69]

I use the 32-bit toolchain, that is probably why. I am on a 32-bit Windows 8 platform.
My build require the same changes as before.

[ljani]

Oh, I'm on 64-bit Windows 8.1. That probably explains it.

Any idea how to use a third party scandir implementation with the configure? Here's one example.

EDIT: Cygwin should contain some scandir implementation, I'm not sure why it's not available in msys2

[ljani]

Oh, it's in msys2-runtime-devel. But if I add -I/usr/include/sys/ to CPPFLAGS, it fails with

checking whether we are cross compiling... configure: error: in `/home/Jani/rar2      fs':
configure: error: cannot run C compiled programs.

I guess that's because msys2-runtime-devel is 64-bit so it gets confused about something.

[hasse69]

You should add /usr/include to the include path, sys is implicit. But that should be part of the default system search path already!? Header files in /usr/include/sys are included using a <sys/whatever.h> directive.

Btw, I was using some old gcc version from previous msys installation! I removed the old MinGW install and installed gcc using pacman. Now I get support for a lot more functions.

[hasse69]

I am sort of giving up on on this one! Now I cannot even compile the driver anymore. It complains about tchar.h not being found! I know way to little about msys2 to make any progress here :(

[ljani]

I guess the include path has something to do with the cross compiling and thus we can't include /usr/include. Btw, with -I/usr/include I get lots of warnings:

...
checking sys/statvfs.h presence... yes
configure: WARNING: sys/statvfs.h: present but cannot be compiled
configure: WARNING: sys/statvfs.h:     check for missing prerequisite headers?
configure: WARNING: sys/statvfs.h: see the Autoconf documentation
configure: WARNING: sys/statvfs.h:     section "Present But Cannot Be Compiled"
configure: WARNING: sys/statvfs.h: proceeding with the compiler's result
configure: WARNING:     ## ---------------------------------------------- ##
configure: WARNING:     ## Report this to http://code.google.com/p/rar2fs ##
configure: WARNING:     ## ---------------------------------------------- ##
checking for sys/statvfs.h... no
...
checking ucontext.h presence... yes
configure: WARNING: ucontext.h: present but cannot be compiled
configure: WARNING: ucontext.h:     check for missing prerequisite headers?
configure: WARNING: ucontext.h: see the Autoconf documentation
configure: WARNING: ucontext.h:     section "Present But Cannot Be Compiled"
configure: WARNING: ucontext.h: proceeding with the compiler's result
configure: WARNING:     ## ---------------------------------------------- ##
configure: WARNING:     ## Report this to http://code.google.com/p/rar2fs ##
configure: WARNING:     ## ---------------------------------------------- ##
checking for ucontext.h... no
...
checking fuse.h presence... yes
configure: WARNING: fuse.h: present but cannot be compiled
configure: WARNING: fuse.h:     check for missing prerequisite headers?
configure: WARNING: fuse.h: see the Autoconf documentation
configure: WARNING: fuse.h:     section "Present But Cannot Be Compiled"
configure: WARNING: fuse.h: proceeding with the compiler's result
configure: WARNING:     ## ---------------------------------------------- ##
configure: WARNING:     ## Report this to http://code.google.com/p/rar2fs ##
configure: WARNING:     ## ---------------------------------------------- ##
checking for fuse.h... no
configure: error:
        Can't find fuse.h - add alternative include search path using
        --with-fuse and rerun configure.

So I'm quite sure that's not the correct way to continue. I should understand how scandir affects configure and try to add a 3rd party implementation.

Heh, have you messed up your include path?

Yeah, I'm kinda in the same boat, plus I don't know autotools/autoconf. One thing I know is the silver searcher can be compiled on Windows if it helps anyone.

[hasse69]

There seems to be some inconsitency here.
The i686-w64 compiler has all the needed header files and libraries for building the driver, but it does not have support for eg. scandir() and select(). Both which are needed to compile rar2fs.
The native msys2 compiler supports both, but then it is not a cross compiler and is missing support for eg. the Windows specific tchar.h header file. So the driver cannot be compiled using it.
A little bit of a chicken and egg problem :(

[ljani]

Yeah, I agree. I kinda gave up on this and decided to experiment with the driver and C#. Got it working pretty nice, but now I need to find a C# library that supports seeking non-compressed RARs.. Without seeking it's working already :)

EDIT: I guess I should abandon C# and become almost a direct competitor with C++ ;)

[hasse69]

There is nothing wrong with a bit of competition :)
I am closing this issue now because even if it is possible (everything is) I think it will take a bit too much time to make it work using msys2 or similar. Maybe Dokan is another path to try as stated before? Feel free to contact me if you need any further assistance in your attempts of getting something to work on Windows.

[hasse69]

Actually I will leave it open. Maybe someone else is more successful in picking up where we left of?

[ljani]

Open issues do invite more discussion indeed.

I wonder how portable the current rar2fs code is, apart from autotools.. If I skipped the whole FUSE API and used matching CBFS/Dokan APIs directly from a different toolchain (VS).

[hasse69]

I would say rar2fs is very portable, assuming a POSIX compliant platform with FUSE support.
Windows was never really up on the table. But it is simply callbacks. They can be replaced. For Windows there is already a pre-compiled UnRAR.dll that can be downloaded from rarlab.com.
The select() system call can easily be replaced by poll() or epoll(). POSIX thread support is also needed unless you rewrite the thread creation etc. I do not know what Windows can offer here.

[jyhpsycho]

WIth msys2, it's possible to use many POSIX-compliant functionalities like pthread, not all though (at least FUSE will not work because it uses system call provided by Linux kernel, and Windows kernel does not provide functions like that).

It seems that "Callback File System" is one of VFS-like driver for Windows, and provides "Compatibility layer" for existing FUSE modules. I have very interesting of that and I'll try it if there's some free time to me... :-)

[hasse69]

There is an alternative method to do this which will be followed up in issue #75 .
Will close this one for now.