Update README.md to explain cross-compiling #922

clanmills · 2019-06-22T17:25:29Z

When I wrote README.md for v0.27, I had never succeeded in cross-compiling for MinGW from Linux. I believe this is now working and Dan has a CI job.

I hope to refresh my brain about this and get it documented for v0.27.2. For sure for v0.27.3

clanmills · 2019-06-23T20:08:53Z

@piponazo I'll deal with README.md and README-CONAN.md. I believe most of the new documentation will go into README-CONAN.md and will be referenced in README.md

I have not been able to get CMake to find and use the expat and zlib which were cross-compiled by conan, so I'm going to defer this for v0.27.3. Here's what I've done on Ubuntu 18.04

Install pip3 and conan:

Install pip3 $ sudo apt install python3-pip
Install conan $ sudo pip3 install conan

Configure for cross-platform development

Install the cross-platform tools $ sudo apt-get install g++-mingw-w64 gcc-mingw-w64
https://docs.conan.io/en/latest/systems_cross_building/cross_building.html
Add this file to <exiv2dir>/cmake/mingw-w64-x86_64.cmake

# Sample toolchain file for building for Windows from an Ubuntu Linux system.
#
# Typical usage:
#    *) install cross compiler: `sudo apt-get install mingw-w64`
#    *) cd build
#    *) cmake -DCMAKE_TOOLCHAIN_FILE=~/mingw-w64-x86_64.cmake ..

set(CMAKE_SYSTEM_NAME Windows)
set(TOOLCHAIN_PREFIX x86_64-w64-mingw32)

# cross compilers to use for C, C++ and Fortran
set(CMAKE_C_COMPILER ${TOOLCHAIN_PREFIX}-gcc)
set(CMAKE_CXX_COMPILER ${TOOLCHAIN_PREFIX}-g++)
set(CMAKE_Fortran_COMPILER ${TOOLCHAIN_PREFIX}-gfortran)
set(CMAKE_RC_COMPILER ${TOOLCHAIN_PREFIX}-windres)

# target environment on the build host system
set(CMAKE_FIND_ROOT_PATH /usr/${TOOLCHAIN_PREFIX})

# modify default behavior of FIND_XXX() commands
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

Add a new profile to ~/.conan/profiles/linux-to-win64

toolchain=/usr/x86_64-w64-mingw32 # Adjust this path
target_host=x86_64-w64-mingw32
cc_compiler=gcc
cxx_compiler=g++

[env]
CONAN_CMAKE_FIND_ROOT_PATH=$toolchain
CHOST=$target_host
AR=$target_host-ar
AS=$target_host-as
RANLIB=$target_host-ranlib
CC=$target_host-$cc_compiler
CXX=$target_host-$cxx_compiler
STRIP=$target_host-strip
RC=$target_host-windres

[settings]
# We are building in Ubuntu Linux
os_build=Linux
arch_build=x86_64

# We are cross-building to Windows
os=Windows
arch=x86_64
compiler=gcc

# Adjust to the gcc version of your MinGW package
compiler.version=7.3
compiler.libcxx=libstdc++11
build_type=Release

Build dependencies, run cmake, and build:

$ cd <exiv2dir> ; mkdir MinGW_build ; cd MinGW_build
$ conan install .. --profile linux-to-win64 --build missing
$ cmake .. -DCMAKE_TOOLCHAIN_FILE=../cmake/mingw-w64-x86_64.cmake

rmills@rmillsmm-ubuntu:~/gnu/github/exiv2/0.27-maintenance/MinGW-build$ cmake .. -DCMAKE_TOOLCHAIN_FILE=../cmake/mingw-w64-x86_64.cmake 
-- Conan: Using cmake targets configuration
-- Library zlib found /home/rmills/.conan/data/zlib/1.2.11/conan/stable/package/040db29ed48cb6d2bb9f4c6d13e74d5e319c7ad8/lib/libzlib.a
-- Library gmock_main found /home/rmills/.conan/data/gtest/1.8.1/bincrafters/stable/package/dc6599c77833e79bb142050659ef8269e4e1d8c5/lib/libgmock_main.dll.a
-- Library gmock found /home/rmills/.conan/data/gtest/1.8.1/bincrafters/stable/package/dc6599c77833e79bb142050659ef8269e4e1d8c5/lib/libgmock.dll.a
-- Library gtest found /home/rmills/.conan/data/gtest/1.8.1/bincrafters/stable/package/dc6599c77833e79bb142050659ef8269e4e1d8c5/lib/libgtest.dll.a
-- Library expat found /home/rmills/.conan/data/Expat/2.2.6/pix4d/stable/package/776dc1175d2af830f77a93cb1550e2a530879c82/lib/libexpat.a
-- Conan: Adjusting language standard
-- Current conanbuildinfo.cmake directory: /home/rmills/gnu/github/exiv2/0.27-maintenance/MinGW-build
-- Conan: Compiler GCC>=5, checking major version 7.3
-- Conan: Checking correct version: 7.3
CMake Error at /usr/share/cmake-3.10/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find EXPAT (missing: EXPAT_LIBRARY EXPAT_INCLUDE_DIR)
Call Stack (most recent call first):
  /usr/share/cmake-3.10/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.10/Modules/FindEXPAT.cmake:61 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  cmake/findDependencies.cmake:31 (find_package)
  CMakeLists.txt:61 (include)


-- Configuring incomplete, errors occurred!
See also "/home/rmills/gnu/github/exiv2/0.27-maintenance/MinGW-build/CMakeFiles/CMakeOutput.log".
See also "/home/rmills/gnu/github/exiv2/0.27-maintenance/MinGW-build/CMakeFiles/CMakeError.log".
rmills@rmillsmm-ubuntu:~/gnu/github/exiv2/0.27-maintenance/MinGW-build$

3a) $ cmake .. -DCMAKE_TOOLCHAIN_FILE=../cmake/mingw-w64-x86_64.cmake -DEXIV2_ENABLE_XMP=Off -DEXIV2_ENABLE_PNG=Off
4) $ make
It's built:

rmills@rmillsmm-ubuntu:~/gnu/github/exiv2/0.27-maintenance/MinGW-build$ ls -l bin/exiv2.exe
-rwxrwxr-x 1 rmills rmills 1185810 Jun 23 20:54 bin/exiv2.exe
rmills@rmillsmm-ubuntu:~/gnu/github/exiv2/0.27-maintenance/MinGW-build$

Testing the build

Don't know yet. To be investigated. It can be tested in MinGW in much the same way as I test Visual Studio builds.

piponazo · 2019-06-30T19:45:35Z

Sorry, I did not see this until now. I remember that I had also some issues with some of the dependencies when I tried to cross-compile from Ubuntu 18.04 to Windows with MinGW, but I think at the end I could solve them. I'll take a look to it at some point.

D4N · 2019-07-04T20:02:47Z

Imho the simplest way how to cross compile from Linux with MinGW is to use Fedora (either grab the vagrant box or a container via podman pull docker.io/library/fedora) as we do on GitLab:

$ sudo dnf install mingw64-gcc-c++ mingw64-filesystem mingw64-expat mingw64-zlib cmake make
$ mkdir build && pushd build
$ mingw64-cmake -DEXIV2_TEAM_EXTRA_WARNINGS=ON -DEXIV2_ENABLE_VIDEO=ON -DEXIV2_ENABLE_WEBREADY=ON -DEXIV2_ENABLE_WIN_UNICODE=ON -DBUILD_SHARED_LIBS=OFF ..
$ make -j $(nproc)
$ popd

That's literally it. The only caveat is that shared libraries don't work due to the issues with the Error type, which I haven't solved yet in #685 and #779.

clanmills · 2019-07-19T14:48:16Z

I can't get this to work on Ubuntu for the following reasons:

Conan doesn't install the expat dependency.
The file cmake/mingw-w64-x86_64.cmake isn't in the repos.
Dan's recipe requires mingw-cmake which does not exist in apt.
Building "static only" is of limited usefulness.
I'd like to document how to test this on Windows or using wine on Linux.

I'm going to defer this for v0.27.3.

D4N · 2019-07-22T20:57:37Z

Robin Mills <notifications@github.com> writes:

I can't get this to work on Ubuntu for the following reasons: 1) Conan doesn't install the expat dependency. 2) The file cmake/mingw-w64-x86_64.cmake isn't in the repos. 3) Dan's recipe requires `mingw-cmake` which does not exist in `apt`.

Because that is only available on Fedora. If you want a painless solution: use Fedora, grab the same packages as on GitLab and it should just work (no need to mess with Ubuntu, conan and cmake toolchain files).

4) Building "static only" is of limited usefulness.

This should be fixed once #685 makes it into master, no chance for 0.27 though.

…

5) I'd like to document how to test this on Windows or using wine on Linux. I'm going to defer this for v0.27.3. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #922 (comment)

clanmills · 2019-07-22T21:10:49Z

Thanks, Dan. That's a bummer.

I would like to get cross-compiling documented for "the dots". I had it working in January with a little magic in contrib and some DLLs which I shouldn't have put into the repos.

I'll revisit this and look at Fedora for 0.27.3. Or maybe I'll focus on the book. We'll see.

clanmills · 2020-04-06T17:56:19Z

I've had another visit to this topic and installed Fedora 31.1.9 on a fresh VM.

Dan's instructions here are correct:#922 (comment)

I had some work to perform to install zlib1-devel and expat-devel on x86_64 to build native, however it built smoothly and passed the test framework. Then I used Dan's instructions to cross-compile, and indeed bin/exiv2.exe was generated.

I haven't figured out how to configure the Fedora Network to "see" rmillsmm-w10 (a Windows VM), so I zipped the 0.27-maintenance directory, and copied it to c:\msys64\home\rmills\temp\0.27-maintence. When I attempted to run the test suite ($ make tests), CMake started regenerating the Makefiles to compile with the GCC/msys2 compiler

So, I tried to run the test manually (from information in README.md)

$ cd ~/temp/0.277-maintenance/tests
$ export EXIV2_BINDIR=$PWD/../build_mingw/bin
$ export EXIV2_EXT=.exe
$ make newtests
514 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/test $ make newtests
( cd ../tests ; python3 runner.py --verbose )
test_run (bugfixes.github.test_CVE_2017_1000126.TestCvePoC) ... ERROR
....... everything fails .......

Amazing. It almost worked. So, I tried execute exiv2.exe from the command-prompt:

518 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/test $ ls -l ../build_mingw/bin/exiv2.exe
-rwxr-xr-x 1 rmills rmills 6748907 Apr  6 17:26 ../build_mingw/bin/exiv2.exe
519 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/test $ ../build_mingw/bin/exiv2.exe
C:/msys64/home/rmills/temp/0.27-maintenance/build_mingw/bin/exiv2.exe: error while loading shared libraries: zlib1.dll: cannot open shared object file: No such file or directory
520 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/test $

It's cross compiled to use dynamic dependencies which are not on PATH or LD_LIBRARY_PATH on rmillsmm-w10.

Good Progress. Tomorrow is another day.

clanmills · 2020-04-07T09:18:52Z

You need to copy 5 DLLs from Fedora as follows:

$ scp /usr/x86_64-w64-mingw32/sys-root/mingw/bin/zlib1.dll rmillsmm-w10:/c:\msys64\home\rmills\temp\0.27-maintenance\build_mingw\bin
$ scp /usr/x86_64-w64-mingw32/sys-root/mingw/bin/libexpat-1.dll  rmillsmm-w10:/c:\msys64\home\rmills\temp\0.27-maintenance\build_mingw\bin
$ scp /usr/x86_64-w64-mingw32/sys-root/mingw/bin/libstdc++-6.dll   rmillsmm-w10:/c:\msys64\home\rmills\temp\0.27-maintenance\build_mingw\bin
$ scp /usr/x86_64-w64-mingw32/sys-root/mingw/bin/libwinpthread-1.dll    rmillsmm-w10:/c:\msys64\home\rmills\temp\0.27-maintenance\build_mingw\bin
$ scp /usr/x86_64-w64-mingw32/sys-root/mingw/bin/libgcc_s_seh-1.dll  rmillsmm-w10:/c:\msys64\home\rmills\temp\0.27-maintenance\build_mingw\bin

The bash tests execute correctly, with the exception of a time issue in conversions.sh which I will not investigate.

532 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/build_mingw/bin $ export EXIV2_BINDIR=$PWD
533 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/build_mingw/bin $ export EXIV2_EXT=.exe
534 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/build_mingw/bin $ cd ../../test
535 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/test $ $EXIV2_BINDIR/exiv2 -vVg exiv2 -g compiler -g time
exiv2 0.27.3.9
exiv2=0.27.3
compiler=G++
time=17:26:17
package_name=exiv2
executable=C:\msys64\home\rmills\temp\0.27-maintenance\build_mingw\bin\exiv2.exe
have_gmtime_r=0
config_path=C:\Users\rmills\exiv2.ini
xmlns=mediapro:http://ns.iview-multimedia.com/mediapro/1.0/
538 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/test $ make test
mkdir -p tmp
rm -rf tmp/test-failed
Running addmoddel.sh ...
all testcases passed.
Running conversions.sh ...
Files /home/rmills/temp/0.27-maintenance/test/tmp/conversions.out and /home/rmills/temp/0.27-maintenance/test/data/conversions.out differ
165c165
< Exif.Image.DateTime                          Ascii      20  2015:04:17 19:10:22
---
> Exif.Image.DateTime                          Ascii      20  2015:04:18 02:10:22
***
*** conversions.sh result = 3
***
Running exifdata-test.sh ...
all testcases passed.
...

The new tests $ make newtests almost all fail due to different line-endings in the Windows. The reference output in the test suite has LF endings, and the output on Windows has CR-LF.

I've tried to change all the python files without success:

551 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/test $ find ../tests -name "*.py" | xargs unix2dos 
nix2dos: converting file ../tests/bugfixes/redmine/test_issue_922.py to DOS format...
unix2dos: converting file ../tests/bugfixes/redmine/test_issue_935.py to DOS format...
unix2dos: converting file ../tests/bugfixes/redmine/test_issue_937.py to DOS format...
...

That didn't fix it. I'll open a new issue about this and ask @D4N if it's possible to modify the test suite to forgive line-ending mismatches. #1146

D4N · 2020-04-07T09:50:27Z

Robin Mills <notifications@github.com> writes:

Running addmoddel.sh ... all testcases passed. Running conversions.sh ... Files /home/rmills/temp/0.27-maintenance/test/tmp/conversions.out and /home/rmills/temp/0.27-maintenance/test/data/conversions.out differ 165c165 < Exif.Image.DateTime Ascii 20 2015:04:17 19:10:22 --- > Exif.Image.DateTime Ascii 20 2015:04:18 02:10:22

Looks like a timezone issue to me.

clanmills · 2020-04-07T10:33:27Z

@D4N It is a time-zone issue and I've see it before. The test is set up for KL, Malaysia which is GMT+8 and I'm running it in BST (GMT+1). And the difference in output is 7 hours. I'm not going to bother investigating this further.

The task here is to document cross compiling and I have sufficient information to update README.md which will reference this discussion.

If you feel it's important to investigate this TZ issue, please open a new issue.

clanmills · 2020-04-07T10:48:10Z

The TZ issue has been investigated and resolved with a "tweak" in the test-harness. #485

The "tweak" can be extended to support mingw64:

504 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/build_mingw/bin $ ./exiv2 -vVg platform
exiv2 0.27.3.9
platform=mingw64
505 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/build_mingw/bin $

clanmills · 2020-04-07T14:35:05Z

I've investigated @D4N 's caveat about cross building shared libraries. I'm pleased to say that it does successfully build, with the same restriction as the static libraries, which are:

You have to copy the 5 "system" dlls in ming_build_shared/bin
The suite passes and exhibits the same behaviour concern TZ and the new python tests.

clanmills · 2020-04-07T15:35:35Z

I've investigated using wine to run exiv2 from Fedora and it works! Both build_mingw and build_mingw_shared are working. As before you have to copy the 5 "system" DLLs into the bin.

Running the test harness is challenging because we don't have make and bash in the wine environment. I can think of two ways to get the test suite to run:

Either:
Install MSYS2 into Wine

Or:
Mount the Fedora build_mingw or build_mingw_shared directories on a Windows machine on which you have installed MSYS2.

There are notes in README.md about setting up MSYS2.

Once you are running MSYS2/bash, you will be able to run the test suite as documented in README.md

The line-ending issue with the python test suite has been fixed. #1146 #1150

clanmills · 2020-04-10T08:05:41Z

EUREKA!

Mounting the Fedora Directory in Windows works! I achieved that by getting Fedora to build into a shared drive on the Mac //Mac/Home/gnu/github/exiv2/0.27-maintenance/build_mingw_fedora. Build in the cross platform way using mingw-cmake.

Copy the 5 DLLs on Fedora to the bin:

$ for i in libexpat-1.dll libgcc_s_seh-1.dll libstdc++-6.dll libwinpthread-1.dll zlib1.dll ; do cp -v /usr/x86_64-w64-mingw32/sys-root/mingw/bin/$i bin ; done

Run the test suite in MSYS on Windows on the build:

$ cd //Mac/Home/gnu/github/exiv2/0.27/maintenance/build_mingw_fedora
$ export EXIV2_BINDIR=$pwd/bin
$ export EXIV2_EXT=.exe
$ cd ../test
$ make test

There are two issues:

The TZ puzzle: test/conversions.sh is failing on TZ test #485 I have submitted a fix for this. PR fix_485_0.27_conversions.sh #1152
Three python tests fail for reasons that will not be investigated.

ERROR: test_run (bugfixes.redmine.test_issue_812.DoNotDestroyHardLinks)
FAIL: test_run (bugfixes.github.test_issue_1099.EmptyValueInCommandFile)
FAIL: test_run (bugfixes.redmine.test_issue_1054.Exiv2jsonRecursiveJsonTreeWithXMP)

I'm not going to work on the python test suite in this obscure environment. The JSON issue is another manifestation of the TZ issue. The other two are obscure.

The test suite works impressively well and I have 100% confidence that the build is good.

clanmills · 2020-04-10T10:33:47Z

#1153

clanmills added the documentation label Jun 22, 2019

clanmills added this to the v0.27.2 milestone Jun 22, 2019

clanmills assigned clanmills and piponazo Jun 22, 2019

clanmills modified the milestones: v0.27.2, v0.27.3 Jun 23, 2019

clanmills mentioned this issue Jul 11, 2019

v0.27.2 #940

Merged

clanmills modified the milestones: v0.27.3, v0.27.2 Jul 11, 2019

clanmills modified the milestones: v0.27.2, v0.27.3 Jul 19, 2019

clanmills added a commit that referenced this issue Jul 19, 2019

Defer #922 for v0.27.3

500b76a

clanmills removed their assignment Oct 10, 2019

clanmills assigned clanmills and unassigned piponazo Mar 27, 2020

clanmills mentioned this issue Apr 7, 2020

Forgive line-ending mismatches in the python test suite. #1146

Closed

clanmills mentioned this issue Apr 10, 2020

test/conversions.sh is failing on TZ test #485

Closed

This was referenced Apr 10, 2020

Python test suite has errors on MSYS2 when cross compiled from Fedora #1151

Closed

fix_922 Cross compiling documentation (0.27->master) #1153

Merged

clanmills closed this as completed Apr 10, 2020

clanmills mentioned this issue Apr 27, 2020

Exiv2 RoadMap #1018

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update README.md to explain cross-compiling #922

Update README.md to explain cross-compiling #922

clanmills commented Jun 22, 2019

clanmills commented Jun 23, 2019 •

edited

Loading

piponazo commented Jun 30, 2019

D4N commented Jul 4, 2019

clanmills commented Jul 19, 2019

D4N commented Jul 22, 2019 via email

clanmills commented Jul 22, 2019

clanmills commented Apr 6, 2020 •

edited

Loading

clanmills commented Apr 7, 2020 •

edited

Loading

D4N commented Apr 7, 2020 via email •

edited by clanmills

Loading

clanmills commented Apr 7, 2020 •

edited

Loading

clanmills commented Apr 7, 2020 •

edited

Loading

clanmills commented Apr 7, 2020 •

edited

Loading

clanmills commented Apr 7, 2020 •

edited

Loading

clanmills commented Apr 10, 2020 •

edited

Loading

clanmills commented Apr 10, 2020

Update README.md to explain cross-compiling #922

Update README.md to explain cross-compiling #922

Comments

clanmills commented Jun 22, 2019

clanmills commented Jun 23, 2019 • edited Loading

Install pip3 and conan:

Configure for cross-platform development

Build dependencies, run cmake, and build:

Testing the build

piponazo commented Jun 30, 2019

D4N commented Jul 4, 2019

clanmills commented Jul 19, 2019

D4N commented Jul 22, 2019 via email

clanmills commented Jul 22, 2019

clanmills commented Apr 6, 2020 • edited Loading

clanmills commented Apr 7, 2020 • edited Loading

D4N commented Apr 7, 2020 via email • edited by clanmills Loading

clanmills commented Apr 7, 2020 • edited Loading

clanmills commented Apr 7, 2020 • edited Loading

clanmills commented Apr 7, 2020 • edited Loading

clanmills commented Apr 7, 2020 • edited Loading

clanmills commented Apr 10, 2020 • edited Loading

clanmills commented Apr 10, 2020

clanmills commented Jun 23, 2019 •

edited

Loading

clanmills commented Apr 6, 2020 •

edited

Loading

clanmills commented Apr 7, 2020 •

edited

Loading

D4N commented Apr 7, 2020 via email •

edited by clanmills

Loading

clanmills commented Apr 7, 2020 •

edited

Loading

clanmills commented Apr 7, 2020 •

edited

Loading

clanmills commented Apr 7, 2020 •

edited

Loading

clanmills commented Apr 7, 2020 •

edited

Loading

clanmills commented Apr 10, 2020 •

edited

Loading