Can TurboJPEG be build for UWP?

[Jure-BB]

I'm trying to build TurboJPEG for use in Windows Store app. Unfortunately Windows App Certification Kit fails when using default windows turbojpeg.dll.

So I created Visual Studio solution with cmake command as it is described in building.md. Then I converted turbojpeg.vcxproj to UWP project and enabled /ZW compile switch like described here:
https://docs.microsoft.com/en-us/cpp/porting/how-to-use-existing-cpp-code-in-a-universal-windows-platform-app
I also changed CharSet to Unicode.

Unfortunately I'm getting the following error: "LINK : fatal error LNK1104: cannot open file 'LIBCMT.lib'"

Now I am wondering if it is even possible to build TurboJPEG for UWP with /ZW compile option?

[dcommander]

I don’t really know anything about UWP, but based on the nature of the error message, try passing -DWITH_CRT_DLL=1 to cmake when generating the project.

[Jure-BB]

Thanks! I added that. I also removed /ZW option for all c files inside turbojpeg project, because I was getting error messages that c files cannot be compiled with /ZW

I'm not sure if it's correct to remove /ZW option for c files. It says so here https://stackoverflow.com/questions/9304862/d8048-cannot-compile-c-file-openssl-applink-c-with-zw-option, but I don't have any experience with C & C++ so I'm kind of flying blind here.

Now I get the following error: "MSVCRTD.lib(utility_app.obj) : error LNK2019: unresolved external symbol __imp_RoInitialize referenced in function __scrt_initialize_winrt"

PS: I only need decoder functionality

[Jure-BB]

I added runtimeobject.lib to turbojpeg project and now it builds :)

https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/86bb1b44-aa7c-4873-8e08-74ecca47cd8a/roinitializewrapper-gives-me-errors?forum=vclanguage
I added it under: Properties configuration->Linker->Input->Additional Dependencies

Will try to test WACK certification again :)

[Jure-BB]

WACK outputs these two errors:

API _putenv in api-ms-win-crt-environment-l1-1-0.dll is not supported for this application type. turbojpeg.dll calls this API.
API getenv in api-ms-win-crt-environment-l1-1-0.dll is not supported for this application type. turbojpeg.dll calls this API.

[Jure-BB]

I have managed to pass WACK certification :)

I had to add -DNO_GETENV command line option to turbojpeg, jpeg & jpeg-static projects.

Also, I had to comment out the following code inside jsimd_x86_64.c

  env = getenv("JSIMD_FORCENONE");
  if ((env != NULL) && (strcmp(env, "1") == 0))
    simd_support = 0;
  env = getenv("JSIMD_NOHUFFENC");
  if ((env != NULL) && (strcmp(env, "1") == 0))
    simd_huffman = 0; 

and inside turbojpeg.c, comment out all instances of:

if(flags&TJFLAG_FORCEMMX) putenv("JSIMD_FORCEMMX=1");
else if(flags&TJFLAG_FORCESSE) putenv("JSIMD_FORCESSE=1");
else if(flags&TJFLAG_FORCESSE2) putenv("JSIMD_FORCESSE2=1");

And now turbojpeg.dll is Windows Store compatible :)

[dcommander]

Regarding /ZW:
Again, I'm no expert when it comes to Microsoft-specific development, but if that option doesn't work on C files, then it is probably useless with respect to libjpeg-turbo. Our project contains only C, assembler, and Java code, and only C code is compiled with the Visual Studio compiler.

Regarding getenv() and putenv():
This has already been brought up on the mailing list. I am not inclined to support Microsoft's "safe string" versions of getenv() and putenv(), but I would be willing to add a CMake option that disables any of those calls at compile time.

I am leaving this issue open until I have a chance to experiment with a Windows Store build and see if I can develop a set of easy instructions for making libjpeg-turbo pass WACK certification. Ideally, this should just involve setting a couple of CMake variables.

[Jure-BB]

I don't know if it's /ZW option or conversion of project to UWP project that is changing the compilation, but here is the WACK report for original turbojpeg.dll:

Binary analyzer

Error Found: The binary analyzer test detected the following errors:

File turbojpeg.dll has failed the AppContainerCheck check.
File turbojpeg.dll has failed the DBCheck check.

How to fix: Apply the required linker options - SAFESEH, DYNAMICBASE, NXCOMPAT, and APPCONTAINER - when you link the app.

Supported APIs

Error Found: The supported APIs test detected the following errors:

API RtlAddFunctionTable in kernel32.dll is not supported for this application type. turbojpeg.dll calls this API.
API RtlCaptureContext in kernel32.dll is not supported for this application type. turbojpeg.dll calls this API.
API RtlVirtualUnwind in kernel32.dll is not supported for this application type. turbojpeg.dll calls this API.
API UnhandledExceptionFilter in kernel32.dll is not supported for this application type. turbojpeg.dll calls this API.
API VirtualProtect in kernel32.dll is not supported for this application type. turbojpeg.dll calls this API.
API __dllonexit in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API __iob_func in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API _amsg_exit in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API _initterm in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API _lock in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API _onexit in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API _putenv in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API _putenv_s in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API _setjmp in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API _unlock in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API _vsnprintf in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API abort in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API calloc in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API exit in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API fprintf in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API free in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API fwrite in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API getenv in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API longjmp in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API malloc in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API memcpy in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API memset in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API signal in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API sprintf in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API sscanf in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API strlen in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API strncmp in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API toupper in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.
API vfprintf in msvcrt.dll is not supported for this application type. turbojpeg.dll calls this API.

How to fix: Review the error messages to identify the API that is not part of the Windows SDK for Windows Store apps. Please note, apps that are built in a debug configuration or without .NET Native enabled (where applicable) can fail this test as these environments may pull in unsupported APIs. Retest your app in a release configuration, and with .NET Native enabled if applicable.

[dcommander]

Some of those C library functions are critical to the operation of libjpeg-turbo, so I’m not sure how you were able to make it work at all if functions like setjmp() and malloc() aren’t allowed. It seems as if Microsoft has created their own version of C for the Windows Store that doesn’t support some of the C standard functions. If that is the case, then there is no way for us to support the Windows Store. If I have misunderstood your comment, then please clarify how you worked around those errors.

[Jure-BB]

I posted that for reference as it seems there are more changes under the hood than just removal of getenv() and putenv().

turbojpeg.dll works. I've tested it on Windows and on Xbox One. Most of those errors went away when I enabled /ZW switch and converted project to UWP project (I haven't tested which one of these two is the key one that got rid of the errors). I guess Microsoft has created their own version that supports only subset of features. When compiling with /ZW and/or when project is set to UWP, it seems like it substitutes it with their own version that contains supported setjmp() and malloc() functions. I don't really know, but it works.

To convert turbojpeg.vcxproj project file to UWP project file, I made the following changes inside the project file:

<!--<WindowsTargetPlatformVersion>10.0.16299.0</WindowsTargetPlatformVersion>-->
<AppContainerApplication>true</AppContainerApplication>  
<ApplicationType>Windows Store</ApplicationType>  
<WindowsTargetPlatformVersion>10.0.16299.0</WindowsTargetPlatformVersion>  
<WindowsTargetPlatformMinVersion>10.0.16299.0</WindowsTargetPlatformMinVersion>  
<ApplicationTypeRevision>10.0</ApplicationTypeRevision>  

Let me know, if I you need any more details.

[dcommander]

I will test this in the coming weeks and make sure that turbojpeg.dll can be straightforwardly built for the Windows Store without modifying the source. I'll check jpeg62.dll as well, but it may not be possible with that library, since it accesses the CRT across the DLL boundary.

[Tryum]

@Jure-BB To target UWP from you can call cmake with "-DCMAKE_SYSTEM_NAME=WindowsStore -DCMAKE_SYSTEM_VERSION=10.0". It'll generate apropriate visual studio solution.

[Jure-BB]

@Tryum Thanks! Does it pass Windows App Certification Kit? I cannot test this on my project at the moment, because of some other UWP issues.

[Tryum]

I'm also working on passing WACK... but have several other libraries to fix before trying again ;)

[dcommander]

@Tryum your suggestion above doesn't work at all. The libjpeg-turbo build uses CMAKE_SYSTEM_NAME to determine which operating system it is running on, which helps it choose the appropriate SIMD extensions. Passing -DCMAKE_SYSTEM_NAME=WindowsStore to the build causes it to break.

[Tryum]

I'm building libjpeg-turbo with the help of https://github.com/Microsoft/vcpkg I checked the build options :
-DCMAKE_SYSTEM_NAME=WindowsStore "-DCMAKE_SYSTEM_VERSION=10.0" -DWITH_SIMD=OFF - DENABLE_STATIC=OFF -DENABLE_SHARED=OFF -DWITH_CRT_DLL=ON
These options builds my system with a fresh checkout from this repository.

The problem is that cmake is unable to handle two platforms at the same time (host and xcompile target)... even if we are building UWP targetting x86/x64 generated binaries won't be runnable directly.

[dcommander]

If you're turning off SIMD acceleration, then you're missing the entire point. This is libjpeg-turbo. Doesn't make sense to build it without the code that makes it turbo.

[dcommander]

Also, if you are building with neither static nor shared libraries, then what's left?!

[Tryum]

My bad, I built with -DENABLE_SHARED=ON.
I'm not using libjeg-turbo directly, it's far down the dependency tree. For now I'm just focusing on getting a UWP-compatible library.
To enable SIMD for UWP, I guess I could use the generated files for win32-desktop.

[Jure-BB]

@Tryum Did you try to build it the way I described? It passed WACK and I didn't had to disable SIMD.

[Tryum]

Unfortunatly, I need libjpeg-turbo to be built in an automated way, so disabling SIMD as provided in the port file of vcpkg is fine for me. I was just troubled by the getenv() call.

[dcommander]

I'm about to push a fix for that. Stand by. This is all rather esoteric to sort out.

[dcommander]

Here's what I did:

• Set the proper PATH, LIB, and INCLUDE variables by running

  C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" x64 store
  

  It turns out that there is a special version of msvcrt.lib that you need to link against for the Windows Store, and on my machine, that library is under C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\lib\store rather than C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\lib. This is one of the "hidden" things the IDE did to enable Windows Store compatibility.

• Configured the build using

  set LDFLAGS=-nxcompat -appcontainer -dynamicbase runtimeobject.lib -manifest:no
  cmake -G"NMake Makefiles" -DCMAKE_BUILD_TYPE=Release -DWITH_CRT_DLL=1 -DCMAKE_C_FLAGS="-DNO_GETENV -DNO_PUTENV" {source_directory}
  

  The LDFLAGS environment variable populates the initial values of CMAKE_EXE_LINKER_FLAGS and CMAKE_SHARED_LINKER_FLAGS. I tried setting those variables directly, but the CMake compiler detection failed (apparently compiler detection doesn't honor the existing values of CMAKE_*_LINKER_FLAGS, but it does honor the LDFLAGS environment. CMake bug.)

• Built libjpeg-turbo using

  nmake install DESTDIR=.\
  

  This installs it to {build_directory}\libjpeg-turbo64.

• Copied the libjpeg-turbo logo from here to {build_directory}\libjpeg-turbo64\logo.png

• Created {build_directory}\libjpeg-turbo64\AppxManifest.xml with the following contents:

  <?xml version="1.0" encoding="utf-8"?>
  <Package xmlns="http://schemas.microsoft.com/appx/2010/manifest">
    <Identity Name="org.libjpeg-turbo.libjpeg-turbo"
              Version="1.5.80.0"
              Publisher="CN=The libjpeg-turbo Project, O=The libjpeg-turbo Project, L=Austin, S=Texas, C=US" />
    <Properties>
      <DisplayName>libjpeg-turbo</DisplayName>
      <PublisherDisplayName>The libjpeg-turbo Project</PublisherDisplayName>
      <Logo>logo.png</Logo>
    </Properties>
    <Prerequisites>
      <OSMinVersion>10.0.0</OSMinVersion>
      <OSMaxVersionTested>10.0.0</OSMaxVersionTested>
    </Prerequisites>
    <Dependencies>
      <PackageDependency Name="Microsoft.VCLibs.140.00.UWPDesktop"
                         MinVersion="12.0.40652.5"
                         Publisher="CN=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=Washington, C=US" />
    </Dependencies>
    <Resources>
      <Resource Language="en-us" />
    </Resources>
  </Package>
  

• Created an appx package using

  "c:\Program Files (x86)\Windows Kits\10\App Certification Kit\makeappx.exe" pack /d libjpeg-turbo64 /o /p libjpeg-turbo.appx
  

• Validated {build_directory}\libjpeg-turbo.appx using WACK. It still complains about the O/S versions listed in the manifest, and there's no clear reason why it's complaining (I tried various values but wasn't able to make that error go away.) However, if using the latest libjpeg-turbo Git code, which includes the appropriate NO_GETENV/NO_PUTENV fixes, all of the API usage errors are now gone.

NOTE: Not only isn't /ZW necessary (because all of libjpeg-turbo is written in C), but when I tried to add it to the C flags, the compiler rejected it and the build failed. It seems that the aforementioned linker options and the special Windows Store msvcrt.lib are the main things that make the build Windows Store-compliant. You may also need to add -safeseh to the linker flags for a 32-bit build.

It would be nice to automate this at some point and officially adopt a procedure for it, but at the moment, it all seems too fragile and undocumented for me to trust. I can at least create a wiki page for it on libjpeg-turbo.org (although it would be nice to figure out the OSMinVersion error first.)

[Tryum]

What about a two step process ?
Producing generated files in a first pass with a win32 target, and consuming those files with a UWP cmake target.
Should produce UWP ready libraries, with SIMD activated.

[dcommander]

The process I describe above has full SIMD acceleration. The only reason you disabled it was to get rid of the getenv() calls, but I just pushed a commit that does that the proper way.

[ras0219-msft]

Fixed in vcpkg master :) Thanks @dcommander!

[danilogr]

I am new to building things with vcpkg.
When I try .\vcpkg.exe install libjpeg-turbo:x86-uwp I get a build error (see
config-x86-uwp-out.log).

PS C:\vcpkg> .\vcpkg.exe install libjpeg-turbo:x86-uwp
Computing installation plan...
The following packages will be built and installed:
    libjpeg-turbo[core]:x86-uwp
Starting package 1/1: libjpeg-turbo:x86-uwp
Building package libjpeg-turbo[core]:x86-uwp...
-- Using community triplet x86-uwp. This triplet configuration is not guaranteed to succeed.
-- [COMMUNITY] Loading triplet configuration from: C:\vcpkg\triplets\community\x86-uwp.cmake
-- Using cached C:/vcpkg/downloads/libjpeg-turbo-libjpeg-turbo-166e34213e4f4e2363ce058a7bcc69fd03e38b76.tar.gz
-- Using source at C:/vcpkg/buildtrees/libjpeg-turbo/src/fd03e38b76-a5b0f5fafe
-- Configuring x86-uwp
CMake Error at scripts/cmake/vcpkg_execute_required_process.cmake:72 (message):
    Command failed: ninja -v
    Working Directory: C:/vcpkg/buildtrees/libjpeg-turbo/x86-uwp-rel/vcpkg-parallel-configure
    Error code: 1
    See logs for more information:
      C:\vcpkg\buildtrees\libjpeg-turbo\config-x86-uwp-out.log

Call Stack (most recent call first):
  scripts/cmake/vcpkg_configure_cmake.cmake:295 (vcpkg_execute_required_process)
  ports/libjpeg-turbo/portfile.cmake:33 (vcpkg_configure_cmake)
  scripts/ports.cmake:90 (include)


Error: Building package libjpeg-turbo:x86-uwp failed with: BUILD_FAILED
Please ensure you're using the latest portfiles with `.\vcpkg update`, then
submit an issue at https://github.com/Microsoft/vcpkg/issues including:
  Package: libjpeg-turbo:x86-uwp
  Vcpkg version: 2020.02.04-nohash

Additionally, attach any relevant sections from the log files above.

The bottom line is the following:

-- Selecting Windows SDK version 10.0.18362.0 to target Windows 10.0.
CMake Error at CMakeLists.txt:7 (project):
  Failed to run MSBuild command:

    C:/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/MSBuild/Current/Bin/MSBuild.exe

  to get the value of VCTargetsPath:

    Microsoft (R) Build Engine version 16.4.0+e901037fe for .NET Framework
    Copyright (C) Microsoft Corporation. All rights reserved.

    Build started 2/24/2020 9:38:33 AM.
    Project "C:\vcpkg\buildtrees\libjpeg-turbo\x86-uwp-rel\CMakeFiles\3.14.2\VCTargetsPath.vcxproj" on node 1 (default targets).
    C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(775,5): error : The OutputPath property is not set for project 'VCTargetsPath.vcxproj'.  Please check to make sure that you have specified a valid combination of Configuration and Platform for this project.  Configuration='Debug'  Platform='Win32'.  You may be seeing this message because you are trying to build a project without a solution file, and have specified a non-default Configuration or Platform that doesn't exist for this project. [C:\vcpkg\buildtrees\libjpeg-turbo\x86-uwp-rel\CMakeFiles\3.14.2\VCTargetsPath.vcxproj]
    Done Building Project "C:\vcpkg\buildtrees\libjpeg-turbo\x86-uwp-rel\CMakeFiles\3.14.2\VCTargetsPath.vcxproj" (default targets) -- FAILED.

    Build FAILED.

What am I doing wrong?

[dcommander]

Please contact the vcpkg developers for support when building libjpeg-turbo using that system. I know nothing about vcpkg, and it is completely external to the libjpeg-turbo code base.

[frozenmistadventure]

May I know if libjpeg-turbo supports arm64-uwp for Hololens 2?
What I tried:

1. compiling via vcpkg successfully, but it seems SIMD was disabled by default. As expected, no impact in performance.
2. modified the portfile and force it -DWITH_SIMD=ON, but the performance doesn't improve.

Thus, before digging further on compiler, hope you could give me some advice and let me know if this is supported on arm64-uwp. thanks.

[dcommander]

I repeat:

Please contact the vcpkg developers for support when building libjpeg-turbo using that system. I know nothing about vcpkg, and it is completely external to the libjpeg-turbo code base. I have no idea, specifically, why SIMD was disabled by default or why enabling it didn't improve performance. That is certainly not the case with the official libjpeg-turbo code base, in which NEON SIMD does produce a very significant speedup.

[frozenmistadventure]

sorry about this, and I am actually referring to this old post.
#69

I noticed that Windows/ARM is labeled as "won't integrate" in 2017. Thus, is official libjpeg-turbo technically possible with Windows/ARM?
I understood that's completely external from official code base, which I will do more research.
But would you please tell me if I am looking for something impossible with current version of libjpeg-turbo?
Thank you so much and I am very appreciated your fast respond.

[dcommander]

At the moment, the NEON SIMD code is specific to the GNU Assembler (GAS) and likely doesn't support the correct calling conventions for Windows/ARM. So it should be possible to build libjpeg-turbo for arm64-uwp, but it will not be fully accelerated. Sorry for my confusion on that. libjpeg-turbo 2.1 will contain a rewrite of the NEON SIMD extensions using intrinsics, and it may be possible to support Windows/ARM with SIMD acceleration after that feature is integrated. However, the problem with Windows/ARM is and always has been lack of funding to support that platform. I don't personally need that platform, so I'm not going to spend any of my time supporting it unless someone is paying for that time.

[dcommander]

libjpeg-turbo 2.1 now fully supports Windows/Arm builds with Visual C++, Visual Studio + Clang, or MinGW.

[jonywalker1998]

I just checked out the main branch about 2.1.3. The mentioned way to build uwp arm64 didn't work.
Build environment was setup using:
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Auxiliary\Build\vcvarsall.bat" x64_arm64 uwp

build error sounds like the asm objs generated were not right to the target platform:
simd\CMakeFiles\simd.dir\x86_64\jsimdcpu.asm.obj : fatal error LNK1112: module machine type 'x64' conflicts with target machine type 'ARM64' NMAKE : fatal error U1077: 'C:\PROGRA~2\MICROS~3\2019\PROFES~1\VC\Tools\MSVC\1429~1.301\bin\Hostx64\arm64\lib.exe' : return code '0x458' Stop. NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\bin\HostX64\ARM64\nmake.exe"' : return code '0x2' Stop. NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\bin\HostX64\ARM64\nmake.exe"' : return code '0x2' Stop.