Unable to open RAW images #1304

rps01 · 2019-05-02T23:59:24Z

Unable to open ARW asset for generating a JPG thumbnail using libvips

Libvips version : vips-8.7.4

Command: vipsthumbnail example1.arw -o tn_%s.jpg --size 1280x1280

(vipsthumbnail:51223): VIPS-WARNING **: 16:45:56.718: Unknown field with tag 50341 (0xc4a5) encountered

(vipsthumbnail:51223): VIPS-WARNING **: 16:45:56.719: Unknown field with tag 50341 (0xc4a5) encountered
vipsthumbnail: unable to thumbnail example1.arw
MissingRequired: TIFF directory is missing required "ImageLength" field
tiff: unable to open "example1.arw" for input

The text was updated successfully, but these errors were encountered:

jcupitt · 2019-05-03T11:33:36Z

Hello @rps01,

libvips does not support camera RAW files directly. You'll need to convert to TIFF using something like dcraw first.

przemyslawpluta · 2019-06-06T21:21:15Z

@jcupitt libvips seems to work fine in terms of providing metadata for Nikon and Canon RAW files without the need of converting them to TIFF but fails with Sony RAW files?

jcupitt · 2019-06-07T04:09:38Z

libvips will attempt to load unknown files via libMagick, so if you can configure your imagemagick to load Sony RAW files, it should work.

przemyslawpluta · 2019-06-07T10:13:35Z

Running $ convert -list format lists ARW file as configured:

   Format  Module    Mode  Description
-------------------------------------------------------------------------------
      3FR  DNG       r--   Hasselblad CFV/H3D39II
      3G2  MPEG      r--   Media Container
      3GP  MPEG      r--   Media Container
      AAI* AAI       rw+   AAI Dune image
       AI  PDF       rw-   Adobe Illustrator CS2
      ART* ART       rw-   PFS: 1st Publisher Clip Art
      ARW  DNG       r--   Sony Alpha Raw Image Format
      ...

With identify -verbose A7rII-Various-15-08-15-3.ARW we get:

  Base filename: A7rII-Various-15-08-15-3.ARW
  Format: ARW (Sony Alpha Raw Image Format)
  Class: DirectClass
  Geometry: 8000x5320+0+0
  Units: Undefined
  Colorspace: sRGB
  Type: TrueColor
  Base type: Undefined
  Endianess: Undefined
  Depth: 16-bit
  Channel depth:
    Red: 16-bit
    Green: 16-bit
    Blue: 16-bit
  Channel statistics:
    Pixels: 42560000
    Red:
      min: 0  (0)
      max: 63796 (0.973465)
      mean: 21089.1 (0.321799)
      standard deviation: 10425.6 (0.159084)
      kurtosis: -0.776482
      skewness: -0.587452
      entropy: 0.947934
    Green:
      min: 0  (0)
      max: 61350 (0.936141)
      mean: 24252.9 (0.370076)
      standard deviation: 11027.7 (0.168272)
      kurtosis: -0.482587
      skewness: -0.920075
      entropy: 0.942753
    Blue:
      min: 0  (0)
      max: 53579 (0.817563)
      mean: 25478 (0.388769)
      standard deviation: 15691.2 (0.239432)
      kurtosis: -1.5413
      skewness: -0.308184
      entropy: 0.933881
  Image statistics:
    Overall:
      min: 0  (0)
      max: 63796 (0.973465)
      mean: 23606.7 (0.360215)
      standard deviation: 12381.5 (0.18893)
      kurtosis: -1.02245
      skewness: -0.371304
      entropy: 0.941523
  Rendering intent: Perceptual
  Gamma: 0.454545
  Chromaticity:
    red primary: (0.64,0.33)
    green primary: (0.3,0.6)
    blue primary: (0.15,0.06)
    white point: (0.3127,0.329)
  Matte color: grey74
  Background color: white
  Border color: srgb(223,223,223)
  Transparent color: none
  Interlace: None
  Intensity: Undefined
  Compose: Over
  Page geometry: 8000x5320+0+0
  Dispose: Undefined
  Iterations: 0
  Compression: Zip
  Orientation: Undefined
  Properties:
    date:create: 2019-06-07T10:10:00+00:00
    date:modify: 2019-06-07T10:10:00+00:00
    dng:Aperture: F/5.6
    dng:AspectRatio: 1.503759
    dng:ChannelMultipliers: 2.394531 1.000000 1.558594 1.000000
    dng:CreateID: 1
    dng:Crop: 0 0 8000 5320
    dng:EXIFSource: DCRaw
    dng:FocalLength: 55.0 mm
    dng:Green: 1.027
    dng:ISOSpeed: 100
    dng:Make: Sony
    dng:MatrixInputProfile: Color matrix
    dng:Model: ILCE-7RM2
    dng:Orientation: 0
    dng:Rotation: 0.000000
    dng:Shutter: 1/800.0 s
    dng:sRGBOutputProfile: sRGB
    dng:Temperature: 4780
    dng:Timestamp: Sat Aug 15 16:22:38 2015
    dng:WB: Camera WB
    dng:WBFineTuning: 0
    png:IHDR.bit-depth-orig: 16
    png:IHDR.bit_depth: 16
    png:IHDR.color-type-orig: 2
    png:IHDR.color_type: 2 (Truecolor)
    png:IHDR.interlace_method: 0 (Not interlaced)
    png:IHDR.width,height: 8000, 5320
    png:sRGB: intent=0 (Perceptual Intent)
    png:text: 2 tEXt/zTXt/iTXt chunks were found
    signature: 0161efe1bcfda28849039417aaaba574d96bab65427ecbed0f73b4585dbabcc6
    Software: UFRaw
    Source: SonyILCE-7RM2
  Artifacts:
    verbose: true
  Tainted: False
  Filesize: 197.932MiB
  Number pixels: 42560000
  Pixels per second: 15.0814MP
  User time: 2.820u
  Elapsed time: 0:03.822
  Version: ImageMagick 7.0.8-44 Q16 x86_64 2019-05-11 https://imagemagick.org

While attempt to get metadata with sharp via Libvips renders:

UnhandledPromiseRejectionWarning: Error: Input file has corrupt header: MissingRequired: TIFF directory is missing required "ImageLength" field tiff: unable to open "A7rII-Various-15-08-15-3.ARW" for input

jcupitt · 2019-06-07T10:17:10Z

It looks like Sony RAW files are TIFFs with extra fields, so libvips is trying to load them itself with libtiff and not falling back to imagemagick.

Will libvips convert to a regular TIFF successfully?

vips magickload A7rII-Various-15-08-15-3.ARW x.tif

przemyslawpluta · 2019-06-07T10:35:55Z

Yes ... running below produces expected TIFF file successfully:

vips magickload A7rII-Various-15-08-15-3.ARW A7rII-Various-15-08-15-3.TIFF

jcupitt · 2019-06-07T10:54:07Z

That's good.

I don't think the command-line vipsthumbnail will work for this, there's no way to force it to use magickload. You'd need to use some kind of scripting language.

For example, in Python you could write:

import os
import pyvips

if os.path.splitext(filename)[1] == 'ARW':
    img = pyvips.Image.magickload(filename, access='sequential')
    thumb = img.thumbnail_image(128)
else:
    img = pyvips.Image.thumbnail(filename)

img.write_to_file('thumb.jpg')

It'd be something very similar for PHP / C# / Ruby / Go / Rust / JavaScript / etc.

przemyslawpluta · 2019-06-07T11:10:59Z

I need to extract JPG from RAW images and currently I use exiftool to extract them but libvips via sharp to get metadata in the first instance to confirm if file is valid and does not pretend to be something else. As exiftool is only used for RAW files extractions most of the times libvips is in action for metadata as majority of files are JPG, PNG at this stage.

While this works fine with NEF and CR2 it fails with ARW where I'd have to put the workaround in. I assume it will not be possible with libvips regarding ARW and metadata retrieval?

przemyslawpluta · 2019-06-07T12:34:20Z

vipsheader complains as well I assume for the same reason ...

vipsheader A7rII-Various-15-08-15-3.ARW

(vipsheader:9898): VIPS-WARNING **: 12:11:03.911: Unknown field with tag 50341 (0xc4a5) encountered

(vipsheader:9898): VIPS-WARNING **: 12:11:03.912: Unknown field with tag 50341 (0xc4a5) encountered
vipsheader: MissingRequired: TIFF directory is missing required "ImageLength" field
tiff: unable to open "A7rII-Various-15-08-15-3.ARW" for input

jcupitt · 2019-06-07T13:29:59Z

Yes, that's right, it's trying to use the TIFF loader.

At the moment the is_tiff recogniser in libvips just checks the first four bytes for the TIFF magic number. I suppose we could change it to do a basic sanity check on the header as well. Then if that failed, libvips would automatically fall back to using magickload.

Would that fix your problem?

przemyslawpluta · 2019-06-07T13:46:25Z

Sounds like a solution that could work.

We were testing for TIFF by checking the magic number at the head of the file. However, formats like ARW are TIFF-like without being TIFF, and will not load with tiffload. Instead, try reading the whole of the first directory. This is enough to stop tiffload trying to load files where it will simply fail immediately, and make libvips fall back to eg. imagemagick. see #1304

jcupitt · 2019-06-07T17:01:02Z

OK, it seems to work now, though it's not quick. I see:

$ time vipsheader RAW_SONY_A900.ARW 

(vipsheader:3630): VIPS-WARNING **: 17:44:26.379: Unknown field with tag 50341 (0xc4a5) encountered
RAW_SONY_A900.ARW: 6080x4049 ushort, 3 bands, rgb16, magickload

real	0m31.708s
user	0m54.503s
sys	0m0.787s

It's trying to load the first directory with libtiff and failing, then falling back tom imagemagick. IM in turn is shelling out to ufraw-batch to convert the file to a PPM in /tmp, loading that into memory, and passing it back to libvips to print the dimensions.

This works too:

$ time vipsthumbnail RAW_SONY_A900.ARW -o x.jpg

(vipsthumbnail:4934): VIPS-WARNING **: 17:55:31.651: Unknown field with tag 50341 (0xc4a5) encountered

(vipsthumbnail:4934): VIPS-WARNING **: 17:55:47.358: Unknown field with tag 50341 (0xc4a5) encountered

(vipsthumbnail:4934): VIPS-WARNING **: 17:56:17.814: Unknown field with tag 50341 (0xc4a5) encountered

real	1m17.041s
user	2m11.217s
sys	0m1.838s

Though it's even slower: it's converting the file once to get the header, then (unfortunately) converting again to get the pixels.

przemyslawpluta · 2019-06-07T21:38:56Z

Overall it's probably better when it resolves with outcome even if it takes longer than fail fast without outcome.

jcupitt · 2019-06-10T07:11:05Z

I found a speedup: it's down to 15s for vipsheader, 30s for vipsthumbnail.

This improvement should be in 8.8.1, thanks for pointing it out!

jcupitt · 2019-07-08T07:30:16Z

OK, 8.8.1 is out now with this improvement. Thanks!

jcupitt added the question label May 3, 2019

lovell mentioned this issue Jun 6, 2019

Sharp fails to provide metadata for Sony .ARW RAW files lovell/sharp#1745

Closed

jcupitt added enhancement in development and removed question labels Jun 10, 2019

jcupitt closed this as completed Jul 8, 2019

jpalumickas mentioned this issue Feb 18, 2020

Support RAW images imgproxy/imgproxy#343

Open

alextran1502 mentioned this issue Dec 22, 2022

[BUG] Sony RAW(.ARW) cannot be uploaded immich-app/immich#1109

Closed

4 tasks

afcapel mentioned this issue Feb 27, 2023

Identify Sony and Canon raw images as subtypes of image/tiff rails/marcel#89

Merged

This was referenced Apr 3, 2023

[BUG] Sony RAW (.ARW) unsupported immich-app/immich#2156

Closed

Support Sony RAW (.ARW) lovell/sharp#3612

Closed

jrasm91 mentioned this issue Aug 11, 2023

Use imagemagick to generate nef and cr2 thumbnails immich-app/immich#3648

Closed

javixuwi mentioned this issue Oct 17, 2023

ARW support bpatrik/pigallery2#746

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to open RAW images #1304

Unable to open RAW images #1304

rps01 commented May 2, 2019

jcupitt commented May 3, 2019

przemyslawpluta commented Jun 6, 2019 •

edited

jcupitt commented Jun 7, 2019

przemyslawpluta commented Jun 7, 2019 •

edited

jcupitt commented Jun 7, 2019

przemyslawpluta commented Jun 7, 2019 •

edited

jcupitt commented Jun 7, 2019

przemyslawpluta commented Jun 7, 2019 •

edited

przemyslawpluta commented Jun 7, 2019 •

edited

jcupitt commented Jun 7, 2019

przemyslawpluta commented Jun 7, 2019

jcupitt commented Jun 7, 2019

przemyslawpluta commented Jun 7, 2019

jcupitt commented Jun 10, 2019

jcupitt commented Jul 8, 2019

Unable to open RAW images #1304

Unable to open RAW images #1304

Comments

rps01 commented May 2, 2019

jcupitt commented May 3, 2019

przemyslawpluta commented Jun 6, 2019 • edited

jcupitt commented Jun 7, 2019

przemyslawpluta commented Jun 7, 2019 • edited

jcupitt commented Jun 7, 2019

przemyslawpluta commented Jun 7, 2019 • edited

jcupitt commented Jun 7, 2019

przemyslawpluta commented Jun 7, 2019 • edited

przemyslawpluta commented Jun 7, 2019 • edited

jcupitt commented Jun 7, 2019

przemyslawpluta commented Jun 7, 2019

jcupitt commented Jun 7, 2019

przemyslawpluta commented Jun 7, 2019

jcupitt commented Jun 10, 2019

jcupitt commented Jul 8, 2019

przemyslawpluta commented Jun 6, 2019 •

edited

przemyslawpluta commented Jun 7, 2019 •

edited

przemyslawpluta commented Jun 7, 2019 •

edited

przemyslawpluta commented Jun 7, 2019 •

edited

przemyslawpluta commented Jun 7, 2019 •

edited