Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panasonic G9 high-resolution RW2 #126

Closed
shoffmeister opened this Issue May 16, 2018 · 22 comments

Comments

Projects
None yet
3 participants
@shoffmeister
Copy link
Contributor

commented May 16, 2018

darktable 2.4.3 (for Windows) appears to fail reading Panasonic LUMIX DC-G9 high-resolution RW2 raw files.

The error message emitted by darktable is "failed to read camera white balance".

The photos have the following dimensions (according to exiftool):

  • high-resolution item ("HR"): 10368x7776
  • "simul record normal shot" (simulate recording a single normal shot = "SIM"): 5184x3888

If one was to trust the documentation, then SIM "should be" a totally plain normal RW2 file: "Saves the first picture of pictures taken before the combining process ... [is started]"

Whatever hides inside HR - I have no idea. Theoretically this is the combined result of a total of eight pixel-shifted exposures.

@shoffmeister

This comment has been minimized.

Copy link
Contributor Author

commented May 16, 2018

FWIW, RawTherapee 5.4 willingly opens both RW2, but reads /renders total garbage for both files.

@LebedevRI

This comment has been minimized.

Copy link
Member

commented May 16, 2018

Is there a sample on RPU?

@shoffmeister

This comment has been minimized.

Copy link
Contributor Author

commented May 16, 2018

I just (thought that I) uploaded two files (32 MB, 125 MB) onto RPU - I got "thanks", but I do not see these files in the repo list?

@shoffmeister

This comment has been minimized.

Copy link
Contributor Author

commented May 16, 2018

FWIW, the Panasonic documentation for the Panasonic LUMIX DC-G9 can be found at https://eww.pavc.panasonic.co.jp/dscoi/DC-G9/EG_EC_EF_EB/DC-G9_DVQP1406ZA_eng.pdf - the magic keyword is "[High Resolution Mode]" (page 196)

@shoffmeister

This comment has been minimized.

Copy link
Contributor Author

commented May 16, 2018

FWIW, SILKYPIX Developer Studio 4.4 SE (Version: 4.4.14.5) - for Panasonic cameras only, runs on Windows and OSX - is able to open these raw files and render them.

Don't slaughter me for mentioning that https://www.isl.co.jp/SILKYPIX/english/p/support/download/ has a download available and that installation on Windows is painless, not needing anything like a serial number or what-not. But that download is limited to processing files produced by Panasonic cameras.

@LebedevRI LebedevRI changed the title Panasonic G9 high-resolution RW2 - "failed to read camera white balance" Panasonic G9 high-resolution RW2 May 16, 2018

@LebedevRI

This comment has been minimized.

Copy link
Member

commented May 16, 2018

I got "thanks", but I do not see these files in the repo list?

There is a pre-review.
Thanks for submitting those samples, but they are really unhelpful.
Please contribute something better/more recognizable, like daylight landscape.

I have taken a quick look at the raws themselves, they look to be still compressed,
but interestingly the width is no longer a multiple of 14, which breaks all later assumptions.

@shoffmeister

This comment has been minimized.

Copy link
Contributor Author

commented May 16, 2018

I will go hunting for daylight ASAP ;)

May I suggest that https://raw.pixls.us/ gets some language which indicates preferences general preferences? Right now the web site makes it appear as if anything is good enough. Perhaps mention: "Ideally a correctly exposed photo, taken in daylight, of a landscape"?

@LebedevRI

This comment has been minimized.

Copy link
Member

commented May 16, 2018

You mean like this one:
image

@shoffmeister

This comment has been minimized.

Copy link
Contributor Author

commented May 16, 2018

I mean the (missing) bullet item / paragraph immediately before "We are NOT looking for:" ;)

@shoffmeister

This comment has been minimized.

Copy link
Contributor Author

commented May 16, 2018

FWIW, Adobe DNG Converter 10.1 is able to read both files and transform them into DNGs of size 13 MB and 48 MB, respectively.

Note how the files shrank quite considerably from RW2 -> DNG (32 -> 13 MB; 125 -> 48 MB), while declared resolution remains at 5184x3888 and 10368x7776 respectively.

@shoffmeister

This comment has been minimized.

Copy link
Contributor Author

commented May 16, 2018

Ah ... and now this gets rather fancy: darktable reads the DNG files produced by Adobe DNG Converter 10.1 - and, according to darktable (image information width/height), the resolution of these DNG files is 5280x3904 and 10480x7794.

I have reconfirmed that exiftool shows 5184x3888 and 10368x7776 for both RW2 and DNG.

@shoffmeister

This comment has been minimized.

Copy link
Contributor Author

commented May 17, 2018

I have just uploaded two new files at https://raw.pixls.us/

These new files show the nice green tree watered by rainy weather just in front of my door. Better subjects are conditional to better weather ;)

@shoffmeister

This comment has been minimized.

Copy link
Contributor Author

commented May 19, 2018

I just discovered that FastRawViewer 1.4.6 (https://www.fastrawviewer.com/blog/FastRawViewer-1-4-6-release) has added support for the Panasonic G9 in high-res mode.

Perhaps @LibRaw might be able to chime in?

@LibRaw

This comment has been minimized.

Copy link

commented May 19, 2018

[s]We'll publish G9 support in next snapshot (end of summer, or a little later). Now we're focused on 0.19 final polishing[/s]

Correction: it is supported in LibRaw 0.19

@shoffmeister

This comment has been minimized.

Copy link
Contributor Author

commented May 19, 2018

Ah: "Alexey Danilchenko's GH5S/G9-hires patch

disable rawspeed for new panasonic decoder"

LibRaw/LibRaw@9fc4e05

@LibRaw

This comment has been minimized.

Copy link

commented May 19, 2018

I was sure, we've added it to 0.19, but looked into code and have not see it,

The problem was: I've looked into 0.19-tarball dcraw/dcraw.c which is original Dave Coffin's code.

Yes, G9/GH5s are already supported by current 0.19 pre-release (beta5)

@LibRaw

This comment has been minimized.

Copy link

commented May 19, 2018

(G9-> read as G9 high res)

@shoffmeister

This comment has been minimized.

Copy link
Contributor Author

commented May 21, 2018

Reading around, it would seem as if Panasonic is switching to a new raw format for the newer cameras. Specifically, the Panasonic DC-GH5S seems to use the same format "natively" as the Panasonic DC-G9 in its dedicated high-resolution mode. (GH5s raws can be found at https://www.dpreview.com/reviews/panasonic-lumix-dc-gh5s-review/6, nothing on RPU as I am writing this)

The commit in the libraw repo takes care of this, by way of (also) supporting the 14 bit resolution of the GH5s for format version 5.

Also in the libraw repo is peculiar handling of black-level information for the new format version 5, see LibRaw/LibRaw@f86a450 (pana_encoding == 4 is old, pana_encoding == 5 is the newly introduced raw format)

There is also interesting information in LibRaw/LibRaw@20efe0c where compression information is read (but, later, this is not used, it would seem).

@LebedevRI

This comment has been minimized.

Copy link
Member

commented May 21, 2018

Reading around, it would seem as if Panasonic is switching to a new raw format for the newer cameras.

And still not LJPEG
(╯°□°)╯︵ ┻━┻)

@shoffmeister

This comment has been minimized.

Copy link
Contributor Author

commented May 21, 2018

... which, clueless as I am, I would consider beneficial for my own personal sanity, as LJPEG seems to be using Huffman encoding to (losslessly) compress? (https://en.wikipedia.org/wiki/Lossless_JPEG)

@LebedevRI

This comment has been minimized.

Copy link
Member

commented May 21, 2018

Yeah, all these manufacturer-specific compressions (RW2, ARW, ...) are pretty bad even at compressing,
and some even introduce artifacts.

@shoffmeister

This comment has been minimized.

Copy link
Contributor Author

commented May 21, 2018

Just for the public record: A very exploratory implementation is currently located at https://github.com/shoffmeister/rawspeed/tree/wip/rw2-g9-gh5s-support

LebedevRI added a commit to LebedevRI/rawspeed that referenced this issue May 25, 2018

Merge branch 'panasonic-v5-decompressor-cleanup' into develop
Adds support for newest Panasonic raw compression algorighm, v5.
Currently used on GH5s camera, and on high-res raws from G9 samples.
This does not actually handle the camera support part, that is TODO.

The ppm dump matches that of the DNG's produced by ADC from these raws.
Except the high-res raw, where there is some one-byte differences.

Even though this is based on the LibRaw code, i have cleaned it up
a little, rewritting basically every single line of source code...

Thanks goes to (in no particular order):
* Stefan Hoffmeister
  Initial porting of this code from LibRaw, raw camera samples.
* Arek Eych
  Raw camera samples.
* Alexey Danilchenko
  Initial implementation of the algo in LibRaw.

Refs. darktable-org#136
Refs. darktable-org#126

* panasonic-v5-decompressor-cleanup: (29 commits)
  Bump Rw2Decoder version now that PanasonicDecompressorV5 is good.
  PanasonicDecompressorV5: rewrite actual pixel decoding with BitPumpLSB.
  PanasonicDecompressorV5::ProxyStream::parseBlock(): use vector::insert()
  PanasonicDecompressorV5: parallelize.
  PanasonicDecompressorV5: rewrite ctor without integer overflow UB
  Add PanasonicDecompressorV5 fuzzer.
  PanasonicDecompressorV5::chopInputIntoBlocks(): fix asserts/use back()
  PanasonicDecompressorV5: make numBlocks member variable
  PanasonicDecompressorV5: cleanup some comments/names
  PanasonicDecompressorV5: unswitch the innermost bps-based if.
  PanasonicDecompressorV5: s/DataPump/ProxyStream/
  PanasonicDecompressorV5: rewrite with more explicit block usage.
  PanasonicDecompressorV5: rewrite block handling logic.
  PanasonicDecompressorV5: small stylistic cleanup
  PanasonicDecompressorV5: slightly tidy-up BlockSize/comments
  PanasonicDecompressorV5: get rid of threading.
  Rw2Decoder::decodeRawInternal(): let ByteStream itself handle overflow
  Rw2Decoder: make offset a local variable
  Rw2Decoder: make section_split_offset a local variable
  PanasonicDecompressorV5: drop nop bad pixel handling.
  ...

LebedevRI added a commit that referenced this issue Jun 6, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.