Support for cinema dng? #368
Comments
|
Cinema DNG is not yet supported. |
|
What can I do to help? |
|
fwiw this rawspeed issue seems to be related: darktable-org/rawspeed#189 edit: ignore me. cytrinox already has support in rawler for the updated prediction models. the issue with files such as these: https://discuss.pixls.us/t/cinemadng-workflow-for-blackmagic-or-sigma-fp-cameras-in-darktable/42191 appears to be that it uses tiling in a way that confuses the decoder. somehow it tries to decode the tiles into an image that has a quarter of the width and half the height or so. i'm a bit confused about the various resolutions and potential doubling factors involved here. |
|
Maybe this is helpful. https://thndl.com/how-dng-compresses-raw-data-with-lossless-jpeg92.html Under the "How does DNG use LJ92?" section. |
|
for the record, i think the blackmagic way of de-swizzling the decoded jpg data is required, as outlined here: darktable-org/rawspeed#334 (comment) i think cytrinox is well aware of this but probably doesn't have the time/priority for it, but i'll leave the link here maybe someone else has in the meantime.. |
|
I'm not sure if it helps, but if I try to convert a 12 bit cinema dng image from a SIGMA fp I get this error: Test raw dng files from the camera shot in cinema mode are available here: https://drive.google.com/drive/folders/1YXz9rU3lpQJkaKTDzU00j2fA8K0VMpo8 8 bit dng variants seem to convert to dng but the output is quite a bit larger than the input image.. not sure why. It would be really cool and useful to be able to specify the transfer function and bit depth of the output dng, as well as enabling lossless jpg compression. For example storing the 12 bit dng as 10 bit log encoded would reduce filesize while preserving quality. |
|
As mentioned in the discuss.pixls.us thread, the Sigma scheme is different to Blackmagic, so this is probably better split up in several issues/feature requests. To sum it up, this is missing:
The latter would also cover recent DJI drone raw video output as well. In addition there is the custom Blackmagic lossy extension (sctrictly speaking breaking the DNG spec because of piggybacking the compression tag in an "illegal" way):
|
|
Theoretically, BPS=12 is already supported but due a bug dnglab could not handle BPS=12 for Little-Endian files. Fixed in #407 I've checked the 3:1/4:1 BlackMagic DNGs - IMHO these files violates DNG spec. Even Adobe software can't open them. I've no intention to "fix" Blackmagic bugs, so support for these specific compression modes are out of scope for me. |
|
Thank you for the fix for 12 bit cinema dng files from the Sigma fp. I tried compiling dnglab from main with #407 merged, and the behavior is still the same for 10 bit images (The Sigma fp can shoot 8, 10, and 12 bit cinema dng images in cinema mode). It would be nice to have 10 bit images supported as well if it is possible. |
|
@jedypod from your sample files, how did you get 2K resolution? Are you using a lossy compression method? |
|
@devonstanczyk Hopefully that helps, let me know if I can clarify or expand on anything! Happy to help if I can. |
|
@jedypod are you using the original sigma FP or FPL? |
|
@devonstanczyk it's the original sigma fp |
|
Just wanted to follow up on this. I compiled dnglab from main with #408 merged and tested it with the sigma fp test frames. 10 bit frames work perfectly now, thanks so much for the fix!
I think I made a mistake before. I'm not sure what I was doing wrong, but I am now quite certain that the raw dng frames out of the camera are uncompressed. Using dnglab to re-encode them with lossless LJpeg92 compression massively reduces the disk usage, which is really awesome. Using something like the following command:
Here's a summary of compressed sizes using the different LJ92 predictor levels for this example frame: I'm not sure exactly what the predictor size does.. reading this article I guess maybe it's the window size or how many pixels to look at? The default of 1 seems good, but 7 seems to compress slightly better for the frames I tested. Anyway I just wanted to follow up. TLDR; |
|
@jedypod so everything is working with Sigma FP DNGs? Where can I find the build? |
|
@devonstanczyk I just compiled from the main branch of the git repo using cargo. Very easy.. One unfortunate thing I realized after testing a bit more is that Resolve won't load the compressed DNGs. It does not seem to be related to the compression because I have other 14bit lj92 compressed dngs which work fine. Maybe it's the 10 and 12 bit variations that break it... :/ |
|
Bummer! Thanks for testing though! - Could it be that DNGlab is creating linear DNGs (no RAW sensor data)? |
I don't think so. I compared the raw bayer data dumped out using I dug up some old raw images shot with magic lantern on a Canon 60D, to see if the behavior is the same in resolve. Magic Lantern shoots raw video in the MLV format, and you can save out dng frames using software like MLVApp. I found an old MLV that is 12 bit, to see if it not working with resolve is because of the combination of bit depth and LJ92 compression.
Both of these images load fine in Resolve. Then I wanted to test the same dng compressed to LJ92 with It looks like it doesn't like the lack of proper camera metadata, which is unfortunate. It seems like it could still carry on and try to process the file.. but maybe there is some technical reason for the fail I don't understand. I even tried copying the metadata from a CR2 image shot with the same camera over to this dng, but am still getting the same error... Anyway, it would be great to see if we could track down why resolve doesn't like those LJ92 compressed DNG files rendered out with dnglab. Here are the two test images i described above if it helps: |
|
Awesome findings! Agreed, if Cinema DNGs processed with DNGlab are unusable in DaVinci, that's a bummer. Are you noticing if the bit depth stays the same after processing with DNGlab? For example, whenever I would use Adobe DNG converter, it would strip all cinema related Metadata (timecode, etc.), and convert any bit depth up to 16 bit. This conversion obviously is a pointless expansion of 12 bit data to fit into 16 bit. The lack of cinema metadata tags is a huge headache too after using Adobe DNG Converter. SlimRAW does an amazing job at leaving all data intact, and only focusing on lossless compression or lossy compression to the RAW data. |
@cytrinox This is great! I'm not sure if you are including the (probably out of scope) issues of the dngs output from dnglab not loading in resolve, or the dngs output from MLVApp failing to load in dnglab.. but I compiled again from main and both of these still seem to have the same behavior.
Yes the bitdepth stays the same after LJ92 compression, which is great. I could see it being useful being able to specify the output bitdepth though. For example if a user wanted to take a 12 bit dng and compress and encode to a 10 bit output using a log linearization table or something like that. Interestingly, if you process an uncompressed output with dnglab it looks like the bitdepth is increased to 16 bit. For example, processing A002_651_20240229_000001_2k_12bit.DNG using seems to result in a 16bit dng output (if Even more interesting, this uncompressed dng also doesn't load in Resolve, so it must not be related to the LJ92 compression... Maybe there is something in the metadata or the way the dng is encoded that makes @devonstanczyk If you can get cargo (the rust build system) installed on your computer (maybe this guide would help?), compiling dnglab is as simple as if all goes well, the compiled binary will be in target/release/dnglab. |
|
Thanks @jedypod! What if you took an uncompressed DNG straight from the memory card, and ran it through adobe DNG converter, and then through DNGLab. You could then compare how the data differs between the two. Maybe compare them with exiftool tool too to check metadata as well. Maybe this could lead to an answer. Use a simple code editor like BBedit to find/compare the differences in data. |
|
Just getting around to testing. I've installed both Rust and DNGlab. Everytime I try to run a command, I get "command not found: dnglab" I'm no newb when it comes to imaging, but I'm certainly a CLI newb :/ |
|
You would have to copy the |
|
Using my own Cinema DNG 12bit images from my Sigma FP, I ran the following code: @jedypod I am having similar issues with DaVinci Resolve. The new lossless DNG does not load correctly into DaVinci Resolve. I am going to convert the original DNG with Adobe DNG Converter, then compare the metadata of it, to the DNG from DNGlab, and original unmodified DNG. I'll output text files from EXIFtool. Using the diff command will show differences in the text and show what's present/missing. Adobe-DNG-Converter-DNG.txt |
|
@cytrinox Found some interesting things. When loading DNG's from DNGlab and DNG's from Adobe DNG Converter into Adobe Camera RAW:
Typically when Adobe's non-RAW profiles are the only ones selectable, it often means that the image isn't RAW/RAW sensor data/is a raster image. Is DNGlab creating "linear DNGs"? That is, demosaiced output? A little more info on Linear DNG here https://helpx.adobe.com/camera-raw/using/adobe-dng-converter.html (Search for "linear" in the page to find the bullet point.) Separate from above; after evaluating output from DNGlab with EXIF tool, I noticed that the "default crop" tag is set to 3832x2155 when it should be 3840x2160. |
|
Did another test. Shot on Sigma FP, cinema DNG, 24FPS, 12bpc. I took the original DNG from the Sigma FP, ran it through DNGlab several times testing each predictor using the following command: Within Davinci Resolve, I've imported each DNG. The DNG's from DNGlab are not loading correctly in Davinci Resolve. They are also reporting the wrong resolution (2048x1556), and reporting 10bit. I had another idea: I ran the DNG's converted from DNGlab, through Adobe DNG Converter, and they loaded correctly. Although they are loading in as 3832x2155 (mentioned in the above comment). See attached image for a screenshot from Davinci Resolve and how it's interpreting the data.
|
|
Recently I've merged two PRs fixing minor issues with EXIF tags and DNG crop, but I don't think these are the root cause for Resolve problems. |
|
@cytrinox I have a hunch about why resolve does not read dng images created by dnglab. I'll start with First I'll create a compressed dng using dnglab. Trying to load both of these dngs into resolve: the original Inspecting the structure of the dng using the following exiv2 commands, (These txt files are also in the zip file) it looks like the dnglab generated dng has multiple sub-images, while the original has only one. I'm not sure if I'm using the right terminology here as I'm not very familiar with the technical aspects of the file format. I'm just wondering if Cinema DNG is not meant to have image data stored in subimages maybe? Structure of original Structure of dnglab generated dng image @devonstanczyk weirdly, for me a dng generated by Adobe DNG Converter 9.3 on windows does not display correctly in Resolve (18.5.1 on linux) either, though the resolution is reported correctly (though with a different crop than the original like you discovered). |
|
Does it load with |
|
@cytrinox Hm, no it seems like the behavior in resolve is the same with The structure also seems to be the same: Hope it helps, and let me know if I can test anything else! Would love to get to the bottom of this... |
|
I've merged PR #423 into main, this contains some minor fixes for DNG files. Thumbnails are written into root IFD, but if no thumbnail is available/written, RAW image could be written into root IFD instead of a sub IFD. I was able to load converted DNGs into DR with/without thumbnails/previews and all (1-7) different LJPEG predictors. The only thing I can't get work is uncompressed DNGs. Usually uncompressed data is split into multiple stripes. If I write one big stripe (that was Sigma Fp DNGs uses), DR would load the DNG. If I use multiple strips, DR rejects the image. After some experiments, I would say this is a bug in DR, as all releveant tags are correct and multistrip DNGs are loading fine in a wide range of applications. As Adobe DNG Converter usually writes multistrip DNGs when in settings "uncompressed dng" is choosen, someone could test if such files would load in DR or not. |
|
I compiled from main and tested again loading dnglab compressed dngs in resolve. It's working!! Thank you so much @cytrinox, this is awesome! This fix unlocks using dnglab for compression in my Sigma fp post workflow and is going to save me a literal crap-ton of disk space. Out of curiosity I did a few more tests with Adobe DNG Converter 14.4. I don't think it's actually possible write out uncompressed dng images from that software. The only compression option I could find is "enable lossy compression", which seems to be something different. Using these settings: Interestingly, the non-lossy export from adobe dng converter also loads fine in resolve! (Not sure what I was doing wrong before). |
|
You have to choose custom at compatibility, then check "uncompressed". |
|
Ohh, got it! Thanks for the help. I saved out an uncompressed version of the same frame using Adobe DNG Converter. Surprisingly, this dng image also loads fine in Resolve. And I've confirmed that an uncompressed dng generated with dnglab does not load sucessfully in resolve. If it helps, here is a zip file containing both images, and exiv2 structure info: Adobe DNG structure: dnglab structure |
|
@cytrinox Yes, agreeing with @jedypod, this is HUGE for us Sigma FP shooters. This will save us soooo much disk space! @cytrinox where can I find DNG specification documents? @jedypod you mentioned earlier that you compared the RAW data between two DNG's. Can you start me down a path to learn how to do the same/how might I export a .txt of the RAW bayer data from a DNG? If resolution is being reported incorrectly in Resolve, does that mean that the image has been reduced in resolution? Or is it just a metadata issue? |
|
@devonstanczyk You can use #!/bin/bash
file_list=("$@")
for ((i = 0; i < ${#file_list[@]}; i++)); do
filepath=$(realpath "${file_list[$i]}")
filename=$(basename -- "$filepath")
extension="${filename##*.}"
filename="${filename%.*}"
echo -e "Processing: \t${filepath}\n-->\t$(pwd)/${filename}.tif"
dcraw_emu -o 0 -4 -T -W -r 1 1 1 1 -mem -disinterp -Z - "${filepath}" > "${filename}_mosaic.tif"
doneAs for Resolve reporting the resolution wrong, it's purely metadata. The image is not being resized. It looks like 2048x1556 is some kind of fallback default. That might be another thing to look at though @cytrinox : The resolution is being reported wrong in resolve for all dngs processed through dnglab. This is not happening with dngs processed through Adobe DNG converter. Maybe it's some issue in the metadata encoding? |
|
Also testing again, I'm getting same issue described above when loading DNGlab DNG's into Adobe Photoshop/Camera RAW: No RAW color profiles are available for DNG's processed with DNGlab. If DNGlab is retaining the bayer data, why does Adobe not recognize this? DNGlab is not creating demosaiced/linear DNG's correct? |
Hi there,
I shoot with a Sigma FP, CinemaDNG. Looking for a solution that preserves all metadata but applies lossless compression. Will this support Sigma FP and CDNG spec metadata?
The text was updated successfully, but these errors were encountered: