RGGB compression somtimes worse than RawSpeedCompress+LZMA2

[StephanBusch]

In my ongoing tests with RGGB created by dcraw compression of FLIF is sometimes worse
than RawSpeedCompress+LZMA2. RawSpeedCompress uses different Delta versions for uncompressed pixel data and applies ZigZag-encoding which helps all kinds of general purpose compressors.

https://drive.google.com/file/d/0ByLIAFlgldSoNFR6RmcwdTgxQTQ/view?usp=sharing
leica_m82_05.rggb 11.359.612 (FLIF 0.1)
leica_m82_05.rggb 10.928.429 (FLIF 0.1.3_slower_but_stronger -n)
leica_m82_05.rggb 5.560.365 (RawSpeedCompress3 + LZMA2)

https://drive.google.com/file/d/0ByLIAFlgldSoalhkSkdsNmJNaDA/view?usp=sharing
nikon_1_v2_17.rggb 9.931.720 (FLIF 0.1)
nikon_1_v2_17.rggb 9.636.696 (FLIF 0.1.3_slower_but_stronger -n)
nikon_1_v2_17.rggb 8.810.004 (RawSpeedCompress3 + LZMA2)

https://drive.google.com/file/d/0ByLIAFlgldSoZmJDUXl2YWVPS2M/view?usp=sharing
nikon_d5200_14.rggb 24.334.053 (FLIF 0.1)
nikon_d5200_14.rggb 23.684.359 (FLIF 0.1.3_slower_but_stronger -n)
nikon_d5200_14.rggb 21.527.675 (RawSpeedCompress3 + LZMA2)

Can you please check why FLIF is worse here?
Will future versions also get something like Delta+ZigZag+LZ?

In almost every case the -n switch provided better compression on RGGB

[psykauze]

Could you try renaming the RGGB file in PPM and compression it in FLIF interlaced ?

[matthiaskrgr]

I wrote a simple crusher script for flif (matthiaskrgr/flifcrush), you can try running it on one of the images:

export FLIF=/path/to/flif/binary
main.py nikon_d5200_14.rggb

It will take a lot of time but it might get the size down a bit further.

edit: BE CAREFUL, it might need a lot of ram

[jonsneyers]

I found out what is happening. I know how to solve this. I just need to implement it. Soon FLIF will beat that other method.

[jonsneyers]

Here you go:

11,359,333  leica_m82_05.rggb.flif0.1.3
5,117,762  leica_m82_05.rggb.flif0.1.4
8,450,275  nikon_1_v2_17.rggb.flif0.1.4
20,384,559 nikon_d5200_14.rggb.flif0.1.4

[StephanBusch]

There are still RGGB that compress better with Delta + ZigZag + LZMA2 than they do with FLIF:

https://drive.google.com/file/d/0ByLIAFlgldSoNlR4ZURaTUE2QTg/view?usp=sharing
pentax_q10_19.rggb 10.692.568 (FLIF 0.1.4)
pentax_q10_19.rggb 10.167.315 (Delta + ZigZag + LZMA2)

https://drive.google.com/file/d/0ByLIAFlgldSoSkZyQzJvWFNpQTQ/view?usp=sharing
panasonic_lumix_dmc_gh3_10.rggb 14.186.367 (FLIF 0.1.4)
panasonic_lumix_dmc_gh3_10.rggb 13.496.039 (Delta + ZigZag + LZMA2)

PLC helped a lot - on most files FLIF is stronger than without it, and sometimes it also
outperforms RawSpeedCompress3 + Delta + ZigZag + MCM 0.83 -x9
but I think it has the capability of beating it on every RGGB image.
can you check that again, please?
See the almost complete test here: http://www.squeezechart.com/camera-raw.html

[psykauze]

As I see, Flif outperform the other method by >1% on average, that is really good. Because each compressor has its own method, sometimes one compressor that usually be worse than others can be more efficient for some case.

You can check some plots at issue #144 you will see that FLIF can sometimes outperform JPEG in its own sandbox, and sometimes webp outperform FLIF.

[jonsneyers]

The Bayer CFA layout is different for different camera models; FLIF currently assumes some specific subpixel layout for its YIQ transform, and if this assumption is wrong, then compression suffers a bit.
You can use flif -R to disable YIQ. Also, disabling channel compactification with -C can help.

14,005,394 panasonic_lumix_dmc_gh3_10.rggb.R.flif
13,416,647 panasonic_lumix_dmc_gh3_10.rggb.RC.flif

10,481,645 pentax_q10_19.rggb.R.flif
10,481,603 pentax_q10_19.rggb.RC.flif

Does anyone have a good source of information on these subpixel layouts? I tried reading the dcraw source code, but that code is not very readable...

[psykauze]

Dcraw's exif output can show the CFA pattern as a matrix {Color1, Color2}{Color3, Color4}. DNGs and many (all?) RAWs formats has an Exif data for it.

Edit: I've found the raw files of panasonic_lumix_dmc_gh3_*.rw2 here http://www.photographyblog.com/reviews/panasonic_lumix_dmc_gh3_review/sample_images/
The CFA patern is GB/RG

[StephanBusch]

maybe RawSpeed can help?
https://github.com/klauspost/rawspeed/tree/Rawspeed
they have a camera.xml database for supporting new cameras:
https://github.com/klauspost/rawspeed/blob/develop/cameras-xml.md

[psykauze]

I found that Exiftools does not mention the CFA pattern of the RW2, libopenraw documentation say that the pattern should be BGGR and DCRAW say the CFA pattern of the file is GBRG.

Which one to trust ?

From "ISO TC 42 N 5737" TIFF & DNGs allows patterns matrix bigger than 2x2 and can use Red, Green, Blue, Cyan, Magenta, Yellow or White colors. TIFFtag name is CFAPattern (0x828e), maybe you could check this value before converting it.

[jonsneyers]

Also: how does DCRAW output the extracted data?
When FLIF reads a .rggb file, it is using the following layout, which I think is what DCRAW produces:

GG
RB

Another complication is that these camera raw formats do not seem to have the same range for R, G and B. I'm not even sure if G1 and G2 are always in the same range. So that also messes up the YIQ.

Probably the way to get best compression (if you're not interested in progressive decoding) is to use flif -Rbn, with or without -C. You can usually improve further by using higher -r and -S (simultaneously), e.g. flif -RCbnS120 -r8.

[psykauze]

I'm sorry but DCRAW outputs with the original pattern, the most common case is:
RG
GB

I'm working at image-rggb.cpp there is also an issue for maxval < 0xff at blue plane

[jonsneyers]

Thanks, @psykauze . Now we should probably preprocess raw input to be in RG GB format.

[psykauze]

You should also change the decoder part ;)

Edit.: I've made some tests with "only" the planes correction here the results:
filename,before,after
TEST/canon_eos_5d_mark_iii_05.rggb,20098313,20175045
TEST/canon_eos_6d_14.rggb,19328964,19460476
TEST/canon_eos_m_04.rggb,18844941,18949992
TEST/fujifilm_finepix_x100_11.rggb,9652032,9640802
TEST/fujifilm_x_e1_20.rggb,11347023,11308835
TEST/fujifilm_xf1_08.rggb,6197773,6216062
TEST/leica_m82_05.rggb,5117725,5132863
TEST/leica_x1_10.rggb,9790809,9735075
TEST/nikon_1_v2_17.rggb,8449538,8416517
TEST/nikon_d4_10.rggb,15701302,15851157
TEST/nikon_d5200_14.rggb,20384199,20444366
TEST/olympus_epm2_16.rggb,13435899,13375579
TEST/olympus_om_d_e_m5_24.rggb,13260184,13345509
TEST/olympus_xz2_10.rggb,10397033,10394995
TEST/panasonic_lumix_dmc_gh3_10.rggb,14186056,14233905
TEST/panasonic_lumix_g5_15.rggb,14095164,14130656
TEST/pentax_k5_ii_12.rggb,18068348,18073325
TEST/pentax_q10_19.rggb,10692439,10398867
TEST/samsung_nx1000_19.rggb,14594902,14513184
TEST/samsung_nx20_01.rggb,15892754,15799060
TEST/sony_a55.rggb,13066285,13096163
TEST/sony_a77_08.rggb,17466963,17564793
TEST/sony_a99_04.rggb,15125553,15079356

[psykauze]

So...
I've made some analysis on the RAW files and found there is also rotation to take account. The results after rotating images for getting the right CFA pattern is worse than before.

Here the new results:
log-newRGGB.txt

[jonsneyers]

Maybe one image per camera model is not enough to really draw conclusions, but I wonder what is going on here. The differences in compression when changing the CFA pattern are not huge anyway, which suggests to me that doing the YIQ transform on R G1 B while storing 1+G2 in the 'alpha' channel is not the best approach -- it was just a quick & dirty hack anyway, to match RGGB with the RGBA functionality that was already there. Maybe a custom color transform for RGGB would be better.

[StephanBusch]

I would also vote for a custom color transformfor RGGB.
At least for Sony there is an explanation:
http://diglloyd.com/blog/2014/20140212_2-SonyA7-RawDigger-posterization.html

The Fuji CFA pattern is different from others - see wikipedia:
https://en.wikipedia.org/wiki/Bayer_filter

[psykauze]

That's why I disqualified the fuji x e1.

Also, I would like to try the R, (G1+G2)/2, B, (G1-G2) algo.

You're right too concerning the R, G and B ranges, there's is correction factor for each color.

[jonsneyers]

Converting to R, (G1+G2)/2, B, (G1-G2) is not lossless: one bit is lost in the division by two and it cannot be recovered. But we could try to directly apply a modified variant of YIQ, maybe something like this:
Y = R+B+G1+G2
I = R - B
Q = R+B -G1-G2
Gdiff = G1-G2

I'm not sure if encoding G1-G2 is really more efficient than just encoding G1 or G2. The difference is expected to be closer to zero, but it needs an extra bit to be represented.

[psykauze]

There's no lost on R, (G1+G2)/2, B, (G1-G2) conversion if you check if numbers is odd or even:

X = (G1+G2)/2 (Always round to lower int like N.5 => N); W = G1-G2;
Case W odd: G1 = X + W/2 ; G2 = X - W/2 ;
Case W even: G1 = X + (W+1)/2 ; G2 = X - (W-1)/2;

You're right, difference need to be represented with an extra bit but I think this W extra bit can be moved to X in some cases (like if Xdec+Wdec/2 > 0xFFFF then that means Xdec = Xenc << 1 + extra_bit). I need some calculations to check if it is possible.

[jonsneyers]

What is Y in that case distinction?

[psykauze]

Y was W I've edited the post sorry.

[jonsneyers]

Ah, right, the parity of the sum is equal to the parity of the difference so there would indeed be enough information to restore the exact sum. I'll try that transformation.

Also I'm assuming that raw files have no more than 14 bpc, is that a valid assumption? It helps if we want to represent G1-G2 in an uint16_t...

[psykauze]

I haven't see yet sensors up to 14bits so I think this is a valid assumption.

Be carreful, (G1 - G2) is a signed value but maybe you want store the data as '(Diff << 1) + sign' ?

[StephanBusch]

I cannot use that scripts raw2flif and raw2rggb because I don't have a linux system.
Here are the results of FLIF 0.1.7 -n without scripts and without rotation:
http://www.squeezechart.com/camera-raw4.html

[psykauze]

@StephanBusch Here is a *.bat that do the same as raw2rggb (It's buggy but it works) You need exiftool.exe, dcraw.exe and sed.exe (and its dependencies, see GnuWin website) in the same directory.
raw2rggb.bat.txt

@jonsneyers Please add this script in the root directory please. Also add this correction.
raw2rggb.sh.txt

[StephanBusch]

@psykauze Thank you for the .bat file.
Since my benchmarks compare more compressors on exctly the same input files, rotating would feel like cheating so I can test that script but I cannot publish that results because then the other compressors also need to be tested with rotated files. This would be different, of course, if Flif itself would detect CFA pattern and if it would rotate automatically.
My benchmark was created to see how far we can push lossless compression of RGGB and I really do appreciate the excellent work all of you do here. And I am very curious how your proposed MDL transform would work like and what further compression gains it will unlock.
Thank you very much for all your efforts

[psykauze]

I'm so sorry to read that :'(
The shell and *.bat files is for dev purpose, we know there is an issue about metadatas in the encoder. I consider this thing like a tweak, not a cheat

So there is some solutions I consider:

1. Working on rggb CFA detection internally but until we can't store the CFA Pattern in the flif file, it would be impossible to restore the original rggb file but we will have a great RGB file.
2. We can also rotate the file internally so the rggb output file can be decoded but rotated.
3. Developing the "channel sorting method" for YIQ (or MDL) transform will make the "CFA Pattern detection" useless (at least for "2x2 square" CFA Pattern)

[jonsneyers]

Storing metadata is trivial: just put it in a file and include it in the
ar, see raw2flif which extracts the exiv metadata and puts it in the
.flif file.
It would be cleaner if we use libraw and avoid shell scripts, and perhaps
let the decoder take rotation/CFA metadata into account directly.
On Nov 25, 2015 15:25, "psykauze" notifications@github.com wrote:

I'm so sorry to read that :'(
The shell and *.bat files is for dev purpose, we know there is an issue
about metadatas in the encoder. I consider this thing like a tweak, not a
cheat

So there is some solutions I consider:

1. Working on rggb CFA detection internally but until we can't store the
   CFA Pattern in the flif file, it would be impossible to restore the
   original rggb file but we will have a great RGB file.
2. We can also rotate the file internally so the rggb output file can be
   decoded but rotated.
3. Developing the "channel sorting method" for YIQ (or MDL) transform will
   make the "CFA Pattern detection" useless (at least for "square" CFA Pattern)

—
Reply to this email directly or view it on GitHub
#151 (comment).

[StephanBusch]

@psykauze I decided to run those tests with your batch because I want to help you developing FLIF by testing. I would also want to have a batch file that produces a .flif.
Instead of the linux-based ar you can use zlib or zstd.

Would it be better to design .flif with metadata as a compressed archive such as LibreOffice/MS Office or Mmiya-Leaf .eip files and store all metadata in a XML inside that archive?
Or would it be better to attach metadata directly to .flif as it is with all original raw files?

I have downloaded raw2rggb.bat and put exiftool.exe, dcraw.exe sed.exe (with libintl3.dll, regex2.dll and libiconv2.dll). Then tried to run raw2rggb.bat canon_eos_5d_mark_iii_05.cr2 canon.rggb but I got an error message: "(" cannot be syntactically processed at this position.

[psykauze]

OK, I've found the issue(s). I'm sorry, I've made some corrections but I haven't tested the bat file :( (Reason, I have not Windows at home). I'm working on it.

[jonsneyers]

Well, currently FLIF supports metadata as arbitrary files, so it could be compressed or plaintext XML, or whatever. The FLIF decoder simply ignores all metadata at the moment, but perhaps we should standardize some metadata in order to allow viewflif to do something sensible with raw files. Information about rotation, CFA layout and channel multipliers would be needed (and sufficient?) to produce a reasonable non-interpolated, half-resolution image.

It would be nice if we can eventually losslessly convert DNG to FLIF and back without any shell/bat scripts, perhaps using libraw.

[StephanBusch]

@psykauze
thank you very much

@jonsneyers
yes, that would be my biggest wish - convert DNG to FLIF and then back to DNG again :))

@psykauze @jonsneyers
Some men see things as they are and say "why?".
We dream of things that never were and say "Why not?"

[psykauze]

Sorry @StephanBusch, I've tried to debug the bat script but for some reason I've got random results (same input file, output file size change each try).

You could use the previous version of bat file. It was tested and it works.

[psykauze]

@StephanBusch @jonsneyers The bat script is now working. I've got some issues due to Batch scripting limitation.

raw2rggb.txt
raw2rggb.bat.txt

[StephanBusch]

thank you for the update
I tried it with: raw2rggb.bat canon_eos_5d_mark_iii_05.cr2 c1.rggb

and no .rggb was created. The message was:
D:\TESTSETS\TEST_CAMERA_flif_rotate>IF "RGGBRGGBRGGBRGGB" == "RGGB" (SET Rotate=-t 0 ) ELSE IF "RGGBRGGBRGGBRGGB" == "GRBG" (SET Rotate=-t 270 ) ELSE IF "RGGBRGGBRGGBRGGB" == "BGGR" (SET Rotate=-t 180 ) ELSE IF "RGGBRGGBRGGBRGGB" == "GBRG" (SET Rotate=-t 90 ) ELSE (
ECHO This pattern is unknown: RGGBRGGBRGGBRGGB
EXIT /B 1
)
This pattern is unknown: RGGBRGGBRGGBRGGB

D:\TESTSETS\TEST_CAMERA_flif_rotate>

on next image canon_eos_6d_14 the error message is:
SET PNMTYPE=P6
GOTO GT_DECOMP
) ELSE (
ECHO This raw color is unknown:
EXIT /B 1
) )
This raw color is unknown:

[psykauze]

Please use this dcraw instead:
http://www.centrostudiprogressofotografico.it/en/dcraw/ (rename the exe as dcraw.exe)

Also, could you edit the bat and delete the first "REM" ? (Just the word, not the line)

[StephanBusch]

thank you.. using the provided DCRAW helped but not on the fujifilm_x_e1_20.raf.. the message here is:

D:\TESTSETS\TEST_CAMERA_flif_rotate>IF "GGRGGBGGBGGRBRGRBGGGBGGRGGRGGBRBGBRG" == "RGGB" (SET Rotate=-t 0 ) ELSE IF "GGRGGBGGBGGRBRGRBGGGBGGRGGRGGBRBGBRG" == "GRBG" (SET Rotate=-t 270 ) ELSE IF "GGRGGBGGBGGRBRGRBGGGBGGRGGRGGBRBGBRG" == "BGGR" (SET Rotate=-t 180 ) ELSE IF "GGRGGBGGBGGRBRGRBGGGBGGRGGRGGBRBGBRG" == "GBRG" (SET Rotate=-t 90 ) ELSE (
ECHO This pattern is unknown: GGRGGBGGBGGRBRGRBGGGBGGRGGRGGBRBGBRG
EXIT /B 1
)
This pattern is unknown: GGRGGBGGBGGRBRGRBGGGBGGRGGRGGBRBGBRG

[psykauze]

Yeah. This is not an issue. This file can't be performed as a RGGB file in FLIF. The result will be worse than converting it in PGM mode.

[StephanBusch]

I get 11.098.920 bytes when input is .pgm
and 11.064.698 bytes if input is .rggb

[StephanBusch]

after trying to compress the first canon image, I get an error with FLIF:

=== flif -v -v -v -v -n c1.rggb c1.flif ===
____ []____
[__ | | | ] FLIF 0.1.9 [23 November 2015]
|_ | || | Free Lossless Image Format
]||_[ (c) 2010-2015 J.Sneyers & P.Wuille, GNU GPL v3+

Loading input file: c1.rggb RGGB file should be a PGM, like the output of "dcraw -E -4". Cannot read other types.
Could not read input file: c1.rggb

Execution time: 2.985 s

[psykauze]

I'm sorry but the script work as I expected. This file has a "weird" CFAPattern. FLIF can encode and decode it as a RGGB file without issues but if we try to decode it as a pam (or PNG) the result will be weird.
I prefer the script fails for this kind of file than not be sure of the result.

[StephanBusch]

you mean the fuji file in the above message?

[psykauze]

Yeah.

I will check the canon issue maybe it's just because your flif is not from the last commit.

[jonsneyers]

It isn't, as you can tell from the date. Renaming it to .pnm should do the trick, @StephanBusch

@psykauze: I agree with not pretending that weird CFA patterns are actually RGGB. What do you think is the best approach to handling such files?

[StephanBusch]

if I use .pnm extension, only the PLC transform will be applied and the canon file is treated as 1 channel 16 bit: compression is also worse than it would be with .rggb extension

[jonsneyers]

Yes but if the file is not a PGM (P5) then it doesn't contain Bayered data anyway, but RGB pixels.

[StephanBusch]

I thought that raw2rggb.bat script creates RGGB rather than RGB pixels

[psykauze]

I found out why flif fails to decode the canon rggb file. It's because there is a 'space' caracter after P5 ("P5 \n" instead of "P5\n").

The new bat will not produce this 'space'
raw2rggb.bat.txt

[jonsneyers]

hm, I'm not sure about the PGM spec but if it allows that space to be there, then this is a bug in the rggb/pgm reader...

[StephanBusch]

The problem is still there - FLIF cannot read the input file.

[psykauze]

@jonsneyers The fuji x e1 CFA pattern contain 8 red pixels, 8 blue pixels and 20 green pixels. We can't debayerize this file in 4 planes with the same dimensions.

1. Easiest way: Encoding it only in greyscale mode
2. Not so easy but bitstream changing: Encoding each planes alone (like RGB mode) but with size different plane
3. Harder: Allowing flif to use more than 4 planes. For fuji x e1, 2 blue + 2 red + 5 green = 9 planes. YIQ and MDL transform can be applied.

@StephanBusch It seems the batch file add an extra non-printable character (0x0d) before the end of line. I think this is a flif issue not really a raw2rggb.bat issue.

raw2rggb.bat.txt

[conradgaunt]

hello ,
im not a c coder, im a pascal coder ..so i won't understand flif code easily .. but ill share my limited knowledge re: raw / how ive been dealing with it in my own adventures .. although i only came on here to ask a question ..

first , a bit about me .. last march (2017) i spent a week throwing together a huffman coder , unhappy with 48bit cinema/ real world image compression options .. not sure what to expect...more as a learning exercise, mixed with frustration at being trapped inside 1980's image formats ... after 2 days i had imagemagick pumping out RGB files , and im compressing (easy)...and more importantly ..5 days later ..decompressing! (..got stoned after the success of compressing ....) ..compression results weren't great .. but reasonable .. so then i tried pre-filtering the image with a dumb filter (no logic like paeth etc). at that point a 300tb image archive test showed that change made files 56% smaller than the openj2k encoder managed (on average / overall etc) ..flif was a bit better of course, at about 59% smaller ..but i didn't know that for another 3weeks ..literally (*). i have j2k files that come out smaller than flif btw, just a few. my system never beats flif , but stays quite close. annoyingly , the dumb filter / first attempt/ placeholder filter seems to be the best ive come up with .. which should be good , but it means since ive picked up the code after a year long break (got stoned again) ..ive now spend several weeks wasting my time testing alternatives without results .. but i do have 7 different reversible color transforms now , and they are ALL useful / get used (hint)

during my initial experiments i also made a modified version to handle bayer patterns . i downloaded the blackmagic cinema camera sample files (all 11gig) , .. DCRAW can output a text file with meta data from the file read , i parse that , and get the RGGB order .. but ive only implemented the one BMCC uses out of laziness .
anyway, thats enough about my unfeasibly sussessful attempts at compressing images losslessly

as for raw ..my advice for handing odd bayer patterns is to just decode them to full RGB . DCRAW can output 16bit linear bayer patterns , but odd patterns i just do a full decode , then throw away 2/3 of the values in any bayer configuration i want (or more precisely .. the same one BMCC uses..as ive implemented it already!)..reverse engineering the raw.. HOWEVER, what i intend to do in the future is count the number of original values in the red green and blue channels , and pick the bayer battern that maximizes the number of original values , OR has the widest dynamic range , with the latter trumping the former. as im compressing sequences of large images , ill do this analysis once / or a few times per sequence of images (same when choosing the correct RCT),.. then assume that pattern for all images in the sequence (or test images from the start/ middle / end of sequences and try using any filter already chosen as optimal by one of those etc) , but its pretty quick analysis. you can't cater of every proprietory pattern , unless perhaps you store metadata that can describe colors other than pure Red, Green, or Blue ( ie, by specify each bayer color component as a seperate set of R/G/B values in the metadata, yellow and purple filters can be used and colour adjusted .. but because im lazy , and manufacturers making sensors with purple and yellow filters can just f**k off / are perverts ..). the reality is , the physical colour filters used on the sensor will not be pure red green or blue anyway , and if you throw away the same amount of information that was never captured by the sensor taking the image in the first place , especially after its been fully decoded .. my tests show that reconstruction using simple bi-linear interolation never produces artifacts

ive been focusing on debayering recent;y..wondering if adaptive debayering techniques can cause jitter in moving images. no artifacts appears in decoded redcode footage when simple bi-linear interpolation is used to debayer a "pseudo-reverse engineered raw file",..even after the image is downsampled to quarter resolution .. so im starting to think thats how red debayer too (no doubt because the sensor has a strong OLPF dropping optical resolution/ edge contrast making demosaicing trivial) .
ive been calling these pseudo raw images "documentary format" , as they save a LOT of space compared to the original 48bit TIFFs i generally get to play with . optionally , the pixels can be converted to 16bit half floats (a lossy transformation resulting in even smaller files , but identical dynamic range) .the main reason im trying to re-invent the wheel is because im not prepared to store more pixels than the cameras i use captured at the time, long -term .. not until storage is free

as for compressing DNG files (or as i call them .. dung files ) ., i reencoded the blackmagic sample files , and compressed them using the correct bayer pattern , think i knocked off between 10-20%. i used the bayer reversible colour transform described on the microsoft research site as the RCT option , and i prefiltered the raw files before compressing by simply sweeping right to left across each image row , and subtracting the value 2 places to the left (filtering right to left during encoding, filtering left to right while decoding etc ) .. but then i also filter the filtered results vertically (from bottom to top, again, subtracting the value 2 rows above from the current value) . the results of pre comrpession filtering are signed values btw, bit depth arbitrary

not sure if any of that is useful , but feel free to reach out to discuss my ramblings etc/

so ..i do have a couple of questions myself .. which is why i came on here :

1. is flif ready for prime time yet , or still work in progress ? .. btw, id like you to add more RCT's ;)
2. i couldn't get the flif encoder on the flif site to compress anything 48bit , so im using imagemagick to deal / make FLIF files.. are there any paramters for image magick that anyone knows of , such that i can do a brute force optimization using all available options ..and is it possible to encode bayer patterns with flif (i know it is ... but using imagemagick?)

thanks in advance

k.r

[conradgaunt]

ps, final thought,

many of the RCT's ive used in my compression system were designed by other people, and based around their understanding of how the human eye/ brain works etc .. however , lossless compression has nothing to do with how the eye works, or the brain recieves colour .. its just about reducing bytes ..but a lot of research papers i read start with lossy transforms yuv etc .. then elaborate them into lossless versions .. why!! i think zig zag patterns largely fall into that category . i believe the zig zag pattern is more useful in a lossy context, and is used for perceptual reasons , less useful in a lossless context