Problem converting CBF files to SFRM

[RufusTFirefly]

Hi Developers

I have a bunch of CBF files generated by a Dectris Pilatus200K detector and I want to convert them to SFRM
using the fabio library.
I used the tutorial script and changed "edf" (which works file) with "sfrm":

import glob
import fabio, os

files = glob.glob("*.cbf")
files.sort()
print("Number of files: %s" % len(files))

dest_format = "sfrm"
dest_dir = "sfrm_format"

if not os.path.exists(dest_dir):
    os.makedirs(dest_dir)
    
for onefile in files:
    dst_name = os.path.join(dest_dir, os.path.splitext(onefile)[0] + "." + dest_format)
    fabio.open(onefile).convert(dest_format).save(dst_name)

I used the cbf file appended.
When runnning the script I get an error message:

(base) C:\Users\xxxxx\Documents\Python Scripts>python testfabio.py
Number of files: 2
C:\Users\xxxxx\Anaconda3\lib\site-packages\fabio\bruker100image.py:288: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  if numpy.issubdtype(self.data.dtype, float):
Traceback (most recent call last):
  File "testfabio.py", line 23, in <module>
    fabio.open(onefile).convert(dest_format).save(dst_name)
  File "C:\Users\xxxxx\Anaconda3\lib\site-packages\fabio\fabioimage.py", line 676, in save
    self.write(fname)
  File "C:\Users\xxxxx\Anaconda3\lib\site-packages\fabio\bruker100image.py", line 309, in write
    if int(self.header["NOVERFL"].split()[0]) > 0:
KeyError: 'NOVERFL'

It appears to search in the header (which at this point is a CBF header) for a keywork "NOVERFL" that I have never seen in the miniCBF header written by Pilatus detectors.

Best regards

RTF

[RufusTFirefly]

Sorry: I think I forgot the CBF file.
testcbf.zip

[vallsv]

Hi,

The Bruker codec provided by Fabio looks to be designed to change few things from an original Bruker file. So in any way, i would not expect it to work without a lot of fixes.

But I found that trick, no idea if the output is readable.

At least it is not anymore readable with Fabio.

import fabio                                                                                
img = fabio.open("issues/401/testcbf.cbf")                                                  
converted = fabio.bruker100image.Bruker100Image(img.data)                                   
converted.basic_translate()
try:
    # it will update few attrbutes and fail
    converted.save("issues/401/testcbf.sfrm")                                                   
except:
    pass
converted.header["NOVERFL"] = str(str(converted.nunderFlows).ljust(24, ' ') + str(converted.nover_one).ljust(24) + str(converted.nover_two))
# Now it show work
converted.save("issues/401/testcbf.sfrm")                                                   

[RufusTFirefly]

Hi

Thanks for the prompt answer.
The conversion with your trick did work this time but the generated sfrm files (the output of the basic_translate and the output generated at the very end) appear to be unreadable (I only tried to display the converted images with Adxv).
I do not know yet whether there is a problem with the header or with the counts: i will try to figure it out.
Thanks again!

RTF

[kif]

There are 2 bruker formats: the variant 86 and 100. The former creates the NOVERFL automatically as expected.

That said, I did not see that the Bruker format was part of the supported format of Adxv neither:
https://www.scripps.edu/tainer/arvai/adxv.html

It is hence difficult for us to check the file are valid.

[RufusTFirefly]

I checked the generated files again and their format is "FORMAT :86", although the files were generated using bruker100.
Could that be the problem? The inconsistency between the format given in the header (86) and actual format used (100)?

That said, I did not see that the Bruker format was part of the supported format of Adxv neither:
I think the webpage is not up to date. The manual however is recent (March 2019) and it lists the bruker format under "Supported files" and several more formats not shown on the web site.
I did use Adxv to display sfrm files generated using another script converter CBF -> SFRM and it did work without problem (apart some graphical issue when running it under windows with cygwin).
The files (format 100) were correctly converted and displayed by ADXV but part of the meta information
contained in the miniCBF header went missing or it was plain wrong and the script required Qt and some tweaking from me.
That is why I also tried fabio.

[vallsv]

Then try using fabio.brukerimage.BrukerImage(img.data)

[RufusTFirefly]

Unfortunately again the conversion works without errors but the file is unreadable by Adxv.

[kif]

Basically the writer available from FabIO is broken ... The fileformat 86 and 100 are too much entangled.
@SigmundNeher, who is the original author, are you available to fix this code ?
The documentation we have on Bruker file-format is now available online in http://www.silx.org/pub/fabio/
Since I have personally no interest in this format, nor my employer, anybody interested in those formats should take over this work.

[kif]

The issue with your pilatus images comes from their unusual shapes ... The various Bruker format enforce padding for header, datablocks and under/overflow tables. I suspect some strong miss-alignment there.

[kif]

I tried differerent variation and none looks compatible with ADXV (playing with padding at various places, ....).
@RufusTFirefly, could you please get in contact with the ADXV developer and ask him for the piece of code in charge of reading those SFRM files

[RufusTFirefly]

Hi Kif
Thanks for looking into the problem.
I meanwhile started using https://github.com/LennardKrause/pilatus3_frame_conversion for CBF->SFRM conversion and it seems to work fine: ADXV displays the frames without problems. It requires Qt which is a pain for such a tiny project (no command line script) and it needed to be changed in a few places to adapt it to my needs. It also does not manage all fromats that fabio manages but CBF->SFRM is anough for me presently.

RTF

[kif]

Interesting ... they are adding many more headers. I'll give a try to create all those empty headers.

[RufusTFirefly]

The padding of the images with 0 until the images' sizes are powers of 2 like 128(default)/256/512/1024 etc might also be the key.

[kif]

Maybe, I did not find anything like this in Bruker's documentation.
In version 0.11 I added a few header which were missing like "VERSION" and "FORMAT", now adxv recognizes some tags but the image reading still ends in error. The image data do not look to be correct either.
I'll have a look at LennardKrause's code when I find some time.