Frequent false positives in aletheia.py auto?

[ragibson]

(From openjournals/joss-reviews#5982)

I tried to run an example of the paper/documentation's mention of "the auto command, which performs an exploratory analysis trying to identify the steganalysis technique used."

Is that functionality only meant to be used for images known to contain covert information? If so, that should be clearly listed in the documentation since it does not seem accurate in determining if an image contains a covert message.

On a handful of images (from a phone), a screenshot, and a 512x512 image of blank white square, the analysis seems to indicate all of them have a high likelihood of being steganographed.

$ ./aletheia.py auto example_images/
                     Outguess  Steghide   nsF5  J-UNIWARD *
-----------------------------------------------------------
20231110_3.jpg         [1.0]     0.0     [0.9]    [0.8]  
white_square_512.jpg    0.1      0.1     [0.6]    [0.8]  
20231110_1.jpg         [1.0]    [1.0]    [1.0]    [0.8]  
20231110_2.jpg         [1.0]     0.0     [0.7]    [0.7]  
20231111_4.jpg         [1.0]     0.0     [1.0]    [0.9]  
screenshot_202311...    0.0      0.0     [0.6]     0.2   

* Probability of being stego using the indicated steganographic method.

However, none of these images have been steganographically altered in any way. In fact, white_square_512.jpg is completely white.

[ragibson]

This is discussed more in several of the practical attack articles -- it turns out the models do not generalize particularly well and are only really reliable on images similar to the training set. C.f. the discussions of the dci command in https://daniellerch.me/stego/aletheia/steghide-attack-en/ and https://daniellerch.me/stego/aletheia/f5-attack-en/.

I still believe this should be more clearly called out beforehand when discussing the auto command, especially since it is introduced as a tool for an initial exploratory analysis.

[YassineYousfi]

Yes all the techniques used in this library suffer from cover source mismatch, a disclaimer/warning would be a great addition to the documentation.

[daniellerch]

I have updated the documentation to clarify that Aletheia models may be susceptible to CSM issues. To determine if the models are reliable for the images you're attempting to analyze, using the 'dci' command is essential. Additionally, I've expanded the documentation to include the 'dci' command in the introduction, although it was already mentioned in other sections.

You can find the changes here:
https://daniellerch.me/stego/aletheia/intro-en/#automated-tools

Thank you for your comments.