Questions about processing WhatsApp backups found into Apple or Google Cloud Warrant Returns

[dafeol]

Boa tarde!
Estou com dados de duas nuvem, uma Google e outra da Apple. Minha dúvida é a respeito da leitura feita pelo IPED dos backups do Whatsapp. É necessário alguma configuração específica antes de realizar a indexação dos arquivos baixados destas nuvens? Ou o IPED por padrão já consegue ler estes backups e montar os diálogos da mesma forma que o CELLEBRITE?

Abaixo tenho por exemplos os arquivos que foram gerados após descriptografar e descompactar a nuvem do usuário, porém após a indexação, o IPED não me trás os diálogos:

Abaixo outro exemplo, mas de uma nuvem da Google:

Desde já agradeço pela atenção!

[lfcnassif]

The WhatsApp database (ChatStorage.sqlite) in the iCloud backup seems encrypted (enc suffix). You would need the encryption key to decrypt it. Anyway, even with the encryption key, currently IPED doesn't support decrypting that database, it needs the database already decrypted to extract the chats, calls, etc.

[dafeol]

Há alguma solução que substitua o Cellebrite para descriptografar esse arquivo antes de fazer a indexação?
Outra dúvida... tendo em vista q o Celebritte consegue realizar esse processo, a chave para descriptografar vem junto aos arquivos baixados da nuvem?

[lfcnassif]

I have never had access to a cloud data warrant return, so unfortunately I can't give you a precise answer. My guess is that the WhatsApp encryption key is not in the cloud (at least in plain form), otherwise that would be a security flaw. I guess you would need access to the physical mobile phone, from where you may be able to extract the encryption key with a FFS extraction. Or maybe the Cellebrite software is able to do a password attack on the encryption key, not sure... Maybe other users can help you better than me on this subject, sorry.

[dafeol]

Entendi! Muito obrigado pela atenção

[Jefresongil]

EM alguns processos em que tive a oportunidade de trabalhar foram recebidos dados contidos em Nuvem de Google e/ou Apple, e haviam conteúdos trazidos em texto claro. Li em uma postagem da Whatsapp LLC, que em outubro/22 teria sido implementado uma senha em sua configuração de forma que o usuário poderia subir seu backup com senha. Então, pelas consultas que realizei, a chave Key seguia junto com o backup realizado na nuvem. Posso estar enganado. Mas entendo que era assim.

[rafael844]

Hi Nassif.
This .enc in icloudrive use to be encrypted, as you said, and even cellebrite cant decrypt. But sometimes ChatStorage.sqlite is in the backup folder, and Cellebrite only needs the FileInfoList.txt that comes with the package.

Inside the FileInfoList.txt it has, as example:

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>relativePath</key>
	<string>ChatStorage.sqlite</string>
	<key>birth</key>
	<string>6834530376009</string>
	<key>groupID</key>
	<string>503451</string>
	<key>mode</key>
	<string>33134588</string>
	<key>modified</key>
	<string>6833453687758</string>
	<key>statusChanged</key>
	<string>683654587758</string>
	<key>userID</key>
	<string>5401</string>
	<key>size</key>
	<string>3915666451</string>
	<key>sizeBeforeCopy</key>
	<string>10956666800</string>
</dict>
</plist>","contents":{"signature":"BCWDLkmXBgE2tM2f34356P4wLD","owner":"20728237465487862341436","size":3915455776,"reference_signature":"AXxfQ168SNeu8H9Yfm0","wrapping_key":"/lNnTOZhCul6kdvcyQdnhKqqIUWX4=","pcsInfo":"7BdgfhfdsHDSKJHLSH3P2Z3+SSl7gutBa5BKMrfJAHSJbJJmbYTVNZicMpXWWwA=="},"___createdBy":{"id":"_2bc76f56cd27498e3456596e8993azazec","type":"WEAK"},"___modifiedBy":{"id":"_2bc76f56cd274678698e96e8993azazec","type":"WEAK"},"_db_generationNum":0,"fileType":0,"___modTime":9780000},"created":{"timestamp":1661995411171,"user":"_2bc76f5698e96e32377a768993ezazac","device":"1"},"modified":{"timestamp":978000,"user":"_2b76f56cd27498e96e32377a768993ec","device":"1"},"plugin_fields":{},"conflictLosersEtags":[]}][{"id":"F:08534FA4-97E0-45B1-AFE0-E6AA57018777:N0Hbft3OLzKTFuguaWkzZpJKqxQ=:7436:bhPK","recordType":"File","tombstone":false,"etag":"0","fields":{"___etag":"0","extension":"sqlite","protectionClass":3,"___recordID":{"id":"F:08345534FA4-934E0-0-E3453465477:N0Hbftfdg3OWkzZpJKqxQ=:7436:bhPK","type":"WEAK"},"version":1,"___createTime":1661995411170,"encryptedAttributes":"<?xml version="1.0" encoding="UTF-8"?>

IPED can process all files, .opus,. jpg, etc... inside the backup folder and those files are in this FileInfoList.txt file, but ChatStorage.sqlite it doesnt process. So we have to process first in PA and then in IPED.

[lfcnassif]

If ChatStorage.sqlite is unencrypted and not being decoded properly, regardless of its name (we detect it based on signature and table names, not on file name) please provide us with a sample not being decoded, so we can investigate.

[wladimirleite]

Or is ChatStorage.sqlite encrypted and Physical Analyzer uses the key in FileInfoList.txt to decrypt it?

[rafael844]

Thanks, I'll try to get a sample for testing.

[lfcnassif]

Have you succeed to get a sample iCloud warrant return, or just the ChatStorage.sqlite, that reproduces the issue/lack of support to share with us for testing?

Or is ChatStorage.sqlite encrypted and Physical Analyzer uses the key in FileInfoList.txt to decrypt it?

Anyone can confirm if the encryption key is really into that file and where?

Or maybe the cause of IPED not decoding unencrypted ChatStorage.sqlite is related to this ticket #1294 I forgot.

If ChatStorage.sqlite is compressed using LZFSE, IPED is not able to decode it today. Again, I would need samples to investigate and test a future implementation.

[rafael844]

Have you succeed to get a sample iCloud warrant return, or just the ChatStorage.sqlite, that reproduces the issue/lack of support to share with us for testing?

Not yet, we asked for it and our superiors are trying to get the aproval to share.

Anyone can confirm if the encryption key is really into that file and where?

The only thing I can say is that without that file Cellebrite cant show ChatStorage.sqlite

[lfcnassif]

Cellebrite software needs the original filenames (or paths) to work. That's why it needs that file, at least.

[tlragazzan]

A título de informação, se não for fornecido o arquivo FileInfoList.txt o Cellebrite não consegue decodificar o ChatStorage

[lfcnassif]

IPED doesn't need that file. If it is not working, we need a ChatStorage.sqlite sample not being decoded to investigate.

[tlragazzan]

FileInfoList.txt contains various information about Apple backup files, including file names.
All files are provided with encoded names, and this relationship is contained in FileInfoList.txt.

Example
File name: F_24118AE9-51F2-4B69-A478-CD54D181F5EC_1VsNrVJaX7qCl2EmPDJ8f2vF6pk=_6075083_PJNp
Name FileInfoList.txt : ChatStorage.sqlite

Specifically ChatStorage.sqlite, and other Whatsapp files and other databases are encoded in LFZSE. Sometimes, in addition to LZFSE, they are also encoded in LZIP.

I'm working on a solution, adopting the strategy of collecting all filenames from FileInfoList.txt, decoding the ones that need it and exporting it to a ZIP.

[lfcnassif]

As I said before, IPED WhatsApp decoding doesn't need original filename or path to work. If LZFSE or LZIP encoding are used, I need samples to implement and test a future support for them. The few WhatsApp samples from the cloud I have are not LZFSE encoded.

[lfcnassif]

Got 01 sample iCloud data with a few LZFSE files from André (PC/MG), thank you! Decompressed them using RagingMoose library, referenced on #1294, then processed the resulting data with IPED. WhatsApp conversations were decoded fine! Attachments were also linked properly, with no file renaming/mapping from FileInfoList.txt, that's not needed by our WhatsApp parser, as I said before. Interestingly, most files in the sample set are not encoded, just a few of them are LZFSE compressed, including ChatStorage.sqlite and other sqlite databases, as @tlragazzan said.

I should have a working draft in the next days. But for a final implementation, I would need more samples to properly test the feature, for example, with LZIP compression. @tlragazzan, do you mean some files are LZFSE compressed and others LZIP compressed? Or some files really are LZIP compressed after LZFSE compression, or the opposite? This seems strange to me... If you or anyone else could provide more samples, that would be very useful to finish the feature.

[lfcnassif]

For those that can help testing, here is a snapshot with a working draft support:
https://github.com/sepinf-inc/IPED/actions/runs/5843809106

There is still a limitation, to be addressed in the next days, described here:
#1812

But I think it should work for most purposes.

[lfcnassif]

I still need LZIP compressed samples to try to handle them properly, I don't have any.

[tlragazzan]

Some files are LZIP compressed after LZFSE.

[lfcnassif]

Could you share some samples privatelly with me? My email is public in my profile.

[rafael844]

For those that can help testing, there is already a snapshot with a working draft here:
https://github.com/sepinf-inc/IPED/actions/runs/5843809106

We tested it and can confirm that the snapshot is working and bringing the whatsapp conversations, including linking media files.

[lfcnassif]

What's the full path of the above selected file in the case processed using the snapshot version? In the 4.1.2 case, it was into a TAR archive, having just the modified date. In the 4.2-snapshot case, I can't see the full path in the picture, but it has all 3 file system dates, so I guess it is not into an archive, like TAR, this would explain the difference, the embeddedRelationShipId property is present in files extracted from containers. It is important to process the exact same input and with the same processing options when comparing different versions.

[lfcnassif]

If you processed with ignoreDuplicates = true that can also lead to this kind of differences: first case took the file from a container and the second case could have picked up a duplicate from outside the container.

[dafeol]

Assim que der aqui, tentarei refazer as indexações, utilizando os mesmo parâmetros de configuração, incluindo a ativação do SHA-256, como o colega @tc-wleite observou também.

[dafeol]

Mas uma dúvida... para a indexação dessas nuvens então o ideal é não mudar a configuração "ignoreDuplicates" ? Deixar ela com o valor default (false) ?

[lfcnassif]

I don't recommend enabling ignoreDuplicates for most cases as explained above that option in IPEDConfig.txt. Actually I thought to remove it because it is inherently unsafe, see:
#931

But ended keeping it because it might be useful when you have to process dozens of backups from the same user/machine, which is not the case here.

[lfcnassif]

Hello all,

Just finished my planned enhancement to better detect WhatsApp iOS account info, to be independent of a specific plist file name. Now the owner account should be detected fine and will be shown in all chat balloons to the right in WA conversations, and also displayed fine in the graph analysis. Snapshot will be ready here in 10 minutes:
edited: https://github.com/sepinf-inc/IPED/actions/runs/5871612960

Please give it a try. After some of you test it and report results are fine, I'll integrate the new feature in main version.

[lfcnassif]

Could you please send a screenshot of your SHA-256 configuration? The right one is below. About searching for a WhatsApp number, it is not the direct way to find shared files, you should click on the returned Chat with that number, then all shared files should be listed into the References tab, but that will work only if the sha-256 configuration in correct into conf/HashTaskConfig.txt, should be lowercase:

About EMBEDDEDRELATIONSHIPID, it was indexed in your 4.1.2 case because the input data was compressed. I asked for the full path of mp4/opus files in the new case to help you diagnose, and I'm still waiting for the info...

[lfcnassif]

So, see file paths are different, in first case the file is inside a TAR archive, in second not. That explains the absence of EMBEDDEDRELATIONSHIPID property in the second case, as I said before. If you search for path:XXXXXX where XXXXX is your number, those files should be returned in the second case.

And please click on "Propriedades Avançadas" in the Metadata viewer and check if sha-256 was computed. Maybe you used another processing profile with a wrong configuration.

[rafael844]

We tested here and its working as expected. Contact name is showing correctly. Thank you @lfcnassif .

[lfcnassif]

Thanks for testing!

[lfcnassif]

I merged the feature yesterday and I plan to release the new version today.

[dafeol]

Obs: Apaguei a postagem anterior pq faltou ocultar alguns dados

[lfcnassif]

Hum... sha-256 was computed properly. If attachments are not being linked to chats in your case, sorry, I'm out of ideas, I would need you to send your data privately to me, so I can try to reproduce and investigate, it doesn't happen to me, neither to @rafael844, neither to @tlragazzan.

[dafeol]

Eu verifiquei e realmente as indexações tinham aquelas diferenças observadas (indexei de origens diferentes, onde uma tinha o arquivos descompactados e a outra não). Também consegui localizar a aba específica, que você apontou como a ideal para identificar os arquivos trocados entre determinados interlocutores do chat (Aba Referências). Porém neste último ponto ainda ficou uma dúvida, pq além dos arquivos que eu encontro nesta aba "Referencias", ainda encontro arquivos que, ao menos da minha leitura, aparenta ter sido trocada entre os mesmo interlocutores (Não sei se por não constar diretamente na pasta de backup, mas sim na pasta iclouddrive por exemplo).
Daí meu questionamento é: Além dos arquivos encontrados nesta guia, os que eu encontrar diretamente em por exemplo "Multimídia > Imagens", eu poderei afirmar que foram trocados entre os dois interlocutores, mesmo não presentes lá no chat? A seguinte interpretação está correta:

Quanto ao pedido dos dados para sua análise, como seria feito? Pergunto pq os arquivos que tenho em mãos ainda estão em fase de análise. Não tenho nenhum outro de investigação já concluída disponível no momento.

[lfcnassif]

Files shown in References tab are files the tool detected were sent or received for sure, they are into the chat body. Maybe the files you found are related to deleted messages, not available in WhatsApp database anymore. Maybe user created those files and put into the WhatsApp folder using a similar name pattern. Anyway, the tool just links attachments still referenced by the WhatsApp database.