iOS cli option

[Debusan13]

Adding full support for iOS backups

The Problem

So I noticed in multiple reddit threads people were asking you if this tool could be used on iOS backups.[1] [2]
I also wanted to do this, but the first issue is "where is the chat.db file?" How does one extract the chat info, attachments, etc when the backup has nonsense hashed file names? Building off of this, how can we retain most of the core code while still allowing backups from iOS? It shouldn't be tedious either and should work from any backup location as people might use Finder, iMazing, or an external drive which all store the backup in different locations.

Finding chat.db

From my limited testing and thanks to @nprezant's work 5 years ago, it seems that iOS backups keep their chat.db file in the same location: $PATH_TO_BACKUP/3d/3d0d7e5fb2ce288813306e4d4636395e047a3d28. After finding this I did a test run and found that the messages backed up no problem but there were no attachments

Notice that all the file paths are paths from the iOS system, meaning that all the attachments are stored somewhere in the backup, they are just hidden by the hash.

If all you want is texts from your iPhone run the following

imessage-exporter -n -f html -p $PATH_TO_BACKUP/3d/3d0d7e5fb2ce288813306e4d4636395e047a3d28 -o $EXPORT_DIR

Finding 'contacts.db'

One thing I also found that might prove helpful for issues like #61 is a contacts.db stored at: $PATH_TO_BACKUP/31/31bb7ba8914766d4ba40d6dfb6113c8b614be442

Since the hash is a consistent pattern, these two file locations shouldn't change, but there is no guarantee.

The Hash

How the files are hashed from "original path" -> "hashed path" is as follows:

def get_hashed_path(raw_path):
    domainpath = 'MediaDomain-' + raw_path[2:] # create the domain path that will be hashed
    filename = sha1(domainpath.encode('utf-8')).hexdigest() # create the SHA1 hash (which is the file name)
    dir = filename[:2] # the first two digets of the filename are the directory name
    return Path(dir, filename) # create an file path to the attachment

so for example,

$ python3 get_hashed_path('~/Library/SMS/Attachments/40/00/9D1D2F2B-FFF9-4791-85A3-AC609E584FF4/IMG_4567.jpeg')
PosixPath('c7/c7c61670bfead910aa7c066be22b14e17ef793c7')

^{Thanks again to nprazent}

The SHA1 hash requires the crate sha1 v0.10.5 to be added to the `Cargo.toml'
3aa1620

imessage-exporter/imessage-exporter/Cargo.toml

Line 19 in 3aa1620

sha1 = "0.10.5"

--ios CLI Argument

Since the backup exists in its own little world, and I didn't want to rewrite a lot of the reader and exporter, I thought the best option was to add a --ios flag which requires the -p, --db-path to point to the folder holding the iOS backup. So this would be Library/Application\ Support/MobileSync/Backup/<backup-version>

This is what the help dialogue says for the option

        --ios
            Specify that the database is from an iOS backup
            Using this option requires a custom path to the iPhone backup directory

imessage-exporter/imessage-exporter/src/app/options.rs

Lines 39 to 58 in 592973c

	pub struct Options<'a> {
	/// Path to database file
	pub db_path: PathBuf,
	/// If true, do not copy files from ~/Library to the export
	pub no_copy: bool,
	/// If true, emit diagnostic information to stdout
	pub diagnostic: bool,
	/// The type of file we are exporting data to
	pub export_type: Option<&'a str>,
	/// Where the app will save exported data
	pub export_path: PathBuf,
	/// Query context describing SQL query filters
	pub query_context: QueryContext,
	/// If true, do not include `loading="lazy"` in HTML exports
	pub no_lazy: bool,
	/// Custom name for database owner in output
	pub custom_name: Option<&'a str>,
	/// If true, enable iOS-specific features, db_path is to a backup, uses hashed filepaths
	pub ios: bool,
	}

imessage-exporter/imessage-exporter/src/app/options.rs

Line 71 in 592973c

let ios = args.is_present(OPTION_IOS);

imessage-exporter/imessage-exporter/src/app/options.rs

Lines 286 to 292 in 8b25dc2

	.arg(
	Arg::new(OPTION_IOS)
	.long(OPTION_IOS)
	.help("Specify that the database is from an iOS backup\nUsing this option requires a custom path to the iPhone backup directory")
	.display_order(9)
	.requires(OPTION_DB_PATH)
	)

Validating the path

I found that the best way to validate the path was to check for the aforementioned chat.db.

imessage-exporter/imessage-exporter/src/app/options.rs

Lines 119 to 127 in 592973c

	// Ensure that if iOS is enabled, that the db_path is to a backup
	if ios && user_path.is_some() {
	let db_path = PathBuf::from(user_path.unwrap());
	if !db_path.join(DEFAULT_IOS_CHATDB_PATH).exists() {
	return Err(RuntimeError::InvalidOptions(format!(
	"Option {OPTION_IOS} is enabled, but the database path does not appear to be a valid iOS backup"
	)));
	}
	}

The value for the path is a public constant, and while I was at it I added the contact.db file location as well in case it's used later, even though it is unused code

imessage-exporter/imessage-exporter/src/app/options.rs

Lines 14 to 17 in 592973c

	/// Default location in an iOS backup to find the iMessage database
	pub const DEFAULT_IOS_CHATDB_PATH: &str = "3d/3d0d7e5fb2ce288813306e4d4636395e047a3d28";
	/// Default location in an iOS backup to find the iMessage contacts database
	pub const DEFAULT_IOS_CONTACTSDB_PATH: &str = "31/31bb7ba8914766d4ba40d6dfb6113c8b614be442"; // unused

Establishing the database connection

Since the iOS Backup lives in its own directory, the path of the chat.db needs to be appended to the path to the iOS backup. However, if the iOS option is not enabled, then the path should remain the same.

imessage-exporter/imessage-exporter/src/app/runtime.rs

Lines 168 to 174 in ae0b29b

	pub fn new(options: Options) -> Result<Config, RuntimeError> {
	let conn = if options.ios {
	let ios_db_path = options.db_path.join(DEFAULT_IOS_CHATDB_PATH);
	get_connection(&ios_db_path).map_err(RuntimeError::DatabaseError)?
	} else {
	get_connection(&options.db_path).map_err(RuntimeError::DatabaseError)?
	};

HTML Exporter

Since the HTML Exporter already has resolved_attachment_path to resolve the relative path, simply adding a control flow statement to determine which path is used was all that was necessary. If the ios flag is true, then the path to the attachment from the database will be rehashed to find its location in the backup. This seemed to work perfectly and attachments were appearing in the html files no problems.

imessage-exporter/imessage-exporter/src/exporters/html.rs

Lines 404 to 415 in 034c908

	let resolved_attachment_path =
	// Resolve the relative attachment path to absolute for macOS
	if path.starts_with("~") & !self.config.options.ios {
	path_str.replace('~', &home())
	// Resolve the attachment path for iOS backup
	} else if self.config.options.ios {
	let salt = "MediaDomain-";
	let hash = format!("{:x}", Sha1::digest(format!("{}{}", salt, &path_str[2..]).as_bytes()));
	format!("{}/{}/{}", self.config.options.db_path.display(), &hash[0..2], &hash)
	} else {
	path_str.to_owned()
	};

Note
Using .display() might yield some issues since it "may perform lossy conversion" meaning it won't display non-UTF-8 characters. Not sure how much of a problem this really is.

The exporter properly creates the Attachments folder, where each subdirectory uses the chat_id and each file name is the "original" file name and not the "hashed" one.

TXT Exporter

I thought it fitting I should still update the TXT Exporter using these file paths, as the ones it would display were files that did not exist on the machine.

I made it so the file path actually points to the file's location in the iOS backup, and whats more I formatted it such that it could be easily pasted into the terminal using `cat` or `bat` allowing someone to have the file in its correct format.

Possible improvements

• Path concatenation and creation done in a more "rusty" way, I am not that experienced and I'm not sure if what I did was proper
• --ios arugment location, I put it at the end, but probably having it appear closer to the front would be better
  • making the help section more clear that the -p argument should be used with it
• removing escape slashes on paths when adding directories. macOS has the feature to drag and drop directories into the terminal however doing so includes the escape slash in front of any whitespace \ . I made sure to add that back in for the TXT export, but having the option to shave it off would be nice as I am sure many people will "drop" their iOS backup directory into the terminal and not know they have to remove the escape character for the path to read properly

Thanks again to @nprezant and their repo iMessageBackup
and yes the emojis are non negotiable 😜

[ReagentX]

Really cool work, thank you for the contribution. This is the most detailed pull request I have ever seen!

I will take a closer look soon; I left a small comment in the meantime.

[ReagentX]

Please don't use unwrap() in runtime code; I try to reserve it for tests only.

[Debusan13]

Gotcha, would this be better?

format!("{}/{}/{} > {}", db_path_os_string, &hash[0..2], hash,
        filename.rsplit_once("/").map(|(_, x)| x).unwrap_or_else(|| "Could not get filename")

or we could reflow the logic so it goes ios, macos, error in the match

match &attachment.filename {
  Some(filename) if self.config.options.ios => {
      // iOS rehash the filename to a path in the backup
      let salt = "MediaDomain-";
      let hash = format!("{:x}", Sha1::digest(
          format!("{}{}", salt, &filename[2..]).as_bytes()
      ));
      // attempting to escape spaces in the path for easier copy-paste
      let db_path_os_string = self.config.options.db_path.to_str()
          .map(|s| s.replace(" ", r#"\ "#))
          .unwrap_or_else(|| self.config.options.db_path.display().to_string());
      // <db_path>/<hash[0..2]>/<hash>/<filename> > <filename> allows for copy-paste conversion
      Ok(format!("{}/{}/{} > {}", db_path_os_string, &hash[0..2], hash,
                 filename.rsplit_once("/").map(|(_, x)| x).unwrap_or_else(|| "Could not get filename")))
  }
  // macOS uses the filename as the path
  Some(filename) => Ok(filename.to_string()), 
  None => Err(attachment.filename()),
}

and we could always swap out the "Could not get a filename" with just a plain old error like so

match filename.rsplit_once("/") {
    Some((_, filename)) => Ok(format!("{}/{}/{} > {}", db_path_os_string, &hash[0..2], hash, filename)),
    None => Err(attachment.filename()),
}

or returning it "macOS" style, which is the path from the internal iPhone file system

match filename.rsplit_once("/") {
    Some((_, filename)) => Ok(format!("{}/{}/{} > {}", db_path_os_string, &hash[0..2], hash, filename)),
    None => Ok(filename.to_string()), 
}

again I'm not entirely sure how you want the program to handle it or what syntactic style you prefer

[ReagentX]

1. I think the logic for the iOS parsing should be split out somewhere so that it can be tested separately.

2. I think this whole block will be a lot cleaner once we use an enum instead of a bool here, so I will wait to review it until that changes.

[ReagentX]

If it makes things easier, I can merge this current branch into another feature branch and make all of these fixes myself.

You've done a lot of work here; since I'm going to reorganize some of this anyway I may as well also build these requested fixes. If you just wrap up the enum aspect, I will merge this and clean the rest up in my own way.

[Debusan13]

I fixed up a lot of the little things (the unwraps, removing the unused constant, made the ios error message more clear) and im currently in progress on the enum. Ill push everything that I have by the end of today (~4:00 AM Wednesday (GMT)).

[Debusan13]

Alright, I took a swing at creating the enum and impl for it. Hopefully it makes the reorganizing easier for you. I would love to see how this feature ends up looking in the end as this was my first time writing "production" rust. Thank you for your help and insight, I learned a lot!

[ReagentX]

This is really good work, you should be proud of it!