Development Blog

[steilerDev]

In short, this project started, because I was looking for a reliable solution to keep a backup of my iCloud Photos library (full motivation here) and I never thought to get to a working solution nor was I expecting other people to be interested in a project like this.

However as interest in this project has grown and given this is passion project of mine, I want to transparently provide updates on the work that I am doing.

Especially since this is (currently) maintained solely by myself in my free time, progress might sometimes appear slow and sometimes actually be slow. I'm therefore going to try and keep this discussion thread up to date with the work I'm putting into the project.

I'm hoping this will build trust in the community about my commitment to the tool, while also helping me to keep track of my thoughts, when there will be inevitable times of inactivity (because of life).

I am not sure if this will be of interest for anyone but me, but this project has already proven me wrong once. I therefore hope you'll enjoy this - and please provide me with feedback in case I can make this valuable to you!

[steilerDev]

Let's address the elephant in the room: The main branch has not seen a commit since the last 1.1.0 release in March (which means for nearly 5 months).

The primary reason for this, has been a massive rework of the underlying CI/CD process. This was done on a separate repository as fiddling around with GH actions and npm/docker releases would have messed up this project. I started this work early April and after a busy personal schedule was able to finalise it in June/July (this is also when the first nightly and beta releases appeared.

This is setting this project up for long-term maintainability and is fully automating the build, test and release process, as every push now leads to a build (sometimes test) and release run of the code. This is achieved by implementing Conventional Commits in combination with semantic release.

Following this, I started thinking about contribution and feedback processes a lot and created assets to support and streamline these processes. All of this is documented in a Contribution Guideline and reflected in various templates. Unfortunately none if this is 'live' yet - as I have not been happy to publicly release the next version of the tool (and therefore could not push to main yet). All of this should (hopefully very) soon look quite different.

[steilerDev]

As this project is leaving its infant status, there are currently a lot of architectural changes going on. Unfortunately none of them lead to very exciting user facing changes short term, but should help out the project in the long run.

This includes:

• A central ResourceManager who manages access to shared resources as well as a flexible resource file. This currently only contains the current trust token (moving away from a dedicated file only containing this token) and a library version tag. As requests about different library structures come up, I think that an update to the library structure will be necessary in the future - this should prepare us early to enable smooth transitions in the future.
• The ResourceManager also takes responsibility of a global validator, using JSON Schema to make sure iCloud requests contain the expected data - this is WIP and already removed a lot of if(var === undefined) statements. However there will also be benefits in the iCloudPhotosLibrary model that will lead to bigger code changes/cleanups. This implementation will be delayed until after the next release.
• Additionally the ResourceManager provides access to a global EventBus (instead of every class extending EventEmitter) - I have never thought about this and this was actually a proposal from GitHub Co-Pilot (that I recently started using and kind of like) - this removed so much weird code that was required to couple event emitters with event consumers.
• Finally I also moved networking into the ResourceManager - the whole tool will (once again) use a single axios object to handle all traffic. This also allows me to better handle cookies and I really hope this will allow me to fix Your Apple ID was used to sign in to iCloud via a web browser. #292 - because all of those emails really annoy me as well! This is the work item I'm engaged with right now: I'm building a HeaderJar class to store and dynamically inject cookies and header values while also trying to capture the network traffic produced by the application into a HAR file. This should allow me to debug some issues related to the iCloud API (as this does not seem to be the same for everyone).
• Unfortunately, my testing also needs to be expanded - those new classes need some love and there is also the iCloudPhotosLibrary class that I have never gotten around writing proper tests for.

Overall a lot going on - and really wanting to push out the public release so main is finally updated to show all the previously mentioned process changes. However given the amount of change that are going on, I should be careful and have an expanded beta period, to have people test the solution.

My plan is to finalise the NetworkManager now to release a bigger beta release for testing, while completing tests (and also implementing an experimental region option to hopefully also support users from China)

[steilerDev]

I think I've removed all dangling references, that were introduced due to the central event hub. I've introduced an event registry to keep track of listeners held by any class to easily remove them once a scheduled run has been completed.

The har tracker and header jar core functionality should be implemented now as well.

As part of that I am removing some restrictions around when to retry a failed sync. Locking at the unhandled errors reported so far, I believe this is fair. Also on a retry, I will not reset the iCloud Session, but only perform a 'stupid' resync. I'm hoping this will make initial syncs or large libraries more stable.

I will try to resync my own library to ensure this is working, but I'll need to rely on more heavy iCloud Photos users to provide feedback here.

[steilerDev]

The header jar and network tracker have been implemented and it seems to work. Additionally the work around ensuring no memory leaks happen because of old object being referenced in event listeners has been concluded, by introducing an EventRegistry.

The header jar might fix the annoying login warning emails.

All of this has been released with v1.2.0-nightly.4 - let's see if any of it works while if fix all the broken tests...

If this works well 'in production' for a few days, I will try and re-sync my full library to check that this also works well. Then, the only thing missing for now would be the region flag.

[steilerDev]

Nightly run showed, that the HAR tracker does not seem to be compatible with responseType: stream - now using a separate axios instance for fetching assets. I actually thought about in the past, because I noticed, that assets are available without any form of authentication - because it seems asset URLs are generated when querying for assets and are only valid for ~30 minutes (this is also why the app fails on long sync runs - those URLs expire)

Let's see if nightly.5 works better - I'll probably won't make much progress this weekend, but at least collecting some sync statistics would be nice (but for that it needs to work first :D )

[steilerDev]

The good news is, that it seems nightly.5 is no longer sending out email warnings when re-authenticating.

The current NetworkManager instance is re-using the aasp cookie, as well as the X-Apple-ID-Session-Id header and the scnt header. I believe the last two are not contributing to this (so I will reset them on a new session to validate this).

The UNKNOWN_FILETYPE_DESCRIPTOR error is now also no longer fatally failing the sync execution. This was enabled by the global event bus.

[steilerDev]

One of the things that is really annoying is the library lock :D Unfortunately it is very much necessary, however I'm already thinking how I can enable releasing the lock gracefully upon an error. I'm thinking about moving the locking logic into the ResourceManager and couple it with some other - potentially long running tasks - that need to be cleaned. E.g. network & disk access.

This could be a future extension of the network manager, where I move queue handling & throttling into there.

[steilerDev]

I believe the memory leak situation is finally under control - I'm going into a feature freeze now, finalise the unit tests and then target to release 1.2.0 by early next week (if I'm not running into any more trouble)

[steilerDev]

Memory leakage still not under control - app grows in size of about 15MB for every sync run on my personal account.

[steilerDev]

After some further tweaking, it seems memory is finally under control:

The tool seems to finally settles around 70Mb between scheduled runs for my personal library. I'm going to expand monitoring a little bit further, but I should now be able to finally focus on getting the release ready.

[steilerDev]

I've had a thought about the future implementation of Album syncing and how to make this more reliable.

I've went down a rabbit hole how rsync is implemented and tree edit distances. Only to come to the conclusion that this is all way too much for this project.

I've discovered a recursive diff implementation that should fit this use case well.

I'm now thinking about an asset level archiving feature. However this would need a full rewrite of everything.

As I move towards incorporating this algorithm and re-implementing the asset parsing, I should keep this idea in mind.

[steilerDev]

I've been busy making some quality of life improvements to the tool - this includes summarized warnings, as well as a new crash reporting system.

As part of that I'm trying to understand (and hopefully improve) reliability of the overall sync process.

It seems, that after 60 minutes the asset URL expire (which requires a re-setup of iCloud Photos) and after 8 hours the session token seems to expire (so we would need to sign in again). In this case, the setup request errors with 421 - I should try and see if handling this could solve the timeout.

Additionally feedback on MFA NOT PROVIDED needs to be improved.

All of this is now on the agenda for the final release, together with a review of the docs and comments.

[steilerDev]

Refreshing of session id has been implemented - will only take 8 hours to validate - let's see if we can get a nice full library sync of my personal library.

[steilerDev]

The sync refreshed the session and continued the process - looking good so far!

[steilerDev]

As I'm currently primarily focusing on cleaning things up and making sure the docs are up to date, I've been coming back to the idea of increasing the level of automation for acquiring a new trust token for the API tests.

The MFA code is already received via a cellular modem attached to one of my machines and processed by the OS - which is why it should not be too hard to automatically forward it to the GH action needing it.

I've been thinking the following approach:

1. If the token check fails - go into token acquisition mode
2. Make the GH runner accessible from the internet - I'm currently using uterm for this, but I think creating Wireguard, or something like wireproxy should be more easy - all I need to do is send one http requests to the machine - alternatively we could publish the code on an event bus that the runner subscribes to, however this feels like more work
3. Start icps and go into MFA flow
4. Request resend via email (it always defaults to device, but I can only receive codes for the test account via sms)
5. Enter MFA code by POSTing to the MFA server
6. Write new trust token to GH secret via GH CLI

A problem for another day - first get a release out :)

[steilerDev]

I've cleaned up the docs and even though test coverage is not there yet (I'm still lacking the NetworkManger class) I really want to get this out of the door this weekend - so here goes nothing....
Let's see how the release workflow breaks ;)

[steilerDev]

I was just talking about reworking the trust acquisition workflow - and just now the trust token is required and it seems upterm is not available - will have to wait another day.

[steilerDev]

upterm.dev stopped working at the worst possible time, as the lack of trust token is currently blocking the release :(

I could not get the workflow working, because wireguard does not seem to be able to connect to the runner - maybe a port issue? I'll be on vacation for the next two weeks - this will therefore delay everything. For now the latest nightly release is basically the next full release - at least this way it will be battle proof, by running unattended the next two weeks.

[steilerDev]

This was significantly more difficult to get working than expected - but finally I was able to connect ot the GH actions runner through wireguard updating the trust token. This finally unblocks deployments!

[steilerDev]

I've addressed some last minute issues that popped up while I've let the tool run for 2 weeks with constant new pictures flowing in - I'm very happy with the results and have now triggered the new release workflow for the first time. Fingers crossed....

[steilerDev]

Deployment went nearly flawless - unfortunately due to its size, I ran into GH's API limit - need to manually clean up some tags, issues and PRs, but overall very happy - onto the next version!