syncservice: implementation and migration

[tburgin]

Santa's syncing architecture is currently based around a user centric command line application (santactl sync). Over the years we have daemon-ized portions of santactl to support features like push notifications and bundle voting. This works but added significant complexity to santactl. This pull request attempts to right this past wrong by splitting out all the syncing code into a separate service.

The syncservice is a new binary/process that is launched on demand as a MachService. If a sync server is configured, syncservice will start up and perform periodic syncs, handle push notifications and handle event and bundle uploads from santad. Some highlights:

• The syncing code have been moved from santactl to syncservice. The bulk of that work was completed in santasyncservice: move sync code to the santasyncservice dir #696 and is now finished in this pull.
• syncservice is started by launchd, as opposed to the previous approach of santad execing santactl sync.
• santactl sync now communicates directly with the syncservice to sync.
• All syncing across the system now takes place in the syncservice. The syncservice ensures concurrent syncs do not happen.

Fixes #653

[pmarkowsky]

Still looking through this.

Annotated High level flow through the syncservice after this PR is.

sequenceDiagram
   santactl ->> santasyncservice: syncWithLogListener
   Note over santactl,santasyncservice: Service is spun up by launchd if needed
   alt is clean sync
      santasyncservice ->> santad: "setSyncCleanRequired:YES"
   else 
   end
   santasyncservice ->> syncserver: preflight
   santasyncservice --> santactl: "didReceiveLog (messages for preflight)"
   syncserver -->> santasyncservice: preflight response
   santasyncservice --> santactl: "didReceiveLog (messages for preflight response)"
   santasyncservice ->> syncserver: eventUpload
   syncserver -->> santasyncservice: eventUploadResponse (200)
   santasyncservice ->> syncserver: ruleDownload
   santasyncservice --> santactl: "didReceiveLog (messages for ruleDownload )"
   syncserver --> santasyncservice: ruleDownload response (200)
   santasyncservice ->> santad: databaseRuleAddRules
   santasyncservice ->> santad: setRuleSyncLastSuccess
   santasyncservice ->> santad: postRuleSyncNotificationWithCustomMessage
   santasyncservice --> santactl: "didReceiveLog (messages for ruleDownload response)"
   santasyncservice ->> syncserver: postFlight request
   santasyncservice -->> santactl: "didReceiveLog (messages for postFlight)"
   syncserver -->> santasyncservice: postFlight response
   santasyncservice -->> santactl: "didReceiveLog (messages for postFlight response)"
   santasyncservice -->> santactl: "didReceiveLog (final messages)"
   

Loading

[russellhancox]

Still looking through this.

Annotated High level flow through the syncservice after this PR is.

santad is triggering the sync service to start in SNTApplication (if a sync server URL is set), this is needed to have push notifications to work and (if I'm not mistaken) perform syncs on interval when needed.

[tburgin]

PTAL

Thanks @mlw for the comment about dispatching syncs onto a queue. That simplified things a lot.

Some additional refactoring was needed due to the serialization. Two new classes SNTPushNotifications and SNTPushNotificationsTracker were created to isolate the push notifications logic as much as possible. It looks like there is a lot of new code, but it is mainly a forklift.

There are two non-blocking TODOs. Copying them here for visibility.

From Source/santasyncservice/SNTSyncLogging.h

// TODO(bur): SLOGD() is temporarily set to LOG_LEVEL_INFO. Once santactl sync supports the
// --debug flag, move this back to LOG_LEVEL_DEBUG. These debug logs are helpful when
// troubleshooting sync issues with users, so let's opt to always log them for now.

From Source/santasyncservice/SNTPushNotificationsTracker.m

// TODO(bur): Add better guaranties for displaying notifications. This will involve checking the
// rules.db to see if the rule associated with the notification has been applied.