Rework sync user handling and metadata storage

This introduces the beginning of a split between sync and app services. Object
store Sync types (almost) don't depend on the `app` namespace, and can be used
independently of it. The SyncUser type now implements only the functionality
required internally for sync, and is backed by a UserProvider interface which
allows it to request things like access token refreshes. All user management
has been removed from SyncManager, and it now only owns the sync client and
active sync sessions. SyncSession is mostly unchanged.

`app::User` is the new equivalent of the old SyncUser type. The user management
which used to be in SyncManager is now in App, which implements the
UserProvider interface.

Metadata storage for sync and App has been completely redesigned. The metadata
store is no longer optional, and instead has an in-memory implementation that
should work identically to the persistent store other than not being
persistent. The interface has been reworked to enable atomic updates to the
metadata store rather than relying on the filesystem mutex in SyncManager,
which will be required for multiprocess sync. This required pushing
significantly more logic into the metadata storage, which fortunately turned
out to also simplify things in the process.

The ownership relationship between `App` and `User` has been inverted, with
`App` now holding a weak cache of users and `User` strongly retaining the
`App`. This ensures that a `SyncConfig` now retains everything it depends on
even when app caching is not used.

Co-authored-by: James Stone <james.stone@mongodb.com>

CHANGELOG.md

@@ -9,6 +9,7 @@
 * Audit event scopes containing zero events to save no longer open the audit realm unneccesarily ([PR #7332](https://github.com/realm/realm-core/pull/7332)).
 * Added a method to check if a file needs upgrade. ([#7140](https://github.com/realm/realm-core/issues/7140))
 * Use `clonefile()` when possible in `File::copy()` on Apple platforms for faster copying. ([PR #7341](https://github.com/realm/realm-core/pull/7341)).
+* Introduce the new `SyncUser` interface which can be implemented by SDKs to use sync without the core App Services implementation (or just for greater control over user behavior in tests). ([PR #7300](https://github.com/realm/realm-core/pull/7300).
 
 ### Fixed
 * Fixed queries like `indexed_property == NONE {x}` which mistakenly matched on only x instead of not x. This only applies when an indexed property with equality (==, or IN) matches with `NONE` on a list of one item. If the constant list contained more than one value then it was working correctly. ([realm-js #7862](https://github.com/realm/realm-java/issues/7862), since v12.5.0) 
@@ -20,9 +21,27 @@
 * Fixed a crash with `Assertion failed: m_initiated` during sync session startup ([#7074](https://github.com/realm/realm-core/issues/7074), since v10.0.0).
 * Fixed a TSAN violation where the user thread could race to read `m_finalized` with the sync event loop ([#6844](https://github.com/realm/realm-core/issues/6844), since v13.15.1)
 * Fix a minor race condition when backing up Realm files before a client reset which could have lead to overwriting an existing file. ([PR #7341](https://github.com/realm/realm-core/pull/7341)).
+* SyncUser::all_sessions() included sessions in every state *except* for waiting for access token, which was weirdly inconsistent. It now includes all sessions. ([PR #7300](https://github.com/realm/realm-core/pull/7300).
+* App::all_users() included logged out users only if they were logged out while the App instance existed. It now always includes all logged out users. ([PR #7300](https://github.com/realm/realm-core/pull/7300).
+* Deleting the active user left the active user unset rather than selecting another logged-in user as the active user like logging out and removing users did.
 
 ### Breaking changes
 * SyncManager no longer supports reconfiguring after calling reset_for_testing(). SyncManager::configure() has been folded into the constructor, and reset_for_testing() has been renamed to tear_down_for_testing(). ([PR #7351](https://github.com/realm/realm-core/pull/7351))
+* The following things have been renamed or moved as part of moving all of the App Services functionality to the app namespace:
+  - SyncUser -> app::User. Note that there is a new, different type named SyncUser.
+  - SyncUser::identity -> app::User::user_id. The "identity" word was overloaded to mean two unrelated things, and one has been changed to user_id everywhere.
+  - SyncUserSubscriptionToken -> app::UserSubscriptionToken
+  - SyncUserProfile -> app::UserProfile
+  - App::Config -> AppConfig
+  - SyncConfig::MetadataMode -> AppConfig::MetadataMode
+  - MetadataMode::NoMetadata -> MetadataMode::InMemory
+  - SyncUser::session_for_on_disk_path() -> SyncManager::get_existing_session()
+  - SyncUser::all_sessions() -> SyncManager::get_all_sessions_for()
+  ([PR #7300](https://github.com/realm/realm-core/pull/7300).
+* Some fields have moved from SyncClientConfig to AppConfig. AppConfig now has a SyncClientConfig field rather than it being passed separately to App::get_app(). ([PR #7300](https://github.com/realm/realm-core/pull/7300).
+* Sync user management has been removed from SyncManager. This functionality was already additionally available on App. ([PR #7300](https://github.com/realm/realm-core/pull/7300).
+* AuditConfig now has a base_file_path field which must be set by the SDK rather than inheriting it from the SyncManager.
+
 
 ### Compatibility
 * Fileformat: Generates files with format v23. Reads and automatically upgrade from fileformat v5.
@@ -37,6 +56,9 @@
 * The minimum CMake version has changed from 3.15 to 3.22.1. ([#6537](https://github.com/realm/realm-core/issues/6537))
 * Update Catch2 to v3.5.2 ([PR #7297](https://github.com/realm/realm-core/pull/7297)).
 * The unused `partition` and `user_local_uuid()` fields have been removed from `FileActionMetadata`. ([PR #7341](https://github.com/realm/realm-core/pull/7341)).
+* App metadaa storage has been entirely rewritten in preparation for supporting sharing metadata realms between processes.
+* The metadata disabled mode has been replaced with an in-memory metadata mode which performs similarly and doesn't work weirdly differently from the normal mode.
+* The ownership relationship between App and User has changed. User now strongly retains App and App has a weak cache of Users.
 
 ----------------------------------------------
 

bindgen/spec.yml

@@ -221,7 +221,7 @@ enums:
       - FUNCTION
       - API_KEY
   MetadataMode:
-    cppName: SyncClientConfig::MetadataMode
+    cppName: app::AppConfig::MetadataMode
     values:
       - NoEncryption
       - Encryption
@@ -275,13 +275,18 @@ enums:
       - DeleteRealm
       - ClientReset
       - ClientResetNoRecovery
+  SyncFileAction:
+    cppName: SyncFileAction
+    values:
+      - DeleteRealm
+      - BackUpThenDeleteRealm
   ProgressDirection:
     cppName: SyncSession::ProgressDirection
     values:
       - upload
       - download
   SyncUserState:
-    cppName: SyncUser::State
+    cppName: UserState
     values:
       - LoggedOut
       - LoggedIn
@@ -434,7 +439,7 @@ records:
         default: false
 
   UserIdentity:
-    cppName: SyncUserIdentity
+    cppName: app::UserIdentity
     fields:
       id:
         type: std::string
@@ -557,11 +562,6 @@ records:
 
   SyncClientConfig:
     fields:
-      base_file_path: std::string
-      metadata_mode:
-        type: MetadataMode
-        default: MetadataMode::Encryption
-      custom_encryption_key: std::optional<EncryptionKey>
       logger_factory: Nullable<LoggerFactory>
       log_level:
         type: LoggerLevel
@@ -620,7 +620,7 @@ records:
       body: std::string
 
   DeviceInfo:
-    cppName: app::App::Config::DeviceInfo
+    cppName: app::AppConfig::DeviceInfo
     fields:
       platform_version: std::string
       sdk_version: std::string
@@ -632,13 +632,19 @@ records:
       bundle_id: std::string
 
   AppConfig:
-    cppName: app::App::Config
+    cppName: app::AppConfig
     fields:
       app_id: std::string
       transport: SharedGenericNetworkTransport
       base_url: std::optional<std::string>
       default_request_timeout_ms: std::optional<uint64_t>
       device_info: DeviceInfo
+      base_file_path: std::string
+      sync_client_config: SyncClientConfig
+      metadata_mode:
+        type: MetadataMode
+        default: MetadataMode::Encryption
+      custom_encryption_key: std::optional<EncryptionKey>
 
   CompensatingWriteErrorInfo:
    cppName: sync::CompensatingWriteErrorInfo
@@ -1132,36 +1138,49 @@ classes:
       provider: AuthProvider
       provider_as_string: std::string
 
-  SyncUserSubscriptionToken:
-    cppName: SyncUser::Token
+  UserSubscriptionToken:
+    cppName: app::User::Token
 
   SyncUser:
     sharedPtrWrapped: SharedSyncUser
     properties:
-      all_sessions: std::vector<SharedSyncSession>
       is_logged_in: bool
-      identity: const std::string&
-      provider_type: const std::string&
-      local_identity: const std::string&
+      user_id: std::string
+      app_id: std::string
+      legacy_identities: std::vector<std::string>
       access_token: std::string
       refresh_token: std::string
+      state: SyncUserState
+      sync_manager: SharedSyncManager
+    methods:
+      access_token_refresh_required: bool
+      request_log_out: '(cb: AsyncCallback<(err: std::optional<AppError>)>&&)'
+      request_refresh_user: '(cb: AsyncCallback<(err: std::optional<AppError>)>&&)'
+      request_refresh_location: '(cb: AsyncCallback<(err: std::optional<AppError>)>&&)'
+      request_access_token: '(cb: AsyncCallback<(err: std::optional<AppError>)>&&)'
+      track_realm: '(std::string_view)'
+      create_file_action: '(action: SyncFileAction, original_path: std::string_view, requested_recovery_dir: std::optional<std::string>, partition_value: std::string_view) -> std::string'
+
+  User:
+    base: SyncUser
+    cppName: app::User
+    sharedPtrWrapped: SharedUser
+    properties:
+      is_anonymous: bool
       device_id: std::string
       has_device_id: bool
       user_profile: UserProfile
       identities: std::vector<UserIdentity>
       custom_data: std::optional<bson::BsonDocument>
-      sync_manager: SharedSyncManager
-      state: SyncUserState
       subscribers_count: count_t
+      app: SharedApp
     methods:
       log_out: ()
-      session_for_on_disk_path: '(path: StringData) -> Nullable<SharedSyncSession>'
-      subscribe: '(observer: (user: IgnoreArgument<const SyncUser&>)) -> SyncUserSubscriptionToken'
-      unsubscribe: '(token: SyncUserSubscriptionToken)'
-      # TODO update methods?
+      subscribe: '(observer: (user: IgnoreArgument<const app::User&>)) -> UserSubscriptionToken'
+      unsubscribe: '(token: UserSubscriptionToken)'
 
   UserProfile:
-    cppName: SyncUserProfile
+    cppName: app::UserProfile
     methods:
       name: '() -> std::optional<std::string>'
       email: '() -> std::optional<std::string>'
@@ -1182,27 +1201,27 @@ classes:
     sharedPtrWrapped: SharedApp
     properties:
       config: const AppConfig&
-      current_user: Nullable<SharedSyncUser>
-      all_users: std::vector<SharedSyncUser>
+      current_user: Nullable<SharedUser>
+      all_users: std::vector<SharedUser>
       sync_manager: SharedSyncManager
       subscribers_count: count_t
     staticMethods:
-      get_app: '(mode: AppCacheMode, config: AppConfig, sync_client_config: SyncClientConfig) -> SharedApp'
+      get_app: '(mode: AppCacheMode, config: const AppConfig&) -> SharedApp'
       get_cached_app: '(app_id: const std::string&) -> SharedApp'
       clear_cached_apps: ()
       close_all_sync_sessions: ()
 
     methods:
-      log_in_with_credentials: '(credentials: AppCredentials, cb: AsyncCallback<(user: const Nullable<SharedSyncUser>&, err: std::optional<AppError>)>&&)'
+      log_in_with_credentials: '(credentials: AppCredentials, cb: AsyncCallback<(user: const Nullable<SharedUser>&, err: std::optional<AppError>)>&&)'
       log_out:
         - '(cb: AsyncCallback<(err: std::optional<AppError>)>&&)'
-        - sig: '(user: SharedSyncUser, cb: AsyncCallback<(err: std::optional<AppError>)>&&)'
+        - sig: '(user: SharedUser, cb: AsyncCallback<(err: std::optional<AppError>)>&&)'
           suffix: user
-      refresh_custom_data: '(user: SharedSyncUser, cb: AsyncCallback<(err: std::optional<AppError>)>&&)'
-      link_user: '(user: SharedSyncUser, credentials: const AppCredentials&, cb: AsyncCallback<(user: const Nullable<SharedSyncUser>&, err: std::optional<AppError>)>&&)'
-      switch_user: '(user: SharedSyncUser)'
-      remove_user: '(user: SharedSyncUser, cb: AsyncCallback<(err: std::optional<AppError>)>&&)'
-      delete_user: '(user: SharedSyncUser, cb: AsyncCallback<(err: std::optional<AppError>)>&&)'
+      refresh_custom_data: '(user: SharedUser, cb: AsyncCallback<(err: std::optional<AppError>)>&&)'
+      link_user: '(user: SharedUser, credentials: const AppCredentials&, cb: AsyncCallback<(user: const Nullable<SharedUser>&, err: std::optional<AppError>)>&&)'
+      switch_user: '(user: SharedUser)'
+      remove_user: '(user: SharedUser, cb: AsyncCallback<(err: std::optional<AppError>)>&&)'
+      delete_user: '(user: SharedUser, cb: AsyncCallback<(err: std::optional<AppError>)>&&)'
       usernamePasswordProviderClient:
         - sig: () -> UsernamePasswordProviderClient
           cppName: provider_client<app::App::UsernamePasswordProviderClient>
@@ -1212,8 +1231,8 @@ classes:
       push_notification_client: '(service_name: const std::string&) -> PushClient'
       subscribe: '(observer: (app: IgnoreArgument<const App&>)) -> AppSubscriptionToken'
       unsubscribe: '(token: AppSubscriptionToken)'
-      call_function: '(user: const SharedSyncUser&, name: std::string, args: EJsonArray, service_name: std::optional<std::string>, cb: AsyncCallback<(result: Nullable<const EJson*>, err: std::optional<AppError>)>)'
-      make_streaming_request: '(user: SharedSyncUser, name: std::string, args: bson::BsonArray, service_name: std::optional<std::string>) -> Request'
+      call_function: '(user: const SharedUser&, name: std::string, args: EJsonArray, service_name: std::optional<std::string>, cb: AsyncCallback<(result: Nullable<const EJson*>, err: std::optional<AppError>)>)'
+      make_streaming_request: '(user: SharedUser, name: std::string, args: bson::BsonArray, service_name: std::optional<std::string>) -> Request'
       update_base_url: '(base_url: std::optional<std::string>, cb: AsyncCallback<(err: std::optional<AppError>)>&&)'
       get_base_url: '() const -> std::string'
 
@@ -1230,8 +1249,8 @@ classes:
   PushClient:
     cppName: app::PushClient
     methods:
-      register_device: '(registration_token: const std::string&, sync_user: const SharedSyncUser&, completion: AsyncCallback<(err: std::optional<AppError>)>&&)'
-      deregister_device: '(sync_user: const SharedSyncUser&, completion: AsyncCallback<(err: std::optional<AppError>)>&&)'
+      register_device: '(registration_token: const std::string&, sync_user: const SharedUser&, completion: AsyncCallback<(err: std::optional<AppError>)>&&)'
+      deregister_device: '(sync_user: const SharedUser&, completion: AsyncCallback<(err: std::optional<AppError>)>&&)'
 
   UsernamePasswordProviderClient:
     cppName: app::App::UsernamePasswordProviderClient
@@ -1247,12 +1266,12 @@ classes:
   UserAPIKeyProviderClient:
     cppName: app::App::UserAPIKeyProviderClient
     methods:
-      create_api_key: '(name: const std::string&, user: SharedSyncUser, completion: AsyncCallback<(apiKey: UserAPIKey&&, err: std::optional<AppError>)>&&)'
-      fetch_api_key: '(id: ObjectId&, user: const SharedSyncUser, completion: AsyncCallback<(apiKey: UserAPIKey&&, err: std::optional<AppError>)>&&)'
-      fetch_api_keys: '(user: const SharedSyncUser, completion: AsyncCallback<(apiKeys: std::vector<UserAPIKey>&&, err: std::optional<AppError>)>&&)'
-      delete_api_key: '(id: ObjectId&, user: const SharedSyncUser, completion: AsyncCallback<(err: std::optional<AppError>)>&&)'
-      enable_api_key: '(id: ObjectId&, user: const SharedSyncUser, completion: AsyncCallback<(err: std::optional<AppError>)>&&)'
-      disable_api_key: '(id: ObjectId&, user: const SharedSyncUser, completion: AsyncCallback<(err: std::optional<AppError>)>&&)'
+      create_api_key: '(name: const std::string&, user: SharedUser, completion: AsyncCallback<(apiKey: UserAPIKey&&, err: std::optional<AppError>)>&&)'
+      fetch_api_key: '(id: ObjectId&, user: const SharedUser, completion: AsyncCallback<(apiKey: UserAPIKey&&, err: std::optional<AppError>)>&&)'
+      fetch_api_keys: '(user: const SharedUser, completion: AsyncCallback<(apiKeys: std::vector<UserAPIKey>&&, err: std::optional<AppError>)>&&)'
+      delete_api_key: '(id: ObjectId&, user: const SharedUser, completion: AsyncCallback<(err: std::optional<AppError>)>&&)'
+      enable_api_key: '(id: ObjectId&, user: const SharedUser, completion: AsyncCallback<(err: std::optional<AppError>)>&&)'
+      disable_api_key: '(id: ObjectId&, user: const SharedUser, completion: AsyncCallback<(err: std::optional<AppError>)>&&)'
 
   # See Helpers::make_loger_factory to construct one.
   # Using an opaque class here rather than exposing the factory to avoid having to
@@ -1269,16 +1288,15 @@ classes:
       log_level: LoggerLevel
       has_existing_sessions: bool
     methods:
-      immediately_run_file_actions: '(original_name: std::string) -> bool'
       set_session_multiplexing: '(allowed: bool)'
       set_log_level: '(level: LoggerLevel)'
       set_logger_factory: '(factory: LoggerFactory)'
       set_user_agent: '(user_agent: std::string)'
       set_timeouts: '(timeouts: SyncClientTimeouts)'
       reconnect: ()
       wait_for_sessions_to_terminate: ()
-      path_for_realm: '(config: SyncConfig, custom_file_name: std::optional<std::string>) -> StringData'
       get_existing_active_session: '(path: const std::string&) -> SharedSyncSession'
+      get_all_sessions_for: '(user: const SyncUser&) -> std::vector<SharedSyncSession>'
 
   ThreadSafeReference: {}
   AsyncOpenTask: