[MOB-19274][MOB-15354] Use Disk for Storage Services #966

cdhoffmann · 2023-08-31T19:29:23Z

Phase 1 of migration from UserDefaults as our storage mechanism. Next, we will be working on migrating existing data from UserDefaults to this new Disk storage.

codecov · 2023-08-31T21:51:58Z

Codecov Report

Merging #966 (8ea84f6) into migrateOffUserDefaults (eb4406a) will decrease coverage by 0.21%.
Report is 13 commits behind head on migrateOffUserDefaults.
The diff coverage is 88.89%.

❗ Current head 8ea84f6 differs from pull request most recent head fef2c3a. Consider uploading reports for the commit fef2c3a to get more accurate results

@@                    Coverage Diff                     @@
##           migrateOffUserDefaults     #966      +/-   ##
==========================================================
- Coverage                   90.22%   90.01%   -0.21%     
==========================================================
  Files                         130      131       +1     
  Lines                        8119     8221     +102     
==========================================================
+ Hits                         7325     7400      +75     
- Misses                        794      821      +27

…ss for migration but remove post migration

TestApps/FileSystemNamedCollection.swift

…o block comments

TestApps/FileSystemNamedCollection.swift

AEPLifecycle/Tests/FunctionalTests/LifecycleFunctionalTests.swift

AEPServices/Sources/ServiceProvider.swift

…ng when clearing something which doesn't exist.

AEPServices/Sources/storage/FileSystemNamedCollection.swift

pfransenadb

Getting there -- quite a few questions/comments on FileSystemNamedCollection and NamedCollectionDataStore

pfransenadb · 2023-09-05T21:54:40Z

AEPServices/Sources/storage/FileSystemNamedCollection.swift

+
+    func setAppGroup(_ appGroup: String?) {
+        self.appGroup = appGroup
+        if let appGroup = appGroup {


Should this function also handle setting appGroup to nil (and clearing the associated appGroupUrl?

also -- why not use get/set semantics on appGroup variable?

Regardless of how it's done, setting/getting of appGroup needs to be done on queue as well.

I agree that the get/set semantics would be cleaner, but the get/set function are part of the NamedCollectionProcessing protocol so we would be breaking compatibility by changing it. Or did you mean keeping the setAppGroup and getAppGroup functions and using the computed variable internally to handle the logic?

Didn't consider that there was a pre-existing protocol -- makes sense, leave as is.

pfransenadb · 2023-09-05T22:09:00Z

AEPServices/Sources/storage/FileSystemNamedCollection.swift

+        guard let fileUrl = fileUrl(for: collectionName) else {
+            return
+        }
+        queue.sync {


should be async, this function doesn't need to return anything.

pfransenadb · 2023-09-05T22:09:16Z

AEPServices/Sources/storage/FileSystemNamedCollection.swift

+    }
+
+    func remove(collectionName: String, key: String) {
+        queue.sync {


can also be async

pfransenadb · 2023-09-05T22:19:04Z

AEPServices/Sources/storage/FileSystemNamedCollection.swift

+    }
+
+    func set(collectionName: String, key: String, value: Any?) {
+        guard let fileUrl = fileUrl(for: collectionName) else {


fileUrl needs to be called from within queue.sync to maintain ordering.

pfransenadb · 2023-09-05T22:21:53Z

AEPServices/Sources/storage/FileSystemNamedCollection.swift

+    private var appGroupUrl: URL?
+    private let fileManager = FileManager.default
+    private let LOG_TAG = "FileSystemNamedCollection"
+    var appGroup: String?


should be private?

pfransenadb · 2023-09-05T22:23:12Z

AEPServices/Sources/storage/FileSystemNamedCollection.swift

+import Foundation
+
+class FileSystemNamedCollection: NamedCollectionProcessing {
+    private let queue = DispatchQueue(label: "FileSystemNamedCollection.barrierQueue")


nit: isn't really a barrier, just a FIFO queue.

pfransenadb · 2023-09-05T22:25:31Z

AEPServices/Sources/storage/NamedCollectionDataStore.swift

-        set(key: key, value: encodedValue)
+        guard let encodedValue = try? encoder.encode(value) else {return}
+        let encodedString = String(data: encodedValue, encoding: .utf8)
+        set(key: key, value: encodedString)


in previous code a failure to encode resulted in a nil value set (removal) -- is the change (failing to set and bailing due to the new guard statement) intentional, or was the previous behavior preferred/depended on?

This is a good point. Not too sure if we were depending on this to be nil in failure scenarios. I think it's a matter of preference. I would prefer to simply return, as setting nil on a value when encoding fails feels odd to me. What do you think @praveek? Should we remove the guard and set nil when encoding fails?

I too prefer simply returning instead of setting nil value. However, I would like to maintain the same behavior as before since I'm uncertain whether other extensions depend on the previous behavior and want to keep all the changes for this migration only within Core.

…n tests. etc.

praveek · 2023-09-06T22:46:18Z

AEPServices/Sources/storage/FileSystemNamedCollection.swift

+            do {
+                try fileManager.createDirectory(at: adobeBaseUrl, withIntermediateDirectories: true)
+            } catch {
+                Log.error(label: adobeDirectory, "Failed to create storage directory with error: \(error)")


Can you check if directory exists and exit before attempting to create a directory?

Apparently Apple doesn't recommend that approach. https://developer.apple.com/documentation/foundation/filemanager/1410277-fileexists

If you see the note towards the bottom which says: "Attempting to predicate behavior based on the current state of the file system or a particular file on the file system is not recommended. Doing so can cause odd behavior or race conditions. It’s far better to attempt an operation (such as loading a file or creating a directory), check for errors, and handle those errors gracefully than it is to try to figure out ahead of time whether the operation will succeed."

I have tested this function, and if the directory exists, it doesn't do anything. So in our case it works just fine.

AEPServices/Sources/storage/FileSystemNamedCollection.swift

cdhoffmann added 5 commits August 24, 2023 15:09

Get objects working somewhat.

3248441

Finish functional tests for data store

11551f6

fix more broken tests, add utility for clearing new storage

d240fd4

tests passing

c1fd15a

add threading tests

8fe157a

cdhoffmann requested a review from praveek August 31, 2023 19:29

cdhoffmann added 6 commits August 31, 2023 13:35

swift format

2e8b8c8

clear storage files in setup

878b394

clear storage files

b570e5a

reset service in functional test

f485416

make file writing sync instead of async

9484426

Protect against traversal in base url for storage service

2ab72e8

cdhoffmann added 4 commits September 1, 2023 10:17

put write in do catch for error handling

aa39b2c

make filesystem class vars private

15a7731

remove extra private function, clean up a little

f046a86

rename userdefault test to filesystem test, will keep userdefault cla…

3e6fbd3

…ss for migration but remove post migration

praveek reviewed Sep 1, 2023

View reviewed changes

TestApps/FileSystemNamedCollection.swift Outdated Show resolved Hide resolved

cdhoffmann added 2 commits September 1, 2023 11:43

Update IDETemplateMacros.plist

0732dae

move file from test app directory, no idea how it got there. Update t…

1a431ea

…o block comments

praveek reviewed Sep 1, 2023

View reviewed changes

TestApps/FileSystemNamedCollection.swift Outdated Show resolved Hide resolved

praveek reviewed Sep 1, 2023

View reviewed changes

AEPLifecycle/Tests/FunctionalTests/LifecycleFunctionalTests.swift Outdated Show resolved Hide resolved

praveek reviewed Sep 1, 2023

View reviewed changes

AEPServices/Sources/ServiceProvider.swift Outdated Show resolved Hide resolved

cdhoffmann added 2 commits September 1, 2023 12:41

combine utility

95c178e

test failing intermittently because cache is already clear and throwi…

b31ce16

…ng when clearing something which doesn't exist.

praveek reviewed Sep 1, 2023

View reviewed changes

AEPServices/Sources/storage/FileSystemNamedCollection.swift Outdated Show resolved Hide resolved

cdhoffmann added 4 commits September 1, 2023 13:35

make set logic simpler

6076d00

Add performance tests

8cfbdc0

add setAppGroup for testing, and add some docs

361391b

swift format

4ca4e57

cdhoffmann requested a review from pfransenadb September 5, 2023 17:50

pfransenadb requested changes Sep 5, 2023

View reviewed changes

make writing async, fix breaking tests by resetting kv service betwee…

94c78bc

…n tests. etc.

cdhoffmann requested review from pfransenadb and praveek September 6, 2023 20:31

cdhoffmann added 2 commits September 6, 2023 14:32

swift format

8ea84f6

rename queue

fef2c3a

pfransenadb approved these changes Sep 6, 2023

View reviewed changes

praveek reviewed Sep 6, 2023

View reviewed changes

AEPServices/Sources/storage/FileSystemNamedCollection.swift Show resolved Hide resolved

change error to warnings, and include more logs

1546b6f

praveek approved these changes Sep 7, 2023

View reviewed changes

cdhoffmann merged commit 909434e into adobe:migrateOffUserDefaults Sep 7, 2023
14 of 16 checks passed

This was referenced Oct 16, 2023

Add attribute for fileprotection #971

Merged

Migrate off UserDefaults for local storage #972

Merged

timkimadobe mentioned this pull request Feb 6, 2024

Update cache clearing helpers adobe/aepsdk-testutils-ios#27

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MOB-19274][MOB-15354] Use Disk for Storage Services #966

[MOB-19274][MOB-15354] Use Disk for Storage Services #966

cdhoffmann commented Aug 31, 2023

codecov bot commented Aug 31, 2023 •

edited

pfransenadb left a comment

pfransenadb Sep 5, 2023

pfransenadb Sep 5, 2023

pfransenadb Sep 5, 2023

cdhoffmann Sep 6, 2023

pfransenadb Sep 6, 2023

pfransenadb Sep 5, 2023

pfransenadb Sep 5, 2023

pfransenadb Sep 5, 2023

pfransenadb Sep 5, 2023

pfransenadb Sep 5, 2023

pfransenadb Sep 5, 2023

cdhoffmann Sep 6, 2023 •

edited

praveek Sep 6, 2023

praveek Sep 6, 2023

cdhoffmann Sep 7, 2023

[MOB-19274][MOB-15354] Use Disk for Storage Services #966

[MOB-19274][MOB-15354] Use Disk for Storage Services #966

Conversation

cdhoffmann commented Aug 31, 2023

codecov bot commented Aug 31, 2023 • edited

Codecov Report

pfransenadb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cdhoffmann Sep 6, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Aug 31, 2023 •

edited

cdhoffmann Sep 6, 2023 •

edited