feat(apple): Handle network changes reliably on macOS and iOS #4133

jamilbk · 2024-03-13T21:30:39Z

Tried to organize this PR into commits so that it's a bit easier to review.

Involves simplifying the logic in Adapter.swift so that us mortals can maintain it confidently:

The .stoppingTunnel, .stoppedTunnelTemporarily, and .stoppingTunnelTemporarily states have been removed.
I also removed the self. prefix from local vars when it's not necessary to use it, to be more consistent.
onTunnelReady and getSystemDefaultResolvers has been removed, and onUpdateRoutes wired up, along with cleanup necessary to support that.

Involves adding the reconnect and set_dns stubs in the FFI and fixing the log filter so that we can log them (see chore(apple): Remove connlib_client_apple log filters because they have no effect #4182 )
Involves getting the path update handler working well on macOS using SystemConfiguration to read DNS servers.
Involves getting the path update handler working well on iOS by employing careful trickery to prevent path update cycles by detecting if path.gateways has changed, and avoid setting new DNS if it hasn't.

Refs #4028
Fixes #4297
Fixes #3565
Fixes #3429
Fixes #4175
Fixes #4176
Fixes #4309

vercel · 2024-03-13T21:30:45Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
firezone	⬜️ Ignored (Inspect)	Visit Preview		Mar 27, 2024 2:48am

github-actions · 2024-03-13T21:32:33Z

Terraform Cloud Plan Output

Plan: 9 to add, 8 to change, 9 to destroy.

Terraform Cloud Plan

github-actions · 2024-03-13T21:41:20Z

Performance Test Results

TCP

Test Name	Received/s	Sent/s	Retransmits
direct-tcp-client2server	225.5 MiB (+3%)	227.0 MiB (+3%)	149 (-43%)
direct-tcp-server2client	226.9 MiB (+1%)	228.2 MiB (+1%)	185 (-17%)
relayed-tcp-client2server	147.8 MiB (+4%)	148.3 MiB (+4%)	127 (-2%)
relayed-tcp-server2client	157.8 MiB (+5%)	158.3 MiB (+5%)	194 (+7%)

UDP

Test Name	Total/s	Jitter	Lost
direct-udp-client2server	50.0 MiB (-0%)	0.05ms (+6%)	0.00% (NaN%)
direct-udp-server2client	50.0 MiB (-0%)	0.01ms (+3%)	0.00% (NaN%)
relayed-udp-client2server	50.0 MiB (+0%)	0.11ms (+8%)	0.00% (NaN%)
relayed-udp-server2client	50.0 MiB (+0%)	0.06ms (-11%)	0.00% (NaN%)

jamilbk · 2024-03-17T14:44:22Z

@conectado @ReactorScram @thomaseizinger Just want to leave this here as an example of how noisy the path update handler can be.

This was all printed from a single state change from ["Wifi On", "Hotspot On", "Ethernet Off"] to ["WiFi On", "Hotspot On", "Ethernet On"] (i.e. I connected an ethernet cable which became the default interface):

tunnel	07:41:32.652889-0700	didReceivePathUpdate(path:): path.availableInterfaces: [en12, en0, en9, utun12]	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:32.653188-0700	didReceivePathUpdate(path:): Detected network change: Online.	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:32.655292-0700	getSystemDefaultResolvers(): ["2600:1700:3ecb:2410::1", "192.168.1.254"]	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
connlib	07:41:32.765691-0700	Unknown resource	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:32.940891-0700	didReceivePathUpdate(path:): path.availableInterfaces: [en12, en0, en9, utun12]	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:32.941060-0700	didReceivePathUpdate(path:): Detected network change: Online.	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:32.943069-0700	getSystemDefaultResolvers(): ["2600:1700:3ecb:2410::1", "192.168.1.254"]	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:33.072648-0700	didReceivePathUpdate(path:): path.availableInterfaces: [en12, en0, en9, utun12]	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:33.073002-0700	didReceivePathUpdate(path:): Detected network change: Online.	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:33.074440-0700	getSystemDefaultResolvers(): ["2600:1700:3ecb:2410::1", "192.168.1.254"]	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:33.202852-0700	didReceivePathUpdate(path:): path.availableInterfaces: [en12, en0, en9, utun12]	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:33.203032-0700	didReceivePathUpdate(path:): Detected network change: Online.	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:33.205651-0700	getSystemDefaultResolvers(): ["2600:1700:3ecb:2410::1", "192.168.1.254"]	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:33.396532-0700	didReceivePathUpdate(path:): path.availableInterfaces: [en12, en0, en9, utun12]	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:33.396701-0700	didReceivePathUpdate(path:): Detected network change: Online.	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:33.402652-0700	getSystemDefaultResolvers(): ["2600:1700:3ecb:2410::1", "192.168.1.254"]	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:33.561598-0700	didReceivePathUpdate(path:): path.availableInterfaces: [en12, en0, en9, utun12]	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:33.561783-0700	didReceivePathUpdate(path:): Detected network change: Online.	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:33.563857-0700	getSystemDefaultResolvers(): ["2600:1700:3ecb:2410::1", "192.168.1.254"]	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:33.723485-0700	didReceivePathUpdate(path:): path.availableInterfaces: [en12, en0, en9, utun12]	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:33.723659-0700	didReceivePathUpdate(path:): Detected network change: Online.	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:33.725301-0700	getSystemDefaultResolvers(): ["2600:1700:3ecb:2410::1", "192.168.1.254"]	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
connlib	07:41:33.770669-0700	Unknown resource	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:33.897114-0700	didReceivePathUpdate(path:): path.availableInterfaces: [en12, en0, en9, utun12]	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:33.897685-0700	didReceivePathUpdate(path:): Detected network change: Online.	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:33.899137-0700	getSystemDefaultResolvers(): ["2600:1700:3ecb:2410::1", "192.168.1.254"]	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:34.049184-0700	didReceivePathUpdate(path:): path.availableInterfaces: [en12, en0, en9, utun12]	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:34.049403-0700	didReceivePathUpdate(path:): Detected network change: Online.	FirezoneNetworkExtensionmacOS	dev.firezone.firezone
tunnel	07:41:34.050539-0700	getSystemDefaultResolvers(): ["2600:1700:3ecb:2410::1", "192.168.1.254"]	FirezoneNetworkExtensionmacOS	dev.firezone.firezone

In the above log, en12 is my ethernet, en0 WiFi, en9 hotspot, and utun12 our tunnel.

jamilbk · 2024-03-17T20:51:50Z

swift/apple/FirezoneNetworkExtension/Adapter.swift

+    // sentinel, which isn't helpful to us.
+    private func resetToSystemDNSGettingBindResolvers() -> [String] {
+      var resolvers: [String] = []
+      let semaphore = DispatchSemaphore(value: 0)


Each path update callback spawns a thread (workQueue.async), so we're ok to block here with a semaphore.

Actually, I noticed this sometimes blocked the main UI thread on iOS, so I enclosed this inside a Task in the calling function.

thomaseizinger · 2024-03-17T21:37:29Z

@conectado @ReactorScram @thomaseizinger Just want to leave this here as an example of how noisy the path update handler can be

Thanks, that is really helpful!

Should connlib wait 500ms after receiving an update before it acts on it? I.e. a new update coming in would reset the timer to 0.

That appears it would end up only apply a single one of these then.

jamilbk · 2024-03-18T13:56:17Z

Should connlib wait 500ms

Yeah I think 500ms is a reasonable debounce timeout to start with.

…ve no effect (#4182) Discovered that tracing doesn't respect the `connlib_client_apple` log filter, which makes it a bit confusing when logging debug in the Apple FFI module. Removing to prevent further casualties. Extracted out of #4133

thomaseizinger

Great work! I don't fully understand all the swift stuff but I have one question regarding on_tunnel_ready.

swift/apple/FirezoneNetworkExtension/CallbackHandler.swift

rust/connlib/clients/apple/src/lib.rs

thomaseizinger · 2024-03-26T01:42:14Z

@jamilbk The PR description might need some adjusting now that more features / fixes have been packed into it. If possible, it would also be nice to perhaps split a few of these refactorings off into their separate PR. From what I've followed it, the bulk of changes that "fix" things seem to be in 72aed61 (#4133). Would it make sense to first merge that and then make more PRs with the remaining fixes on top?

jamilbk · 2024-03-26T03:38:48Z

@thomaseizinger Yeah I can stop adding to this. Until today main was broken for the apple clients, and xcode crashes when you change branches which makes it difficult to switch between them easily. The compilation cache needs to be reset on each branch switch too.

It's really a pita.

thomaseizinger · 2024-03-26T04:24:47Z

@thomaseizinger Yeah I can stop adding to this. Until today main was broken for the apple clients, and xcode crashes when you change branches which makes it difficult to switch between them easily. The compilation cache needs to be reset on each branch switch too.

It's really a pita.

Don't worry if it is too difficult :)

Perhaps now that we have a working version, splitting it apart isn't too much work?

jamilbk · 2024-03-26T05:02:23Z

Perhaps now that we have a working version, splitting it apart isn't too much work?

Unfortunately the history is quite clobbered in this PR. There were a lot of style changes as well, and multiple changes to the same pieces of code for different issues :-/.

I can rebase and at least the last few commits could be gone through on their own. I'll do that.

jamilbk · 2024-03-26T05:03:30Z

Going forward, I don't expect any more near-full-rewrites, but I wasn't confident that I could compartmentalize small changes to go into main. It kinda turned into a ball of yarn...

jamilbk · 2024-03-26T05:05:32Z

Last thought: we at least need unit tests in the clients. I reckon with a combo of:

unit tests
connlib smoke tests in CI with a stubbed out TUN or similar
a bit of manual smoke testing

We could get pretty far without having to build a massive e2e brittle test harness.

jamilbk · 2024-03-26T05:20:37Z

@conectado @ReactorScram Ready for final review. Apologies for the size... this one was quite a doozie.

ReactorScram

I left a lot of questions relating to trying to learn Swift.

This was hard to review. I see a lot of stuff changed names, some diffs look like just formatting changes too. So I may have overlooked semantic changes, in addition to the expected amount of not understanding it all yet.

I'd rather have formatting and renaming separated from semantic changes in general.

The control flow also seems confusing to me, but maybe that's something we could clean up after GA. Maybe the flow in the Tauri Client only makes sense to me because I wrote it.

ReactorScram · 2024-03-26T16:21:10Z

swift/apple/FirezoneNetworkExtension/NetworkSettings.swift

+  // Constants
+  private weak var packetTunnelProvider: NEPacketTunnelProvider?
+  private(set) var logger: AppLogger


These are not constant in the sense of Rust's const, but they're objects that won't be replaced during the lifetime of this NetworkSettings object?

Correct, that's probably not a good comment. I'll fix it.

swift/apple/README.md

ReactorScram · 2024-03-26T16:24:53Z

rust/connlib/clients/apple/src/lib.rs

+        fn reconnect(&mut self);
+        #[swift_bridge(swift_name = "setDns")]
+        fn set_dns(&mut self, dns_servers: String);


I'll leave some redundant questions just to make sure I understand how the Apple Client fits together.

So this part is exposing the Rust functions to Swift, we're declaring these two new functions. This is still on the Rust side, so I assume it would error if the signature is wrong.

Yeah! The macros in here are used to generate the bindings.

ReactorScram · 2024-03-26T16:30:53Z

swift/apple/Firezone/Application/FirezoneApp.swift

-        }
-      }
-    }
+    public var appStore: AppStore?


"store" here means some persistent variable storage, it's not related to the Apple App Store package manager, right?

Yeah, I was confused by that too at first. That's correct.

It could probably use a rename.

ReactorScram · 2024-03-26T16:36:33Z

swift/apple/FirezoneKit/Sources/FirezoneKit/Features/AskPermissionView.swift

-      .filter { $0.isInitialized }
-      .sink { [weak self] tunnelAuthStatus in
+    tunnelStore.$status
+      .sink { [weak self] status in
        guard let self = self else { return }


This is equivalent to the let-else we use in Rust? The else is required to diverge, and it doesn't do anything special, it's just a more idiomatic way to do an early return?

That's correct. Guards are pretty heavily used in Swift and they turn Optional(obj) into just obj so you don't need the optional modifier in the remainder of the block body.

ReactorScram · 2024-03-26T18:32:28Z

swift/apple/FirezoneNetworkExtension/Adapter.swift

+
+      // Tell the system the tunnel is up, moving the tunnelManager status to
+      // `connected`.
+      completionHandler(nil)


Completion handlers are async callbacks, like the Promises in JS?

ReactorScram · 2024-03-26T18:34:08Z

swift/apple/FirezoneNetworkExtension/Adapter.swift

    workQueue.async { [weak self] in
      guard let self = self else { return }

-      if referenceVersionString == self.displayableResources.versionString {
+      if hash == Data(SHA256.hash(data: Data((resourceListJSON ?? "").utf8))) {


Why wouldn't string equality work here?

ReactorScram · 2024-03-26T18:35:57Z

swift/apple/FirezoneNetworkExtension/Adapter.swift

+      }
+
+      if shouldFetchSystemResolvers(path: path) {
+        // Spawn a new thread to avoid blocking the UI on iOS


No chance that thread / task can leak?

swift/apple/FirezoneNetworkExtension/Adapter.swift

ReactorScram · 2024-03-26T18:40:23Z

swift/apple/FirezoneKit/Sources/FirezoneKit/Helpers/AppLogger.swift

-  private let target: Target
-  private let folderURL: URL
-  private let logger: Logger
-
  // All log writes happen in the workQueue
  private let workQueue: DispatchQueue


Does this do backpressure if the queue gets too long?

jamilbk · 2024-03-27T02:06:49Z

swift/apple/FirezoneNetworkExtension/NetworkSettings.swift

+  // Constants
+  private weak var packetTunnelProvider: NEPacketTunnelProvider?
+  private(set) var logger: AppLogger


Correct, that's probably not a good comment. I'll fix it.

jamilbk · 2024-03-27T02:07:28Z

swift/apple/FirezoneNetworkExtension/NetworkSettings.swift

+  public var tunnelAddressIPv4: String?
+  public var tunnelAddressIPv6: String?
+  public var dnsAddresses: [String] = []


Yeah, Apple's API expects them as Strings, so there's no point going further than that. They're sent from connlib and assumed to be valid.

jamilbk · 2024-03-27T02:13:32Z

swift/apple/FirezoneNetworkExtension/PacketTunnelProvider.swift

+        // Remove the passwordReference from our configuration so that it's not used again
+        // if the app is re-launched. There's no good way to send data like this from the
+        // Network Extension to the GUI, so save it to a file for the GUI to read later.
+        try String(reason.rawValue).write(to: SharedAccess.providerStopReasonURL, atomically: true, encoding: .utf8)


Not really, the sequence of events is:

Tunnel process writes reason to file

Tunnel process exits

GUI receives NEVPNStatusDidChange

GUI reads the file and acts accordingly, deleting it

Hm, it could, but this file is solely for (at this point) showing a UI notification. It's a boolean flag that tells it "You were unexpectedly signed out, sign in again". If the GUI crashes before it's able to "consume" the notification, it'll just show it again on next launch, no biggie (other than an annoyance to the user).

Yes

swift/apple/FirezoneKit/Sources/FirezoneKit/Views/MenuBar.swift

jamilbk · 2024-03-27T02:16:01Z

swift/apple/Firezone/Application/FirezoneApp.swift

-        }
-      }
-    }
+    public var appStore: AppStore?


Yeah, I was confused by that too at first. That's correct.

It could probably use a rename.

jamilbk · 2024-03-27T02:41:22Z

swift/apple/FirezoneNetworkExtension/DeviceMetadata.swift

+  // if that doesn't exist. The Firezone ID is a UUIDv4 that is used to dedup this device
+  // for upsert and identification in the admin portal.
+  public static func getOrCreateFirezoneId(logger: AppLogger) -> String {
+    let fileURL = SharedAccess.baseFolderURL.appendingPathComponent("firezone-id")


These files aren't really accessible outside Firezone. I guess you could append extensions but extensions aren't really common for random app-specific files on Unix I think.

jamilbk · 2024-03-27T02:41:50Z

swift/apple/FirezoneNetworkExtension/DeviceMetadata.swift

+      let newUUIDString = UUID().uuidString
+
+      do {
+        try newUUIDString.write(to: fileURL, atomically: true, encoding: .utf8)


Yeah there are some nice things actually in the Apple Garden™

swift/apple/FirezoneNetworkExtension/NetworkSettings.swift

jamilbk · 2024-03-27T02:43:18Z

swift/apple/FirezoneNetworkExtension/SystemConfigurationResolvers.swift

+
+class SystemConfigurationResolvers {
+  private let logger: AppLogger
+  private var dynamicStore: SCDynamicStore?


It's actually the system's configuration and is not app specific. That's why this API is not available for iOS. I don't believe it's writable due to App sandboxing.

jamilbk · 2024-03-27T02:45:00Z

swift/apple/FirezoneNetworkExtension/SystemConfigurationResolvers.swift

+  private var dynamicStore: SCDynamicStore?
+
+  // Arbitrary name for the connection to the store
+  private let storeName = "dev.firezone.firezone.dns" as CFString


Nah, I think we might be able to write to the SC store, but I'm not sure, since our tunnel process is supposed to be sandboxed (may need an entitlement for that).

https://developer.apple.com/documentation/systemconfiguration/scpreferencessetspecific

…removed (#4324) - Handles an edge case where we would crash if the tunnel configuration was removed in the middle of saving Settings. - Handles an edge case where our tunnel configuration may have been changed by the system -- if it was disabled, we disconnect.

When binding to published variables from other objects, we must use the parameter received in the callback to act upon, and not re-read the stored parameter on the associated object again. The stored parameter can be stale, which leads to UI bugs where we transitioned state but the menubar icon or menu did not get updated. <img width="274" alt="Screenshot 2024-03-25 at 11 45 35 PM" src="https://github.com/firezone/firezone/assets/167144/17a26b52-0599-4523-a7d1-d1bb2e617c44"> This fixes a long-standing bug that has occurred off and on since the early versions.

Co-authored-by: Reactor Scram <ReactorScram@users.noreply.github.com> Signed-off-by: Jamil <jamilbk@users.noreply.github.com>

Signed-off-by: Jamil <jamilbk@users.noreply.github.com>

jamilbk force-pushed the feat/handle-network-changes-apple branch 5 times, most recently from 42e3189 to a7bff54 Compare March 17, 2024 13:15

jamilbk mentioned this pull request Mar 17, 2024

chore(apple): Remove connlib_client_apple log filters because they have no effect #4182

Merged

jamilbk force-pushed the feat/handle-network-changes-apple branch 5 times, most recently from 10860a5 to 6527c5b Compare March 17, 2024 14:37

jamilbk marked this pull request as ready for review March 17, 2024 20:27

jamilbk changed the title ~~feat(apple): Handle network changes~~ feat(apple): Handle network changes reliably on macOS and iOS Mar 17, 2024

jamilbk force-pushed the feat/handle-network-changes-apple branch from c559e97 to 2c0ae74 Compare March 17, 2024 20:29

jamilbk requested review from conectado, roop and ReactorScram March 17, 2024 20:29

jamilbk force-pushed the feat/handle-network-changes-apple branch from 2d8fabb to 543ccba Compare March 17, 2024 20:42

jamilbk commented Mar 17, 2024

View reviewed changes

jamilbk force-pushed the feat/handle-network-changes-apple branch from 52feab4 to 67c828f Compare March 18, 2024 21:17

thomaseizinger mentioned this pull request Mar 18, 2024

connlib: debounce reconnects by 500ms #4202

Closed

thomaseizinger reviewed Mar 19, 2024

View reviewed changes

swift/apple/FirezoneNetworkExtension/CallbackHandler.swift Show resolved Hide resolved

thomaseizinger reviewed Mar 19, 2024

View reviewed changes

rust/connlib/clients/apple/src/lib.rs Outdated Show resolved Hide resolved

jamilbk force-pushed the feat/handle-network-changes-apple branch from 1aec868 to 5895f8a Compare March 26, 2024 05:19

ReactorScram approved these changes Mar 26, 2024

View reviewed changes

jamilbk commented Mar 27, 2024

View reviewed changes

jamilbk and others added 14 commits March 26, 2024 19:47

Greatly simplify Apple client state

e77d5ae

Refactor how we store, parse, and display resources

b78d67a

Check for tunnel status moving back to .invalid on iOS

1363672

Stricter settings field validators

34ba208

Fix spelling

7953bcc

Don't allow clicking Sign out when disconnecting

d464659

Set DNS immediately after WrappedSession.connect

36c1bc8

Use kSecAttrAccessGroup for shared Keychain access on macOS too

828e062

Simplify signed out state

2de10df

rework AppLogger

91b9a2f

Update swift/apple/FirezoneKit/Sources/FirezoneKit/Views/MenuBar.swift

b81821a

Co-authored-by: Reactor Scram <ReactorScram@users.noreply.github.com> Signed-off-by: Jamil <jamilbk@users.noreply.github.com>

Update swift/apple/FirezoneNetworkExtension/NetworkSettings.swift

13b3765

Signed-off-by: Jamil <jamilbk@users.noreply.github.com>

jamilbk force-pushed the feat/handle-network-changes-apple branch from 696f3ea to 13b3765 Compare March 27, 2024 02:48

jamilbk enabled auto-merge March 27, 2024 02:48

jamilbk added this pull request to the merge queue Mar 27, 2024

Merged via the queue into main with commit ab598ef Mar 27, 2024
152 checks passed

jamilbk deleted the feat/handle-network-changes-apple branch March 27, 2024 03:13

feat(apple): Handle network changes reliably on macOS and iOS #4133

feat(apple): Handle network changes reliably on macOS and iOS #4133

Conversation

jamilbk commented Mar 13, 2024 • edited

vercel bot commented Mar 13, 2024 • edited

github-actions bot commented Mar 13, 2024 • edited

Terraform Cloud Plan Output

github-actions bot commented Mar 13, 2024 • edited

Performance Test Results

TCP

UDP

jamilbk commented Mar 17, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thomaseizinger commented Mar 17, 2024

jamilbk commented Mar 18, 2024

thomaseizinger left a comment

Choose a reason for hiding this comment

thomaseizinger commented Mar 26, 2024

jamilbk commented Mar 26, 2024

thomaseizinger commented Mar 26, 2024

jamilbk commented Mar 26, 2024

jamilbk commented Mar 26, 2024

jamilbk commented Mar 26, 2024

jamilbk commented Mar 26, 2024 • edited

ReactorScram left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jamilbk commented Mar 13, 2024 •

edited

vercel bot commented Mar 13, 2024 •

edited

github-actions bot commented Mar 13, 2024 •

edited

github-actions bot commented Mar 13, 2024 •

edited

jamilbk commented Mar 17, 2024 •

edited

jamilbk commented Mar 26, 2024 •

edited