Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libQuic integration via libSession #960

Merged
merged 55 commits into from
May 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
470323e
Plugged in some debug code to try and start testing libQuic
mpretty-cyro Mar 14, 2024
1602cd7
Added missing 'sendRequest' function
mpretty-cyro Mar 15, 2024
8548a86
Latest libSession build config
mpretty-cyro Mar 15, 2024
defd664
[WIP] Latest tweaks, still have linker errors...
mpretty-cyro Mar 18, 2024
0ae4327
Latest libSession with tweaks for libQuic linking
mpretty-cyro Mar 19, 2024
09e8fed
Merge remote-tracking branch 'upstream/dev' into feature/lib-quic-int…
mpretty-cyro Mar 20, 2024
756e256
Moving libSession into SessionUtilitiesKit to expose it to other targets
mpretty-cyro Mar 20, 2024
8ef1c24
[WIP] Working on the libQuic onion requests
mpretty-cyro Mar 22, 2024
1ee4c29
[WIP] Pulling in a bunch of changes around request management
mpretty-cyro Mar 26, 2024
7eb7eda
[WIP] Cleaned up interface, error handling and redundant code
mpretty-cyro Mar 28, 2024
9833ac9
Fixed the broken unit tests
mpretty-cyro Apr 4, 2024
097f9c2
Fixed scheme and SessionSnodeKitTest runs
mpretty-cyro Apr 4, 2024
4818a6d
Merge remote-tracking branch 'origin/dev' into feature/lib-quic-integ…
mpretty-cyro Apr 4, 2024
0f370ab
Changed the SnodeAPIError to be CustomStringConvertible
mpretty-cyro Apr 4, 2024
5fa9e31
Latest libSession commit
mpretty-cyro Apr 4, 2024
b16851b
Updated to the latest libQuic commit
mpretty-cyro Apr 4, 2024
c830c9f
Updated the CI config to use SSH and the new build machine
mpretty-cyro Apr 4, 2024
927f820
Update to the latest build config
mpretty-cyro Apr 5, 2024
fa40b09
Use keep-alive for simulators
jagerman Apr 5, 2024
e3a0c3e
Reformat .drone.jsonnet with jsonnetfmt
jagerman Apr 5, 2024
b65ca55
set LANG in environment; add xcresultparser/pip status
jagerman Apr 5, 2024
57824e1
Fire unit tests on push
jagerman Apr 5, 2024
0c7baa0
Bump libsession-util with libquic update
jagerman Apr 5, 2024
9aeb94d
Merge pull request #14 from jagerman/libquic-integration
mpretty-cyro Apr 7, 2024
95d9e5c
Changed unit test pipeline back to PR only (codecov will error otherw…
mpretty-cyro Apr 7, 2024
9bc837e
Updated codecov config to ignore some UI-only files
mpretty-cyro Apr 8, 2024
c065abc
Cleaned up a bunch of warnings, renamed a couple of folders
mpretty-cyro Apr 8, 2024
8c467dc
Merge remote-tracking branch 'origin/fix/sync-message-issues' into fe…
mpretty-cyro Apr 12, 2024
c6c2881
Reworked the libSession network to be instance based
mpretty-cyro Apr 12, 2024
afe1efb
Deduped path building and attempted to improve extension logging
mpretty-cyro Apr 12, 2024
2cffda1
Merge remote-tracking branch 'upstream/dev' into feature/lib-quic-int…
mpretty-cyro Apr 12, 2024
a3188eb
Fixed a number of bugs found in the internal release
mpretty-cyro Apr 15, 2024
5c0fd4a
Merge remote-tracking branch 'upstream/dev' into feature/lib-quic-int…
mpretty-cyro Apr 15, 2024
3ea5868
Increased build number
mpretty-cyro Apr 15, 2024
352f6d7
Fixed a number of bugs found while testing the internal build
mpretty-cyro Apr 16, 2024
8d99f64
Fixed a few more bugs
mpretty-cyro Apr 17, 2024
5ee15bb
Moved a bunch of networking logic into libSession
mpretty-cyro Apr 23, 2024
99abcde
Removed some unneeded code and fixed a couple of bugs
mpretty-cyro Apr 24, 2024
9491b4a
Fixed a crash and config issue
mpretty-cyro Apr 29, 2024
eccaa29
Fixed the background crash issues
mpretty-cyro May 2, 2024
4b4c69a
Fixed broken unit tests, tweak to CI script
mpretty-cyro May 2, 2024
fa7a584
Updated to the fixed version of libQuic
mpretty-cyro May 2, 2024
93f136c
Fixed an issue where reactions wouldn't appear
mpretty-cyro May 2, 2024
fee7f09
Fixed keyboard obscuring the calls UI in one case, fixed QA ids
mpretty-cyro May 7, 2024
fb8a162
Updated the CI sim build command to use unbuffered io
mpretty-cyro May 7, 2024
9e5a938
Use the new logger, fixed build script bug, latest libSession
mpretty-cyro May 9, 2024
47001b7
Added manual log calls for testing
mpretty-cyro May 9, 2024
481b515
Logging testing
mpretty-cyro May 10, 2024
bc6bab2
Build script update to handle submodule errors/warnings and disable t…
mpretty-cyro May 10, 2024
994854f
Expose libSession submodule errors in build script, libSession path b…
mpretty-cyro May 10, 2024
7dc811a
Lastest libSession tweaks
mpretty-cyro May 10, 2024
f50d7fc
Updated to the latest libSession, added new build arg
mpretty-cyro May 11, 2024
e62e088
Removed test logs
mpretty-cyro May 11, 2024
021d05d
Fixed a build script bug, updated to latest libSession
mpretty-cyro May 13, 2024
444655f
Latest libSession
mpretty-cyro May 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
133 changes: 74 additions & 59 deletions .drone.jsonnet
Original file line number Diff line number Diff line change
@@ -1,38 +1,42 @@
// This build configuration requires the following to be installed:
// Git, Xcode, XCode Command-line Tools, Cocoapods, Xcbeautify, Xcresultparser, pip
// Git, Xcode, XCode Command-line Tools, Cocoapods, xcbeautify, xcresultparser, pip

// Log a bunch of version information to make it easier for debugging
local version_info = {
name: 'Version Information',
environment: { LANG: 'en_US.UTF-8' },
commands: [
'git --version',
'LANG=en_US.UTF-8 pod --version',
'pod --version',
'xcodebuild -version',
'xcbeautify --version'
]
'xcbeautify --version',
'xcresultparser --version',
'pip --version',
],
};

// Intentionally doing a depth of 2 as libSession-util has it's own submodules (and libLokinet likely will as well)
local clone_submodules = {
name: 'Clone Submodules',
commands: [ 'git submodule update --init --recursive --depth=2 --jobs=4' ]
commands: ['git submodule update --init --recursive --depth=2 --jobs=4'],
};

// cmake options for static deps mirror
local ci_dep_mirror(want_mirror) = (if want_mirror then ' -DLOCAL_MIRROR=https://oxen.rocks/deps ' else '');

// Cocoapods
//
//
// Unfortunately Cocoapods has a dumb restriction which requires you to use UTF-8 for the
// 'LANG' env var so we need to work around the with https://github.com/CocoaPods/CocoaPods/issues/6333
local install_cocoapods = {
name: 'Install CocoaPods',
environment: { LANG: 'en_US.UTF-8' },
commands: [
'LANG=en_US.UTF-8 pod install || (rm -rf ./Pods && LANG=en_US.UTF-8 pod install)'
'pod install || (rm -rf ./Pods && pod install)',
],
depends_on: [
'Load CocoaPods Cache'
]
'Load CocoaPods Cache',
],
};

// Load from the cached CocoaPods directory (to speed up the build)
Expand All @@ -41,26 +45,26 @@ local load_cocoapods_cache = {
commands: [
|||
LOOP_BREAK=0
while test -e /Users/drone/.cocoapods_cache.lock; do
while test -e /Users/$USER/.cocoapods_cache.lock; do
sleep 1
LOOP_BREAK=$((LOOP_BREAK + 1))

if [[ $LOOP_BREAK -ge 600 ]]; then
rm -f /Users/drone/.cocoapods_cache.lock
rm -f /Users/$USER/.cocoapods_cache.lock
fi
done
|||,
'touch /Users/drone/.cocoapods_cache.lock',
'touch /Users/$USER/.cocoapods_cache.lock',
|||
if [[ -d /Users/drone/.cocoapods_cache ]]; then
cp -r /Users/drone/.cocoapods_cache ./Pods
if [[ -d /Users/$USER/.cocoapods_cache ]]; then
cp -r /Users/$USER/.cocoapods_cache ./Pods
fi
|||,
'rm -f /Users/drone/.cocoapods_cache.lock'
'rm -f /Users/$USER/.cocoapods_cache.lock',
],
depends_on: [
'Clone Submodules'
]
'Clone Submodules',
],
};

// Override the cached CocoaPods directory (to speed up the next build)
Expand All @@ -69,69 +73,80 @@ local update_cocoapods_cache(depends_on) = {
commands: [
|||
LOOP_BREAK=0
while test -e /Users/drone/.cocoapods_cache.lock; do
while test -e /Users/$USER/.cocoapods_cache.lock; do
sleep 1
LOOP_BREAK=$((LOOP_BREAK + 1))

if [[ $LOOP_BREAK -ge 600 ]]; then
rm -f /Users/drone/.cocoapods_cache.lock
rm -f /Users/$USER/.cocoapods_cache.lock
fi
done
|||,
'touch /Users/drone/.cocoapods_cache.lock',
'touch /Users/$USER/.cocoapods_cache.lock',
|||
if [[ -d ./Pods ]]; then
rsync -a --delete ./Pods/ /Users/drone/.cocoapods_cache
rsync -a --delete ./Pods/ /Users/$USER/.cocoapods_cache
fi
|||,
'rm -f /Users/drone/.cocoapods_cache.lock'
'rm -f /Users/$USER/.cocoapods_cache.lock',
],
depends_on: depends_on,
};

local boot_simulator(device_type) = {
name: 'Boot Test Simulator',
commands: [
'devname="Test-iPhone-${DRONE_COMMIT:0:9}-${DRONE_BUILD_EVENT}"',
'xcrun simctl create "$devname" ' + device_type,
'sim_uuid=$(xcrun simctl list devices -je | jq -re \'[.devices[][] | select(.name == "\'$devname\'").udid][0]\')',
'xcrun simctl boot $sim_uuid',

'mkdir -p build/artifacts',
'echo $sim_uuid > ./build/artifacts/sim_uuid',
'echo $devname > ./build/artifacts/device_name',

'xcrun simctl list -je devices $sim_uuid | jq -r \'.devices[][0] | "\\u001b[32;1mSimulator " + .state + ": \\u001b[34m" + .name + " (\\u001b[35m" + .deviceTypeIdentifier + ", \\u001b[36m" + .udid + "\\u001b[34m)\\u001b[0m"\'',
],
};
local sim_keepalive = {
name: '(Simulator keep-alive)',
commands: [
'/Users/$USER/sim-keepalive/keepalive.sh $(<./build/artifacts/sim_uuid)',
],
depends_on: ['Boot Test Simulator'],
};
local sim_delete_cmd = 'if [ -f build/artifacts/sim_uuid ]; then rm -f /Users/$USER/sim-keepalive/$(<./build/artifacts/sim_uuid); fi';

[
// Unit tests (PRs only)
{
kind: 'pipeline',
type: 'exec',
name: 'Unit Tests',
platform: { os: 'darwin', arch: 'arm64' },
trigger: { event: { exclude: [ 'push' ] } },
trigger: { event: { exclude: ['push'] } },
steps: [
version_info,
clone_submodules,
load_cocoapods_cache,
install_cocoapods,
{
name: 'Clean Up Old Test Simulators',
commands: [
'./Scripts/clean-up-old-test-simulators.sh'
]
},
{
name: 'Pre-Boot Test Simulator',
commands: [
'mkdir -p build/artifacts',
'echo "Test-iPhone14-${DRONE_COMMIT:0:9}-${DRONE_BUILD_EVENT}" > ./build/artifacts/device_name',
'xcrun simctl create "$(<./build/artifacts/device_name)" com.apple.CoreSimulator.SimDeviceType.iPhone-14',
'echo $(xcrun simctl list devices | grep -m 1 $(<./build/artifacts/device_name) | grep -E -o -i "([0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12})") > ./build/artifacts/sim_uuid',
'xcrun simctl boot $(<./build/artifacts/sim_uuid)',
'echo "Pre-booting simulator complete: $(xcrun simctl list | sed "s/^[[:space:]]*//" | grep -o ".*$(<./build/artifacts/sim_uuid).*")"',
]
},

boot_simulator('com.apple.CoreSimulator.SimDeviceType.iPhone-15'),
sim_keepalive,
{
name: 'Build and Run Tests',
commands: [
'NSUnbufferedIO=YES set -o pipefail && xcodebuild test -workspace Session.xcworkspace -scheme Session -derivedDataPath ./build/derivedData -resultBundlePath ./build/artifacts/testResults.xcresult -parallelizeTargets -destination "platform=iOS Simulator,id=$(<./build/artifacts/sim_uuid)" -parallel-testing-enabled NO -test-timeouts-enabled YES -maximum-test-execution-time-allowance 10 -collect-test-diagnostics never 2>&1 | xcbeautify --is-ci',
],
depends_on: [
'Pre-Boot Test Simulator',
'Install CocoaPods'
'Boot Test Simulator',
'Install CocoaPods',
],
},
{
name: 'Unit Test Summary',
commands: [
sim_delete_cmd,
|||
if [[ -d ./build/artifacts/testResults.xcresult ]]; then
xcresultparser --output-format cli --failed-tests-only ./build/artifacts/testResults.xcresult
Expand All @@ -142,8 +157,8 @@ local update_cocoapods_cache(depends_on) = {
],
depends_on: ['Build and Run Tests'],
when: {
status: ['failure', 'success']
}
status: ['failure', 'success'],
},
},
update_cocoapods_cache(['Build and Run Tests']),
{
Expand All @@ -157,15 +172,15 @@ local update_cocoapods_cache(depends_on) = {
which codecovcli > ./build/artifacts/codecov_path
fi
|||,
'$(<./build/artifacts/codecov_path) --version'
'$(<./build/artifacts/codecov_path) --version',
],
},
{
name: 'Convert xcresult to xml',
commands: [
'xcresultparser --output-format cobertura ./build/artifacts/testResults.xcresult > ./build/artifacts/coverage.xml',
],
depends_on: ['Build and Run Tests']
depends_on: ['Build and Run Tests'],
},
{
// No token needed for public repos
Expand All @@ -175,8 +190,8 @@ local update_cocoapods_cache(depends_on) = {
],
depends_on: [
'Convert xcresult to xml',
'Install Codecov CLI'
]
'Install Codecov CLI',
],
},
],
},
Expand All @@ -186,23 +201,23 @@ local update_cocoapods_cache(depends_on) = {
type: 'exec',
name: 'Check Build Artifact Existence',
platform: { os: 'darwin', arch: 'arm64' },
trigger: { event: { exclude: [ 'push' ] } },
trigger: { event: { exclude: ['push'] } },
steps: [
{
name: 'Poll for build artifact existence',
commands: [
'./Scripts/drone-upload-exists.sh'
]
}
]
'./Scripts/drone-upload-exists.sh',
],
},
],
},
// Simulator build (non-PRs only)
{
kind: 'pipeline',
type: 'exec',
name: 'Simulator Build',
platform: { os: 'darwin', arch: 'arm64' },
trigger: { event: { exclude: [ 'pull_request' ] } },
trigger: { event: { exclude: ['pull_request'] } },
steps: [
version_info,
clone_submodules,
Expand All @@ -212,22 +227,22 @@ local update_cocoapods_cache(depends_on) = {
name: 'Build',
commands: [
'mkdir build',
'xcodebuild archive -workspace Session.xcworkspace -scheme Session -derivedDataPath ./build/derivedData -parallelizeTargets -configuration "App Store Release" -sdk iphonesimulator -archivePath ./build/Session_sim.xcarchive -destination "generic/platform=iOS Simulator" | xcbeautify --is-ci'
'NSUnbufferedIO=YES set -o pipefail && xcodebuild archive -workspace Session.xcworkspace -scheme Session -derivedDataPath ./build/derivedData -parallelizeTargets -configuration "App Store Release" -sdk iphonesimulator -archivePath ./build/Session_sim.xcarchive -destination "generic/platform=iOS Simulator" | xcbeautify --is-ci',
],
depends_on: [
'Install CocoaPods'
'Install CocoaPods',
],
},
update_cocoapods_cache(['Build']),
{
name: 'Upload artifacts',
environment: { SSH_KEY: { from_secret: 'SSH_KEY' } },
commands: [
'./Scripts/drone-static-upload.sh'
'./Scripts/drone-static-upload.sh',
],
depends_on: [
'Build'
]
'Build',
],
},
],
},
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,6 @@ Index/

# CocoaPods
Pods

# VSCode
.vscode
2 changes: 1 addition & 1 deletion LibSession-Util
Submodule LibSession-Util updated 132 files
23 changes: 17 additions & 6 deletions Podfile
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,14 @@ install! 'cocoapods', :warn_for_unused_master_specs_repo => false
# Dependencies to be included in the app and all extensions/frameworks
abstract_target 'GlobalDependencies' do
# FIXME: If https://github.com/jedisct1/swift-sodium/pull/249 gets resolved then revert this back to the standard pod
pod 'Sodium', :git => 'https://github.com/oxen-io/session-ios-swift-sodium.git', branch: 'session-build'
pod 'Sodium', :git => 'https://github.com/oxen-io/session-ios-swift-sodium.git', commit: '310c343'
pod 'GRDB.swift/SQLCipher'

# FIXME: Would be nice to migrate from CocoaPods to SwiftPackageManager (should allow us to speed up build time), haven't gone through all of the dependencies but currently unfortunately SQLCipher doesn't support SPM (for more info see: https://github.com/sqlcipher/sqlcipher/issues/371)
pod 'SQLCipher', '~> 4.5.3'
pod 'SQLCipher', '~> 4.5.7'
pod 'WebRTC-lib'

target 'Session' do
pod 'Reachability'
pod 'PureLayout', '~> 3.1.8'
pod 'NVActivityIndicatorView'
pod 'YYImage/libwebp', git: 'https://github.com/signalapp/YYImage'
Expand All @@ -36,7 +35,6 @@ abstract_target 'GlobalDependencies' do
pod 'SignalCoreKit', git: 'https://github.com/oxen-io/session-ios-core-kit', :commit => '3acbfe5'

target 'SessionNotificationServiceExtension'
target 'SessionSnodeKit'

# Dependencies that are shared across a number of extensions/frameworks but not all
abstract_target 'ExtendedDependencies' do
Expand All @@ -49,15 +47,13 @@ abstract_target 'GlobalDependencies' do

target 'SignalUtilitiesKit' do
pod 'NVActivityIndicatorView'
pod 'Reachability'
pod 'SAMKeychain'
pod 'SwiftProtobuf', '~> 1.5.0'
pod 'YYImage/libwebp', git: 'https://github.com/signalapp/YYImage'
pod 'DifferenceKit'
end

target 'SessionMessagingKit' do
pod 'Reachability'
pod 'SAMKeychain'
pod 'SwiftProtobuf', '~> 1.5.0'
pod 'DifferenceKit'
Expand Down Expand Up @@ -86,6 +82,21 @@ abstract_target 'GlobalDependencies' do
end
end
end

target 'SessionSnodeKit' do
target 'SessionSnodeKitTests' do
inherit! :complete

pod 'Quick'
pod 'Nimble'

# Need to include these for the tests because otherwise it won't actually build
pod 'SAMKeychain'
pod 'PureLayout', '~> 3.1.8'
pod 'YYImage/libwebp', git: 'https://github.com/signalapp/YYImage'
pod 'DifferenceKit'
end
end
end

target 'SessionUIKit' do
Expand Down
Loading