Skip to content

Commit

Permalink
feat: Create and restore backup (AR-1421) (#1097)
Browse files Browse the repository at this point in the history
* empty commit

* feat: Add a new ChaCha20 encryption class for backup files (#1098)

* feat: add ChaCha20 encrypting class

* Added ChaCha20Test class

* Added libsodium library

* Moved ChaCha20Utils class to commonMain

* Refactored Backup object and reworked encryption/decryption methods

* Fixed issues not hashing password correctly

* Passed backup version as additional data to encrypt/decrypt processes

* fix: wip but works

* Fixed tests with big files

* Renamings

* Bumped minimum Android version

* Addressed PR requests

* Revert changes

* Code cleaning

* Revert okio to 3.0.0

* Flushing to write the header data

* Fix issue with clashing libs

* Added mechanism to map a Backup.Header object from buffered source

* Fixed detekt issues

* detekt

* Added manual buffer reading operations

* More detekt

* More detekt

* solved libsodium conflicts on Core

* Last final touches

* Removed suspendable modifiers

Co-authored-by: Jacob Persson <7156+typfel@users.noreply.github.com>

* feat: perform the restore of the database data from another database file (AR-2668) (#1096)

* feat: [WIP] import backed up database

* feat: [WIP] Import data from an older database backup

* work on unit testing

* in progress

* work in progress on unit tests

* unit tests in progress

* cover assets

* revert

* test

* address the comments

* add the comments

* address the pr comments

* address the changes

* add remaining test

* all is looking fine

* chore: edit auth tokens refresh tokens (#1125)

* chore: add http method network logs (#1126)

* chore: add http method network logs

* fix add missing "

* fix: MaxLineLength - (#1127)

* ready for review

* suppress

* fix

* fix

* Empty-Commit

* Empty-Commit

* suppress

* push

* test

* suppress file

* fix

* further fix

* bump up android version

* Trigger notification

* add arch

* fix the import

* fix

* fix

* add dispatcher

* Update persistence/src/commonTest/kotlin/com/wire/kalium/persistence/backup/UserDatabaseDataGenerator.kt

Co-authored-by: Gonzo <gongracr@gmail.com>

* Update persistence/src/commonTest/kotlin/com/wire/kalium/persistence/backup/UserDatabaseDataGenerator.kt

Co-authored-by: Gonzo <gongracr@gmail.com>

* Update persistence/src/commonTest/kotlin/com/wire/kalium/persistence/backup/UserDatabaseDataGenerator.kt

Co-authored-by: Gonzo <gongracr@gmail.com>

* test

* fix it

* fix

* fix

* fix

* fix

* fix

* fix

* Empty-Commit

* change

* fix detekt

* done

Co-authored-by: Vitor Hugo Schwaab <vitor@schwaab.xyz>
Co-authored-by: mateusz.pachulski <mateusz.pachulski@appunite.com>
Co-authored-by: Gonzalo Gran Crespo <gongracr@gmail.com>
Co-authored-by: Mohamad Jaara <mohamad.jaara@wire.com>

* added libsodium bindings library to catalogue

* feat: Wire all needed logic to create and restore backups [AR-1421] [AR-1422] (#1158)

* Added ClientPlatform class

* feat: glued logic to enable encryption mechanism for backups

* Added final compression for encrypted backup file

* Added method to extract compressed files

* Added logic to restore encrypted backup files

* Forwarding invalid password error

* Fixed detekt issues

* refactor: share token cache among all http clients (#1159)

* refactor: share token cache among all http clients

* fix tests

* fix styling

* Address PR comments

* fix: session upgrade (#1156)

* fix: clear cached token after upgrading the session

* fix: start slow sync after we've upgraded the session

* refactor: rename PersistRegisteredClientIdUseCase to VerifyExistingClientUseCase since it's no longer persists anything

* test: verify that client id gets persisted in GetOrRegisterClientUseCase

* fix: don't expose RegisterClientUseCase, it should be considered an internal use case

* test: update test since RegisterClientUseCase no longer persist the client id

* chore: fix detekt

* chore: fix test service

* refactor: clear cached token on shared auth bearer provider

* fix: assets download silently failing (#1162)

* fix: assets download silently failing

* fix: assets download silently failing

* fix tests

* revert ApiTest back to interface

* fix tests with non json response expecting json string

* fix styling

* Improved read and write compression methods for large files

* chore: obfuscate conversation ids for calling (#1166)

Co-authored-by: Mohamad Jaara <mohamad.jaara@wire.com>

* fix: introduce ImportClientUseCase for importing an existing client (#1165)

* fix: introduce ImportClientUseCase for importing an existing client

* test: add test for persisting retained client id

* test: add ImportClientUseCaseTest

* feat(mls): join self &  team conversation (#1157)

* feat: add support for global team conversation type

* feat: distinguish between MLS and proteus self conversations

* feat: establish group if necessary when joining

* feat: fetch global team conversation during slow sync

* chore: cleanup

* refactor: take group info as a parameter instead of fetching it

* test: update tests

* chore: fix detekt

* test: add tests for establishing self / team conversation when epoch is zero

* revert: accidental key press

* chore: return proper API not supported response

* chore: update comment with issue links

* Update network/src/commonMain/kotlin/com/wire/kalium/network/api/base/authenticated/conversation/ConversationResponse.kt

Co-authored-by: Mojtaba Chenani <chenani@outlook.com>

* chore: disable fetching GTC

* chore: fix detekt

Co-authored-by: Mojtaba Chenani <chenani@outlook.com>

* Added more granular error handling

* Detekt fixes

* More detekt fixes

* feat: multi account persistence web socket [AR-2257] (#1147)

* persistent web socket for multiple account

* delete duplicated function

* update the sync repo

* fix some tests

* update tests

* generate db schema

* resolve pr comment s

* update ObservePersistentWebSocketConnectionStatusUseCase

* import boolean in the migration file

* fix detekt

* update the logic for connection policy

* fix detekt

Co-authored-by: Mohamad Jaara <mohamad.jaara@wire.com>

* Moved BackupUtils to common jvm android folder

* Fixed final detekt issue

Co-authored-by: Mohamad Jaara <mohamad.jaara@wire.com>
Co-authored-by: Jacob Persson <7156+typfel@users.noreply.github.com>
Co-authored-by: Oussama Hassine <oussama.has100@gmail.com>
Co-authored-by: Mojtaba Chenani <chenani@outlook.com>
Co-authored-by: Mohammed Mokresh <mokresh.mohammed@gmail.com>

* Added extra exclusion rules for generated code

* Added extract initial backup file use case class

* Renamed use case to make it more generic

* detekt issue

* Addressed PR comments

* Added tests

* Added more tests

* Detekt issues

* Updated detekt config file to ignore build types

* Improved readability

* Addressed some requested comments

* Fixed test issues

* Added extractCompressedBackupUseCaseTest

* Fixed more tests

* Added comment

* Detekt issues

* Removed spaces from unit tests

* Removed more spaces

* run android CI on 30

* fix

* fix

* fix

* refactor - implement IsBackupEncryptedUseCase

* Added mechanism to provide passphrase for encrypted db

* Simplified code

* is database encrypted flag should come from kalium configs

* postpone passphrase reading from metadata and change to bytearray

* Added createBackupUseCase tests

* changed IsBackupEncryptedUseCase to VerifyBackupUseCase, added tests

* test: don't create a static test file system

* partially fix RestoreBackupUseCaseTest

* partially fix RestoreBackupUseCaseTest

* partially fix RestoreBackupUseCaseTest

* Fixed CreateBackupUseCaseTest

* Fixed tests

* fix unit test failing

* Fixed test

* Fixed detekt issue

* More detekt

* Final touch

Co-authored-by: Gonzalo Gran Crespo <gongracr@gmail.com>
Co-authored-by: Jacob Persson <7156+typfel@users.noreply.github.com>
Co-authored-by: Vitor Hugo Schwaab <vitor@schwaab.xyz>
Co-authored-by: mateusz.pachulski <mateusz.pachulski@appunite.com>
Co-authored-by: Mohamad Jaara <mohamad.jaara@wire.com>
Co-authored-by: Oussama Hassine <oussama.has100@gmail.com>
Co-authored-by: Mojtaba Chenani <chenani@outlook.com>
Co-authored-by: Mohammed Mokresh <mokresh.mohammed@gmail.com>
Co-authored-by: Jakub 呕erko <zerasop@gmail.com>
Co-authored-by: Micha艂 Saleniuk <saleniuk@gmail.com>
  • Loading branch information
11 people committed Dec 15, 2022
1 parent 53b39cf commit 108ea84
Show file tree
Hide file tree
Showing 57 changed files with 3,206 additions and 90 deletions.
132 changes: 67 additions & 65 deletions .github/workflows/gradle-android-tests.yml
Original file line number Diff line number Diff line change
@@ -1,84 +1,86 @@
name: "Android Tests"

on:
pull_request:
types: [ opened, synchronize ] # Don't rerun on `edited` to save time
pull_request:
types: [ opened, synchronize ] # Don't rerun on `edited` to save time

concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
cancel-in-progress: true

jobs:
gradle-run-tests:
runs-on: macos-latest
strategy:
matrix:
api-level: [29]
gradle-run-tests:
runs-on: macos-latest
strategy:
matrix:
api-level: [30]

steps:
- name: Checkout
uses: actions/checkout@v3
with:
fetch-depth: 0
steps:
- name: Checkout
uses: actions/checkout@v3
with:
fetch-depth: 0

- name: Set up JDK 11
uses: actions/setup-java@v3
with:
java-version: '11'
distribution: 'adopt'
- name: Set up JDK 11
uses: actions/setup-java@v3
with:
java-version: '11'
distribution: 'adopt'

- name: Gradle Setup
uses: gradle/gradle-build-action@v2
- name: Gradle Setup
uses: gradle/gradle-build-action@v2

- name: Validate Gradle wrapper
uses: gradle/wrapper-validation-action@e6e38bacfdf1a337459f332974bb2327a31aaf4b
- name: Validate Gradle wrapper
uses: gradle/wrapper-validation-action@e6e38bacfdf1a337459f332974bb2327a31aaf4b

- name: AVD cache
uses: actions/cache@v3
id: avd-cache
with:
path: |
~/.android/avd/*
~/.android/adb*
key: avd-${{ matrix.api-level }}
- name: AVD cache
uses: actions/cache@v3
id: avd-cache
with:
path: |
~/.android/avd/*
~/.android/adb*
key: avd-${{ matrix.api-level }}

- name: Create AVD and generate snapshot for caching
if: steps.avd-cache.outputs.cache-hit != 'true'
uses: reactivecircus/android-emulator-runner@v2
with:
api-level: ${{ matrix.api-level }}
force-avd-creation: false
emulator-options: -no-window -gpu swiftshader_indirect -noaudio -no-boot-anim -camera-back none
disable-animations: false
script: echo "Generated AVD snapshot for caching."
- name: Create AVD and generate snapshot for caching
if: steps.avd-cache.outputs.cache-hit != 'true'
uses: reactivecircus/android-emulator-runner@v2
with:
api-level: ${{ matrix.api-level }}
force-avd-creation: false
target: google_apis
emulator-options: -no-window -gpu swiftshader_indirect -noaudio -no-boot-anim -camera-back none
disable-animations: false
script: echo "Generated AVD snapshot for caching."

- name: Android Instrumentation Tests
uses: reactivecircus/android-emulator-runner@v2
with:
api-level: ${{ matrix.api-level }}
script: ./gradlew connectedAndroidOnlyAffectedTest
env:
GITHUB_USER: ${{ github.actor }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Android Instrumentation Tests
uses: reactivecircus/android-emulator-runner@v2
with:
api-level: ${{ matrix.api-level }}
target: google_apis
script: ./gradlew connectedAndroidOnlyAffectedTest
env:
GITHUB_USER: ${{ github.actor }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

- name: Archive Test Reports
if: always()
uses: actions/upload-artifact@v3
with:
name: test-reports
path: ./**/build/reports/tests/**
- name: Archive Test Reports
if: always()
uses: actions/upload-artifact@v3
with:
name: test-reports
path: ./**/build/reports/tests/**

- name: Publish Unit Test Results
uses: EnricoMi/publish-unit-test-result-action/composite@v1.25
if: always()
with:
files: |
**/build/test-results/**/*.xml
**/build/outputs/androidTest-results/**/*.xml
- name: Publish Unit Test Results
uses: EnricoMi/publish-unit-test-result-action/composite@v1.25
if: always()
with:
files: |
**/build/test-results/**/*.xml
**/build/outputs/androidTest-results/**/*.xml
- name: Cleanup Gradle Cache
# Remove some files from the Gradle cache, so they aren't cached by GitHub Actions.
# Restoring these files from a GitHub Actions cache might cause problems for future builds.
run: |
rm -f ~/.gradle/caches/modules-2/modules-2.lock
rm -f ~/.gradle/caches/modules-2/gc.properties
- name: Cleanup Gradle Cache
# Remove some files from the Gradle cache, so they aren't cached by GitHub Actions.
# Restoring these files from a GitHub Actions cache might cause problems for future builds.
run: |
rm -f ~/.gradle/caches/modules-2/modules-2.lock
rm -f ~/.gradle/caches/modules-2/gc.properties
2 changes: 1 addition & 1 deletion buildSrc/src/main/kotlin/Android.kt
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
object Android {
const val testRunner = "androidx.test.runner.AndroidJUnitRunner"
object Sdk {
const val min = 23
const val min = 24
const val compile = 33
const val target = compile
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ fun LibraryExtension.commonAndroidLibConfig(includeNativeInterop: Boolean) {
}
packagingOptions {
resources.pickFirsts.add("google/protobuf/*.proto")
jniLibs.pickFirsts.add("**/libsodium.so")
}
// No Android Unit test. JVM does that. Android runs on emulator
sourceSets.remove(sourceSets.getByName("test"))
Expand Down
6 changes: 6 additions & 0 deletions cryptography/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,14 @@ kotlin {
implementation(libs.coroutines.core)
api(libs.ktor.core)

// KTX
implementation(libs.ktxDateTime)

// Okio
implementation(libs.okio.core)

// Libsodium
implementation(libs.libsodiumBindingsMP)
}
}
val commonTest by getting {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ internal class AESEncrypt {
}
}
} catch (e: Exception) {
kaliumLogger.e("There was an error while encrypting the asset:\n $e}")
kaliumLogger.e("There was an error while encrypting the asset with AES256:\n $e}")
} finally {
assetDataSource.close()
outputSink.close()
Expand Down Expand Up @@ -124,7 +124,7 @@ internal class AESDecrypt(private val secretKey: AES256Key) {
}
kaliumLogger.d("WROTE $size bytes")
} catch (e: Exception) {
kaliumLogger.e("There was an error while decrypting the asset:\n $e}")
kaliumLogger.e("There was an error while decrypting the asset with AES256:\n $e}")
} finally {
encryptedDataSource.close()
outputSink.close()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,12 @@ package com.wire.kalium.cryptography.utils

import com.wire.kalium.cryptography.kaliumLogger
import io.ktor.util.encodeBase64
import okio.BufferedSink
import okio.HashingSink
import okio.Sink
import okio.Source
import okio.blackholeSink
import okio.buffer
import java.security.MessageDigest
import kotlin.io.use

actual fun calcMd5(bytes: ByteArray): String = bytes.let {
val md = MessageDigest.getInstance("MD5")
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
package com.wire.kalium.cryptography.backup

import com.ionspin.kotlin.crypto.pwhash.PasswordHash
import com.ionspin.kotlin.crypto.pwhash.crypto_pwhash_ALG_DEFAULT
import com.ionspin.kotlin.crypto.pwhash.crypto_pwhash_SALTBYTES
import com.wire.kalium.cryptography.CryptoUserID
import com.wire.kalium.cryptography.backup.BackupHeader.HeaderDecodingErrors
import com.wire.kalium.cryptography.backup.BackupHeader.HeaderDecodingErrors.INVALID_FORMAT
import com.wire.kalium.cryptography.backup.BackupHeader.HeaderDecodingErrors.INVALID_USER_ID
import com.wire.kalium.cryptography.backup.BackupHeader.HeaderDecodingErrors.INVALID_VERSION
import com.wire.kalium.cryptography.kaliumLogger
import okio.Buffer
import okio.IOException
import okio.Source
import kotlin.random.Random
import kotlin.random.nextUBytes

@OptIn(ExperimentalUnsignedTypes::class)
class BackupCoder(val userId: CryptoUserID, val passphrase: Passphrase) {

fun encodeHeader(): BackupHeader {
val salt = Random.nextUBytes(crypto_pwhash_SALTBYTES)
val hashedUserId = hashUserId(userId, salt, OPSLIMIT_INTERACTIVE_VALUE, MEMLIMIT_INTERACTIVE_VALUE)
return BackupHeader(format, version, salt, hashedUserId, OPSLIMIT_INTERACTIVE_VALUE, MEMLIMIT_INTERACTIVE_VALUE)
}

fun decodeHeader(encryptedDataSource: Source): Pair<HeaderDecodingErrors?, BackupHeader> {
val decodedHeader = encryptedDataSource.readBackupHeader()

// Sanity checks
val expectedHashedUserId = hashUserId(userId, decodedHeader.salt, decodedHeader.opslimit, decodedHeader.memlimit)
val storedHashedUserId = decodedHeader.hashedUserId
val decodingError = handleHeaderDecodingErrors(decodedHeader, expectedHashedUserId, storedHashedUserId)
return decodingError to decodedHeader
}

private fun handleHeaderDecodingErrors(
decodedHeader: BackupHeader,
expectedHashedUserId: UByteArray,
storedHashedUserId: UByteArray
): HeaderDecodingErrors? =
when {
!expectedHashedUserId.contentEquals(storedHashedUserId.toUByteArray()) -> {
kaliumLogger.e("The hashed user id in the backup file header does not match the expected one")
INVALID_USER_ID
}
decodedHeader.format != format -> {
kaliumLogger.e("The backup format found in the backup file header is not a valid one")
INVALID_FORMAT
}
decodedHeader.version.toInt() < version.toInt() -> {
kaliumLogger.e("The backup version found in the backup file header is not a valid one")
INVALID_VERSION
}
else -> null
}

@Suppress("ComplexMethod")
@Throws(IOException::class)
private fun Source.readBackupHeader(): BackupHeader {
val readBuffer = Buffer()

// We read the backup header and execute some sanity checks
val format = this.read(readBuffer, BACKUP_HEADER_FORMAT_LENGTH).let { size ->
readBuffer.readByteArray(size).decodeToString().also {
readBuffer.clear()
}
}

// We skip the extra gap
read(readBuffer, BACKUP_HEADER_EXTRA_GAP_LENGTH).also { readBuffer.clear() }

val version = read(readBuffer, BACKUP_HEADER_VERSION_LENGTH).let { size ->
readBuffer.readByteArray(size).decodeToString().also {
readBuffer.clear()
}
}

val salt = read(readBuffer, crypto_pwhash_SALTBYTES.toLong()).let { size ->
readBuffer.readByteArray(size).toUByteArray().also { readBuffer.clear() }
}

val hashedUserId = read(readBuffer, PWD_HASH_OUTPUT_BYTES.toLong()).let { size ->
readBuffer.readByteArray(size).toUByteArray().also { readBuffer.clear() }
}

val opslimit = this.read(readBuffer, UNSIGNED_INT_LENGTH).let {
readBuffer.readInt().also { readBuffer.clear() }
}

val memlimit = this.read(readBuffer, UNSIGNED_INT_LENGTH).let {
readBuffer.readInt().also { readBuffer.clear() }
}

return BackupHeader(
format = format,
version = version,
salt = salt,
hashedUserId = hashedUserId,
opslimit = opslimit,
memlimit = memlimit
)
}

// ChaCha20 SecretKey used to encrypt derived from the passphrase (salt + provided password)
internal fun generateChaCha20Key(header: BackupHeader): UByteArray {
return PasswordHash.pwhash(
PWD_HASH_OUTPUT_BYTES,
passphrase.password,
header.salt,
header.opslimit.toULong(),
header.memlimit,
crypto_pwhash_ALG_DEFAULT
)
}

private fun hashUserId(userId: CryptoUserID, salt: UByteArray, opslimit: Int, memlimit: Int): UByteArray {
return PasswordHash.pwhash(
PWD_HASH_OUTPUT_BYTES,
userId.toString(),
salt,
opslimit.toULong(),
memlimit,
crypto_pwhash_ALG_DEFAULT
)
}

companion object {
// Defined by given specs on: https://wearezeta.atlassian.net/wiki/spaces/ENGINEERIN/pages/59867179/Exporting+history+v2
private const val MEMLIMIT_INTERACTIVE_VALUE = 33554432
private const val OPSLIMIT_INTERACTIVE_VALUE = 4
private const val PWD_HASH_OUTPUT_BYTES = 32
private const val UNSIGNED_INT_LENGTH = 4L
private const val BACKUP_HEADER_EXTRA_GAP_LENGTH = 1L
private const val BACKUP_HEADER_FORMAT_LENGTH = 4L
private const val BACKUP_HEADER_VERSION_LENGTH = 2L

// Wire Backup Generic format identifier
private const val format = "WBUX"

// Current Wire Backup version
const val version = "03"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
package com.wire.kalium.cryptography.backup

import okio.Buffer

@OptIn(ExperimentalUnsignedTypes::class)
data class BackupHeader(
val format: String,
val version: String,
val salt: UByteArray,
val hashedUserId: UByteArray,
val opslimit: Int,
val memlimit: Int
) {

private val extraGap = byteArrayOf(0x00)

fun toByteArray(): ByteArray {
val buffer = Buffer()
buffer.write(format.encodeToByteArray())
buffer.write(extraGap)
buffer.write(version.encodeToByteArray())
buffer.write(salt.toByteArray())
buffer.write(hashedUserId.toByteArray())
buffer.writeInt(opslimit)
buffer.writeInt(memlimit)

return buffer.readByteArray()
}

enum class HeaderDecodingErrors {
INVALID_USER_ID, INVALID_VERSION, INVALID_FORMAT
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
package com.wire.kalium.cryptography.backup

data class Passphrase(val password: String)

0 comments on commit 108ea84

Please sign in to comment.