Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[expo-av] Streaming Audio Buffers in realtime (JSI Audio) #13516

Closed
wants to merge 188 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
188 commits
Select commit Hold shift + click to select a range
4fec599
Hide SplashScreen on `bare-expo` app launch
mrousavy Jul 5, 2021
c71ead0
Remove all Ad modules from TestModules
mrousavy Jul 5, 2021
d97d7c1
Add `tests/Notifications` back to `TestModules`
mrousavy Jul 5, 2021
6de58bc
raw prototype: pass audio sample frames through JSI
mrousavy Jul 6, 2021
c589dc7
prototype: visualize audio in scale form
mrousavy Jul 6, 2021
82ab687
Calculate `vDSP_rmsqv` (effective "loudness") and animate circle scale
mrousavy Jul 6, 2021
c601f50
Update AudioScreen.tsx
mrousavy Jul 6, 2021
bfe3b37
[av] Add `AudioSample`, `AudioChannel` types and `getAverageLoudness`…
mrousavy Jul 6, 2021
2674790
Fix typo
mrousavy Jul 6, 2021
eff42b9
interpolate bass even more
mrousavy Jul 6, 2021
60ccd7c
Add Android `Visualizer`
mrousavy Jul 6, 2021
4f8e153
whoops
mrousavy Jul 6, 2021
97fc68e
revert changes in TestModules.js
mrousavy Jul 7, 2021
3d45442
Revert "Remove all Ad modules from TestModules"
mrousavy Jul 7, 2021
2d69568
Update EXAVPlayerData.mm
mrousavy Jul 7, 2021
46f0663
Update CHANGELOG.md
mrousavy Jul 7, 2021
6a4ad7c
Replace `AVQueuePlayer` with `AVAudioEngine`: First step
mrousavy Jul 7, 2021
870428f
1/n: Implement `_loadNewPlayer`
mrousavy Jul 7, 2021
2811a3b
Create player by id dictionary
mrousavy Jul 8, 2021
24fbc2b
use `NSMapTable` for more efficient memory configuration (weak)
mrousavy Jul 8, 2021
8013cd2
Revert "use `NSMapTable` for more efficient memory configuration (weak)"
mrousavy Jul 8, 2021
00c8739
Revert "Create player by id dictionary"
mrousavy Jul 8, 2021
c2ea046
Implement JS front-end for `__av_sound_setOnAudioSampleReceivedCallback`
mrousavy Jul 8, 2021
ba110e3
check `this._key` type
mrousavy Jul 8, 2021
72a76fd
Update Sound.ts
mrousavy Jul 8, 2021
54fd276
rename `EXAppDelegateWrapper.m` -> `EXAppDelegateWrapper.mm`
mrousavy Jul 8, 2021
84677c3
Update HEADER_SEARCH_PATHS for ExpoModulesCore
mrousavy Jul 8, 2021
894ec57
Create JSI installer
mrousavy Jul 8, 2021
2999b44
fix package.json path of rn
mrousavy Jul 8, 2021
17fe85a
fix: add support for Hermes on RN 65
mrousavy Jul 8, 2021
25cca5b
Fix folly include
mrousavy Jul 8, 2021
f87b831
Manually install Reanimated
mrousavy Jul 8, 2021
6a99087
keep `AVQueuePlayer` to make project build for now
mrousavy Jul 8, 2021
3ea49eb
Register `__av_sound_setOnAudioSampleReceivedCallback` in iOS JS Runtime
mrousavy Jul 8, 2021
12c967e
Move `ModuleRegistryAdapter` from `AppDelegate.swift` to `EXAppDelega…
mrousavy Jul 8, 2021
60421e7
Export JS funcs for all modules
mrousavy Jul 9, 2021
99ca129
Install all objects with correct arg size
mrousavy Jul 9, 2021
3c26731
move JSI binding installation to separate file (`ExpoModuleJSIBinder.…
mrousavy Jul 9, 2021
908b7c5
Create `JSIConverter` utils
mrousavy Jul 9, 2021
581fdcc
EXAppDelegateWrapper: `.mm` -> `.m`
mrousavy Jul 9, 2021
117c942
convert id -> jsi::Value for constants
mrousavy Jul 9, 2021
eeb8ce0
Actually call module with promise.
mrousavy Jul 9, 2021
95287a7
fix ExpoModules initialization
mrousavy Jul 9, 2021
870868c
Lazily initialize Expo Modules by using HostObjects
mrousavy Jul 9, 2021
6244d54
Update LazyExpoModule.mm
mrousavy Jul 9, 2021
987537a
test with IP
mrousavy Jul 9, 2021
29e2c41
Revert "test with IP"
mrousavy Jul 9, 2021
fa56878
Revert "Update LazyExpoModule.mm"
mrousavy Jul 9, 2021
278cdd2
Revert "Lazily initialize Expo Modules by using HostObjects"
mrousavy Jul 9, 2021
4452dd9
Revert "fix ExpoModules initialization"
mrousavy Jul 9, 2021
35a36a0
Revert "Actually call module with promise."
mrousavy Jul 9, 2021
f7ec05d
Revert "convert id -> jsi::Value for constants"
mrousavy Jul 9, 2021
4e50071
Revert "EXAppDelegateWrapper: `.mm` -> `.m`"
mrousavy Jul 9, 2021
c25c553
Revert "Create `JSIConverter` utils"
mrousavy Jul 9, 2021
4de1cb4
Revert "move JSI binding installation to separate file (`ExpoModuleJS…
mrousavy Jul 9, 2021
a2698fc
Revert "Install all objects with correct arg size"
mrousavy Jul 9, 2021
1f0c0fe
Revert "Export JS funcs for all modules"
mrousavy Jul 9, 2021
2eb711d
Revert "Move `ModuleRegistryAdapter` from `AppDelegate.swift` to `EXA…
mrousavy Jul 9, 2021
e2970a8
Revert "Register `__av_sound_setOnAudioSampleReceivedCallback` in iOS…
mrousavy Jul 9, 2021
3950418
Revert "keep `AVQueuePlayer` to make project build for now"
mrousavy Jul 9, 2021
79cf01a
Revert "Manually install Reanimated"
mrousavy Jul 9, 2021
b1ea10c
Revert "Fix folly include"
mrousavy Jul 9, 2021
04db766
Revert "fix: add support for Hermes on RN 65"
mrousavy Jul 9, 2021
8adde1e
Revert "fix package.json path of rn"
mrousavy Jul 9, 2021
cf8ae4c
Revert "Create JSI installer"
mrousavy Jul 9, 2021
7255870
Revert "Update HEADER_SEARCH_PATHS for ExpoModulesCore"
mrousavy Jul 9, 2021
7906833
Revert "rename `EXAppDelegateWrapper.m` -> `EXAppDelegateWrapper.mm`"
mrousavy Jul 9, 2021
1cec39a
EXAV to mm
mrousavy Jul 9, 2021
50d4b52
throw if not found
mrousavy Jul 9, 2021
9d56a01
Add callback base
mrousavy Jul 9, 2021
a54e58b
ADD TEMP PLAYER VAR SO IT COMPILES
mrousavy Jul 9, 2021
003445c
EXAVPlayerData: .mm -> .m
mrousavy Jul 9, 2021
4677e41
add missing imports
mrousavy Jul 9, 2021
84a9504
Extract JSI setup code to separate file (extension)
mrousavy Jul 9, 2021
fa2e052
fix JSI registration
mrousavy Jul 12, 2021
72988d1
Remove/Add
mrousavy Jul 12, 2021
23673df
Convert `AVAudioPCMBuffer` to JS AudioSample type
mrousavy Jul 12, 2021
2e5fcf6
Update EXAV+AudioSampleCallback.mm
mrousavy Jul 12, 2021
4382430
rewrite to AVAudioEngine (1/??)
mrousavy Jul 12, 2021
1eaf822
unload sound
mrousavy Jul 12, 2021
31c15d8
Stop player on file completion
mrousavy Jul 12, 2021
8219881
Run on separate dispatch queue, don't call callback if not node playing
mrousavy Jul 12, 2021
f1d8660
fix `isLooping`
mrousavy Jul 12, 2021
15e8ab7
Update EXAVPlayerData.m
mrousavy Jul 12, 2021
0a11024
Try applying pitch
mrousavy Jul 12, 2021
b20276b
Fix setting new position
mrousavy Jul 12, 2021
0d79f42
Fix setting new position
mrousavy Jul 12, 2021
24feec6
create Android setup
mrousavy Jul 12, 2021
72aeedf
fix pitch calculation
mrousavy Jul 12, 2021
e2176fd
Update EXAVPlayerData.m
mrousavy Jul 12, 2021
31543d9
Update CMakeLists.txt
mrousavy Jul 12, 2021
ca26e65
Update EXAVPlayerData.m
mrousavy Jul 12, 2021
c38e2b6
Update .gitignore
mrousavy Jul 12, 2021
0a871fa
Update MediaPlayerData.java
mrousavy Jul 12, 2021
ccd824c
clean up android setup to a minimal one
mrousavy Jul 12, 2021
08e246d
fix: Enable prefab and setup native CMake build
mrousavy Jul 12, 2021
c0cc9bb
Revert "fix: Enable prefab and setup native CMake build"
mrousavy Jul 12, 2021
4afbd34
Revert "clean up android setup to a minimal one"
mrousavy Jul 12, 2021
74fd218
Create Hybrid Class
mrousavy Jul 12, 2021
48d8605
Update MediaPlayerData.java
mrousavy Jul 12, 2021
3feecca
Use minimal Android gradle/CMake setup with prefab again
mrousavy Jul 12, 2021
a4acfa9
fix c++
mrousavy Jul 12, 2021
d2b9572
Add audio `timestamp` to sample buffer structure
mrousavy Jul 12, 2021
9d4b2bc
Include fbjni with CMake
mrousavy Jul 13, 2021
4024b63
Fix `BUILD_DIR` path
mrousavy Jul 13, 2021
6eb8b50
Install REAv2 for bare-expo/android
mrousavy Jul 13, 2021
d2a2ab7
Add `JAVManager` hybrid class base
mrousavy Jul 13, 2021
1790204
Pass runtime and callinvoker to native hybrid object
mrousavy Jul 13, 2021
b89da51
`installJSIBindings()`...
mrousavy Jul 13, 2021
274479a
Initialize JNI
mrousavy Jul 13, 2021
ce8994c
Load native EXAV library
mrousavy Jul 13, 2021
91cd1fd
Update AVManager.java
mrousavy Jul 13, 2021
4f620bd
base
mrousavy Jul 13, 2021
4b62fe6
Get Media Player by ID
mrousavy Jul 13, 2021
311449f
Build native JNI/JSI array
mrousavy Jul 13, 2021
bc35a0a
fix JSI setup
mrousavy Jul 13, 2021
a8438b8
Fix frames array creation
mrousavy Jul 13, 2021
16f8d03
Use abstract PlayerData class instead of concrete MediaPlayer (add ex…
mrousavy Jul 13, 2021
c56b301
Move Visualizer management/init to `PlayerData`
mrousavy Jul 13, 2021
de9fcfb
Ask for RECORD_AUDIO permission in visualizer
mrousavy Jul 13, 2021
b1d61b1
Update PlayerData.java
mrousavy Jul 13, 2021
e936b36
Call on JS thread
mrousavy Jul 13, 2021
18b2557
Use `unique_ptr` for move
mrousavy Jul 13, 2021
4d2931c
Fix callback rate
mrousavy Jul 13, 2021
ee9b642
Remove a few log statements
mrousavy Jul 13, 2021
a2f2e3a
Normalize to -1.0 ... 1.0
mrousavy Jul 13, 2021
465beef
Fix buffer index insert
mrousavy Jul 13, 2021
c148a49
reinterpret whole array
mrousavy Jul 13, 2021
812df22
Update JAVManager.cpp
mrousavy Jul 13, 2021
fda5ec7
Correctly release Visualizer
mrousavy Jul 13, 2021
a2b624a
Update AudioScreen.tsx
mrousavy Jul 13, 2021
be7f693
Android: Add timestamp to sample buffer callback
mrousavy Jul 13, 2021
b0d8eae
Add TODOs to use jsi::ArrayBuffer
mrousavy Jul 13, 2021
a18b84a
avoid smart pointer move
mrousavy Jul 13, 2021
571deb6
revert all iOS changes.
mrousavy Jul 27, 2021
1f3d45d
Try using C API `MTAudioProcessingTapRef`
mrousavy Jul 27, 2021
f2d598c
play around with MTAudioProcessingTap
mrousavy Jul 27, 2021
b9182a4
Fix `installTap`
mrousavy Jul 27, 2021
9348c5f
Fix multi-channel interleaved calculation
mrousavy Jul 27, 2021
51ebc9f
Disable progress update intervals for now
mrousavy Jul 27, 2021
20a9272
interpret as correct type (float)
mrousavy Jul 28, 2021
59ac4f5
fix calculating array size instead of byte size
mrousavy Jul 28, 2021
d5e648b
Fix position getting
mrousavy Jul 28, 2021
fb33848
cleanup
mrousavy Jul 28, 2021
0181378
move to class funcs
mrousavy Jul 28, 2021
1073114
Re-Install tap when item changes
mrousavy Jul 28, 2021
d93cbe1
Add possibility to uninstall tap
mrousavy Jul 28, 2021
a3da6c2
Show waveform in Audio Screen
mrousavy Jul 28, 2021
2fff2e2
Only install tap if buffer callback is set
mrousavy Jul 28, 2021
dd8d09c
Add `null` check too
mrousavy Jul 28, 2021
bc8883d
remove x song
mrousavy Jul 28, 2021
317a42e
remove unused code
mrousavy Jul 28, 2021
deb2adb
Revert unrelated changes, add docs
mrousavy Jul 28, 2021
3648948
rebuild expo-av
mrousavy Jul 28, 2021
b513166
fix NODE_MODULES path resolution
mrousavy Jul 28, 2021
c0bf95a
fix imports
mrousavy Jul 28, 2021
a292323
Update MainApplication.java
mrousavy Jul 28, 2021
3cdc2c0
fix imports
mrousavy Jul 28, 2021
4b4d6ce
Install JSI bindings for EXAV on correct Thread (`mqt_js`)
mrousavy Jul 28, 2021
24c8419
Use single `**/`
mrousavy Jul 28, 2021
88b4ced
Log error if .aar cannot be found
mrousavy Jul 28, 2021
0f322e7
Avoid SharedPointer by using `__block` var
mrousavy Jul 29, 2021
ada224f
Update packages/expo-av/android/src/main/cpp/EXAV.cpp
mrousavy Jul 29, 2021
50548ea
Update packages/expo-av/android/src/main/cpp/JAVManager.cpp
mrousavy Jul 29, 2021
52d1eb3
fix copyright notes
mrousavy Jul 29, 2021
75eea10
Use `declare global`
mrousavy Jul 29, 2021
3a07a94
rebuild expo-av
mrousavy Jul 29, 2021
eea1ccc
Use `EXLogInfo` instead of `NSLog`
mrousavy Jul 29, 2021
c6c2aac
Update packages/expo-av/ios/EXAV/EXAV.m
mrousavy Jul 29, 2021
051c304
use `std::to_string()`
mrousavy Jul 29, 2021
0162ba3
blyat
mrousavy Jul 29, 2021
cd7ab44
Fix `type` instead of `interface`
mrousavy Jul 30, 2021
642ae47
Fix indents in `EXAV.m`
mrousavy Jul 30, 2021
93ec18b
apply suggestions
mrousavy Jul 30, 2021
44cbb7e
rebuild expo-av
mrousavy Jul 30, 2021
f068b86
namespace to `expo::av`
mrousavy Jul 30, 2021
1e055f9
Remove `using namespace ...`
mrousavy Jul 30, 2021
15513c9
fix jni namespace
mrousavy Jul 30, 2021
3f55d25
fix namespace
mrousavy Jul 30, 2021
3599b17
Fix `expo-av` library loading
mrousavy Jul 31, 2021
f3dd048
remove `EXLogInfo` log
mrousavy Aug 2, 2021
57e8a06
fix `AudioSample` import
mrousavy Aug 2, 2021
55a3355
Fix JNI C++ types for ctor
mrousavy Aug 2, 2021
f21086b
load native library for `PlayerData`
mrousavy Aug 2, 2021
c246011
Only invoke sample buffer if player should be playing
mrousavy Aug 2, 2021
2350afe
Use `reinterpret_cast` for `uint8_t` casting
mrousavy Aug 2, 2021
0145bf6
rebuild expo-av
mrousavy Aug 19, 2021
1183776
checkout react-native-lab from master
mrousavy Aug 19, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions apps/bare-expo/android/app/proguard-rules.pro
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,4 @@
#-keepclassmembers class fqcn.of.javascript.interface.for.webview {
# public *;
#}
-keep class com.facebook.react.turbomodule.** { *; }
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@
import com.facebook.react.ReactPackage;
import com.facebook.soloader.SoLoader;

import com.facebook.react.bridge.JSIModulePackage;
import com.swmansion.reanimated.ReanimatedJSIModulePackage;

import java.util.List;

import androidx.annotation.NonNull;
Expand Down Expand Up @@ -36,6 +39,11 @@ protected List<ReactPackage> getPackages() {
protected String getJSMainModuleName() {
return "index";
}

@Override
protected JSIModulePackage getJSIModulePackage() {
return new ReanimatedJSIModulePackage();
}
});

@Override
Expand Down
1 change: 1 addition & 0 deletions apps/bare-expo/app.json
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
}
},
"android": {
"jsEngine": "hermes",
"scheme": [
"com.googleusercontent.apps.29635966244-knmlpr1upnv6rs4bumqea7hpit4o7kg2"
],
Expand Down
1 change: 1 addition & 0 deletions apps/bare-expo/ios/BareExpo.xcodeproj/project.pbxproj

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.