Tx5 Holochain WebRTC Integration #1741

neonphog · 2023-01-04T20:30:55Z

Summary

Adds feature flipper "tx5" which enables WebRTC networking in holochain.
There are a few TODOs added here that I want to leave in until we have more infrastructure available to start testing.
This is mostly just a minimal abstraction so we can run our QUIC networking in parallel until we're ready to switch the default to tx5.

TODO:

CHANGELOG(s) updated with appropriate info
Just before pressing the merge button, ensure new entries to CHANGELOG(s) are still under the UNRELEASED heading

neonphog · 2023-01-23T23:24:04Z

nix/shells.nix

@@ -58,6 +58,7 @@ rec {
      gh
      nixpkgs-fmt
      cargo-sweep
+      go


@steveej - is this the correct place to put a new build dependency for holochain?

this just provides it to the developer shell.

native toolchains that are used during the build process go here

we're in the process of switching our nix derivations for which i can't blindly suggest the place. which crate(s) will need the go binary available for its builds?

@steveej - kitsune_p2p is the direct dependent, but everything downstream of that, including holochain will obviously need it too, then.

hnrg... @steveej maybe it makes the most sense to wait for your refactor to finish before trying to get go into holonix?

jost-s

It looks orderly and readable to me. Are there any tests with tx5 enabled?

neonphog · 2023-01-24T20:01:27Z

It looks orderly and readable to me. Are there any tests with tx5 enabled?

@jost-s - no we need some more infrastructure (TURN servers, etc) before we can start testing it... We'll need some local docker TURN servers for local testing, but that will come in a later PR.

maackle

Some questions about the necessity of lair integration, and whether the two tx features are meant to be XOR or OR.

maackle · 2023-01-25T18:42:47Z

crates/holochain/src/conductor/kitsune_host_impl.rs

@@ -32,6 +34,8 @@ pub struct KitsuneHostImpl {
    ribosome_store: RwShare<RibosomeStore>,
    tuning_params: KitsuneP2pTuningParams,
    strat: ArqStrat,
+    lair_tag: Option<Arc<str>>,
+    lair_client: Option<lair_keystore_api::LairClient>,


hmm kitsune needs to be integrated with lair now?

also, should these Options go together? Do they both need to be Some or None together or can they be independent?

They could go together, but there are also some small options if they are separate:

if you specify the lair_client and not the tag, you'll get a new node id each time. maybe useful?

if you specify the tag and not the client, you'll always know which tag is the node id, even in new lair clients. maybe useful?

not sure : )

hmm kitsune needs to be integrated with lair now?

Yeah, this is the most expeditious path forward. If we want to remove that specific dependency in the future, we can write an abstraction, but I don't think it's a high priority.

maackle · 2023-01-25T18:44:24Z

crates/kitsune_p2p/kitsune_p2p/src/config.rs

+        #[cfg(feature = "tx2")]
+        {
+            #[cfg(feature = "tx5")]
+            {


This code requires both features tx2 and tx5, is that intended?

Yes. We only have to do the match to determine if we're in tx2 mode if tx5 is enabled, otherwise we know we're in tx2 mode. This will be different if we ever have a third option

maackle · 2023-01-25T18:48:13Z

crates/kitsune_p2p/kitsune_p2p/src/spawn/actor.rs

+        #[cfg(feature = "tx2")]
+        if ep_hnd.is_none() && config.is_tx2() {
+            tracing::trace!("tx2");
+            let (h, e) = MetaNet::new_tx2(config.clone(), tls_config, metrics).await?;
+            ep_hnd = Some(h);
+            ep_evt = Some(e);
+        }

-        // capture endpoint handle
-        let ep_hnd = ep.handle().clone();
+        #[cfg(feature = "tx5")]
+        if ep_hnd.is_none() && config.is_tx5() {
+            tracing::trace!("tx5");
+            let signal_url = match config.transport_pool.get(0).unwrap() {
+                TransportConfig::WebRTC { signal_url } => signal_url.clone(),
+                _ => unreachable!(),
+            };
+            let (h, e) =
+                MetaNet::new_tx5(config.tuning_params.clone(), host.clone(), signal_url).await?;
+            ep_hnd = Some(h);
+            ep_evt = Some(e);
+        }


Is it desired that the tx2 endpoint is clobbered if both features are enabled?

It won't be clobbered because of the ep_hnd.is_none() check. So, if tx2 is enabled && the config is_tx2() we will get a tx2 hnd.

neonphog · 2023-01-25T20:02:03Z

@maackle

and whether the two tx features are meant to be XOR or OR.

The intention is that if both features are enabled, the conductor config will control which is used. I believe I've accomplished that : )

…p uninstall (#1805) (#1838) * feat(nix): fix nextest derivation for macos (#1822) * fix(github/workflows/holonix-integration): use multi-arch runner (#1826) * fix(github/workflows/holonix-integration): use multi-arch runner * fix(github/workflows/holonix-integration): make work on multi-arch * update holochain-nixpkgs on develop (#1830) update nix sources see VERSIONS.md for the exact changes Co-authored-by: Holochain Release Automation <hrabot@holochain.org> * update holochain-nixpkgs on develop (#1832) update nix sources see VERSIONS.md for the exact changes Co-authored-by: Holochain Release Automation <hrabot@holochain.org> * update holochain-nixpkgs on develop (#1833) update nix sources see VERSIONS.md for the exact changes Co-authored-by: Holochain Release Automation <hrabot@holochain.org> * feat(CHANGELOGs): facilitate bump to next minor pre-release (#1823) * Tx5 Holochain WebRTC Integration (#1741) * tx4 feature flipper * checkpoint * webrtc config * abstraction checkpoint * networking abstraction * tx4 integration checkpoint * tx4 request * checkpoint integration * merge fixes * remove local crate overrides * test fixes * run tx5 tests on CI * ci config fix * add go to ci build * add go to coreDev flavor * tx5 update * var fix * changelog * limit jobs to prevent OOM * nextest build jobs * shrink debug info * no go debug symbols; revert some unhelpful ci config * fix(ssh-session): correct nix-daemon settings (#1834) * refactor(generate-readme): switch to cargo-rdme (#1820) * docs(hdi): prepare readme and lib for cargo-rdme * docs(hdk): prepare lib and readme for cargo-rdme * docs(holochain_keystore): prepare readme for cargo-rdme * docs(holochain_state): prepare lib and readme for cargo-rdme * docs(hdk+hdi): delete obsolete tpl files * refactor(generate_readmes): replace cargo-readme with cargo-rdme * fix(hdk): intra-doc link to hdk_extern macro * ci(readme): remove cargo-readme and install cargo-rdme * ci(create-readme): configure git * build(generate-readme): unset git name & email afterwards * docs(crate-level): generate readmes from doc comments * fix clippy --------- Co-authored-by: release-ci <ci@holo.host> * update holochain-nixpkgs on develop (#1836) update nix sources see VERSIONS.md for the exact changes Co-authored-by: Holochain Release Automation <hrabot@holochain.org> * Delete authored data from db on cell cleanup after app uninstall (#1805) * Delete all authored and dht data when last cell in a space leaves * Changelog * Incorporate PR feedback * Clean up authored and dht db when space is empty of local cells * Little cleanup * build.rs checks for schema diff * Rename schema files to version 0 * Create our first db migration * Some better docs * Set schema diff to original ref, filter out additions * Fix migrations logic, add tests * Fix migration * Clippy fix * Fix panicky trace * Use transactions when applying migrations * Check against develop for schema changes * update holochain-nixpkgs on develop (#1841) update nix sources see VERSIONS.md for the exact changes Co-authored-by: Holochain Release Automation <hrabot@holochain.org> * docs(nix): add comments to nix flake (#1829) * add comments to flake * add default dev shell module * feat(nix): add dev shells to flake * update changelog * delete redundant dev shells module * undo changelog update --------- Co-authored-by: Stefan Junker <1181362+steveeJ@users.noreply.github.com> --------- Co-authored-by: DavHau <hsngrmpf+github@gmail.com> Co-authored-by: Stefan Junker <1181362+steveeJ@users.noreply.github.com> Co-authored-by: Holochain Release Automation <100725712+holochain-release-automation2@users.noreply.github.com> Co-authored-by: Holochain Release Automation <hrabot@holochain.org> Co-authored-by: David Braden <neonphog@gmail.com> Co-authored-by: Jost Schulte <jost.schulte@protonmail.com> Co-authored-by: release-ci <ci@holo.host>

neonphog added 13 commits January 4, 2023 13:30

tx4 feature flipper

acbfad9

checkpoint

3c5a8a4

Merge branch 'develop' into tx4-integration-rust-ver

434d7df

webrtc config

9034261

Merge branch 'develop' into tx4-integration

4140ae9

abstraction checkpoint

7426bf9

networking abstraction

a026d30

Merge branch 'develop' into tx4-integration

dfc3c41

tx4 integration checkpoint

f05085d

tx4 request

da6d9c7

checkpoint integration

18adf41

Merge branch 'develop' into tx4-integration

909260f

Merge branch 'develop' into tx4-integration

ef55fa9

neonphog changed the title ~~WIP - tx4 holochain integration~~ WIP - tx5 holochain integration Jan 19, 2023

neonphog added 14 commits January 20, 2023 10:24

merge fixes

f820f78

Merge branch 'develop' into tx4-integration

8dc5923

Merge branch 'develop' into tx4-integration

26d1489

remove local crate overrides

3a0393b

test fixes

a857991

Merge branch 'develop' into tx4-integration

1ffafdf

run tx5 tests on CI

ebdf52e

ci config fix

50e5a13

add go to ci build

444e7d4

Merge branch 'develop' into tx4-integration

4c0b352

add go to coreDev flavor

9319929

Merge branch 'develop' into tx4-integration

39d3f7d

tx5 update

bba5d52

var fix

3c89f3b

neonphog changed the title ~~WIP - tx5 holochain integration~~ Tx5 Holochain Integration Jan 23, 2023

neonphog marked this pull request as ready for review January 23, 2023 23:15

neonphog requested review from maackle, jost-s, steveej and thedavidmeister January 23, 2023 23:16

neonphog added 2 commits January 23, 2023 16:20

Merge branch 'develop' into tx4-integration

d17e5b9

changelog

7d4863e

neonphog commented Jan 23, 2023

View reviewed changes

neonphog changed the title ~~Tx5 Holochain Integration~~ Tx5 Holochain WebRTC Integration Jan 23, 2023

thedavidmeister approved these changes Jan 24, 2023

View reviewed changes

neonphog added 3 commits January 24, 2023 09:43

Merge branch 'develop' into tx4-integration

cca6e84

limit jobs to prevent OOM

6ab5cc0

nextest build jobs

88826d2

jost-s approved these changes Jan 24, 2023

View reviewed changes

neonphog added 5 commits January 24, 2023 13:04

shrink debug info

19667c1

no go debug symbols; revert some unhelpful ci config

bae1fb8

Merge branch 'develop' into tx4-integration

e42927f

Merge branch 'develop' into tx4-integration

704161a

Merge branch 'develop' into tx4-integration

b48cd45

maackle reviewed Jan 25, 2023

View reviewed changes

neonphog added 3 commits January 25, 2023 14:17

Merge branch 'develop' into tx4-integration

0362d53

Merge branch 'develop' into tx4-integration

b04a0f2

Merge branch 'develop' into tx4-integration

bbd580f

neonphog merged commit c145f08 into develop Jan 30, 2023

neonphog deleted the tx4-integration branch January 30, 2023 16:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tx5 Holochain WebRTC Integration #1741

Tx5 Holochain WebRTC Integration #1741

neonphog commented Jan 4, 2023 •

edited

neonphog Jan 23, 2023

steveej Jan 24, 2023

neonphog Jan 24, 2023

neonphog Jan 24, 2023

jost-s left a comment

neonphog commented Jan 24, 2023

maackle left a comment

maackle Jan 25, 2023

neonphog Jan 25, 2023

neonphog Jan 25, 2023

maackle Jan 25, 2023

neonphog Jan 25, 2023

maackle Jan 25, 2023

neonphog Jan 25, 2023

neonphog commented Jan 25, 2023

Tx5 Holochain WebRTC Integration #1741

Tx5 Holochain WebRTC Integration #1741

Conversation

neonphog commented Jan 4, 2023 • edited

Summary

TODO:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jost-s left a comment

Choose a reason for hiding this comment

neonphog commented Jan 24, 2023

maackle left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neonphog commented Jan 25, 2023

neonphog commented Jan 4, 2023 •

edited