Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: native plugins #3372

Open
wants to merge 27 commits into
base: master
from

Conversation

@afinch7
Copy link
Contributor

afinch7 commented Nov 18, 2019

Now that we are using futures 0.3 I should be able to finish this.

Api will look a little different this time to reflect some of the refactors that have occurred since #2385.

I'm targeting something like this on the plugin side:

use deno::CoreOp;
use deno::Op;
use deno::{Buf, PinnedBuf};
use deno::PluginInitContext;
use futures::future::FutureExt;

#[macro_use]
extern crate deno;

fn init(context: &mut dyn PluginInitContext) {
  context.register_op("some_op", Box::new(some_op));
}
init_fn!(init);

pub fn some_op(data: &[u8], zero_copy: Option<PinnedBuf>) -> CoreOp {
  if let Some(buf) = zero_copy {
    let data_str = std::str::from_utf8(&data[..]).unwrap();
    let buf_str = std::str::from_utf8(&buf[..]).unwrap();
    println!(
      "Hello from native bindings. data: {} | zero_copy: {}",
      data_str, buf_str
    );
  }
  let result = b"test";
  Op::Sync(result.into())
}

and something like this on the typescript side:

const plugin = Deno.openPlugin("./path/to/some/lib.so");
const some_op = plugin.ops.some_op;

function doSomeOp(): null | ArrayBufferView {
  const response = some_op.dispatch(new UInt8Array([1,2,3,4]));
  return response;
}

refs #3366 #2473

@afinch7 afinch7 force-pushed the afinch7:native_plugins_2 branch 2 times, most recently from fef9c0c to 661e5a4 Nov 18, 2019
Copy link
Contributor

bartlomieju left a comment

There are a few things I have questions about, but I believe this might be very solid MVP for native plugins;
a) it's built on ops
b) this is the simplest iteration, doesn't expose state to op dispatcher in Rust, we can iterate on API safely

cli/ops/native_plugins.rs Outdated Show resolved Hide resolved
cli/ops/native_plugins.rs Outdated Show resolved Hide resolved
@afinch7 afinch7 force-pushed the afinch7:native_plugins_2 branch from 4cdeeb2 to 51910ee Nov 18, 2019
cli/ops/native_plugins.rs Outdated Show resolved Hide resolved
Copy link
Collaborator

ry left a comment

I like that this is only +400 lines

@afinch7 afinch7 force-pushed the afinch7:native_plugins_2 branch 2 times, most recently from 7df0cbd to 1fc95be Nov 19, 2019
@afinch7 afinch7 force-pushed the afinch7:native_plugins_2 branch from 1fc95be to 112622d Nov 19, 2019
@afinch7 afinch7 marked this pull request as ready for review Nov 19, 2019
@afinch7

This comment has been minimized.

Copy link
Contributor Author

afinch7 commented Nov 20, 2019

For some reason workers_startup_bench.ts doesn't terminate now. I don't think that I changed anything that would break that.

@afinch7 afinch7 force-pushed the afinch7:native_plugins_2 branch from 45f301d to cb94110 Nov 20, 2019
cli/worker.rs Outdated Show resolved Hide resolved
@afinch7 afinch7 force-pushed the afinch7:native_plugins_2 branch from 5c796ea to 62c126c Nov 22, 2019
@afinch7 afinch7 changed the title [WIP] feat: native plugins feat: native plugins Nov 22, 2019
Copy link
Contributor

bartlomieju left a comment

It is very solid PR, especially because all changes introduced in it are minimal - it allows to iterate on the API before 1.0 release. It's well aligned with our ops APIs and has minimal impact on current architecture 👍 I'd love to see this landed to give users chance to use it and break it 🚀

@@ -129,6 +129,7 @@ jobs:
- name: Test debug
if: matrix.kind == 'test_debug'
run: |
cargo build --locked -p test_native_plugin

This comment has been minimized.

Copy link
@bartlomieju

bartlomieju Nov 22, 2019

Contributor

Can this step be somehow integrated into cli/tests/integration_tests.rs flow?

This comment has been minimized.

Copy link
@afinch7

afinch7 Nov 23, 2019

Author Contributor

I tried to add a single test to test_native_plugin, but it still doesn't build automatically when running cargo test --all-targets. Not sure if there may be some other simple workaround here.

This comment has been minimized.

Copy link
@ry

ry Nov 24, 2019

Collaborator

We should figure out a way to test this without making changes to ci.yml... I'm looking into it.

cli/flags.rs Outdated Show resolved Hide resolved
cli/js/dispatch.ts Show resolved Hide resolved
cli/js/native_plugins.ts Outdated Show resolved Hide resolved
cli/ops/native_plugins.rs Outdated Show resolved Hide resolved
cli/permissions.rs Outdated Show resolved Hide resolved
cli/tests/test_native_plugin/src/lib.rs Outdated Show resolved Hide resolved
@afinch7 afinch7 force-pushed the afinch7:native_plugins_2 branch from f780f53 to 0bb50b3 Nov 23, 2019
* Rename `allow_native`/`allow-native` to `allow_plugin`/`allow-plugin`. This makes more sense with the primary public api function being `openPlugin`.
* Fully intagrate the newly added permission type `plugin`.
* Add docs to `Deno.openPlugin`.
* Generate op names for native plugin ops that avoid potential collision.

Lastly this commit adds a delayed future to native plugins async test op. This should properly test the function of contexts/tasks/wakers in async native plugin ops. This would not function correctly with futures 0.1, and thus was a major blocker. This test proves this issue no longer exists in this implementation. The most analogous problem I can think of here is loading two copies of the same TS lib: the share the same type thus are interoperable, but don't share the same module local data(closest comparison to TLS used to store current task in futures). This isn't a problem anymore, since futures now pass `Context` values directly during calls to `poll` with the new API.
@afinch7 afinch7 force-pushed the afinch7:native_plugins_2 branch from 0bb50b3 to 96d4afc Nov 23, 2019
Cargo.toml Outdated Show resolved Hide resolved
Copy link
Collaborator

ry left a comment

@afinch7 This is wonderful - this has come so far since the the original +5000 line patch. It's really uplifting to see that a lot of the small changes we've done like dynamic op registration allow for this major feature with relatively little bits of code.

One general comment - you alternatively use the terms "native plugin" and just "plugin" throughout the code. I think "native plugin" is redundant and potentially confusing (one might ask 'what's a non-native plugin?'). For consistency, I suggest changing all usages of "native plugin" to just "plugin".

ry added 4 commits Nov 24, 2019
fix
fix
@ry

This comment has been minimized.

Copy link
Collaborator

ry commented Nov 27, 2019

I've merged this branch with master and moved a few things around. I think the plugin infrastructure shouldn't depend on cli, so the tests shouldn't be in cli. I'll try to do that now...

ry added 4 commits Nov 27, 2019
@ry

This comment has been minimized.

Copy link
Collaborator

ry commented Nov 28, 2019

So.. I'm still having the same trouble you were having, @afinch7, with getting the plugin to build for the test. I guess moving the test_plugin crate to the root didn't help. I've done some superficial clean ups. I'm pretty much ready to land this if we can get it to go green somehow.

afinch7 added 6 commits Dec 3, 2019
fixes
* Removed "native" wording from plugins
* Build `test_plugin` before running debug tests in CI
fn basic() {
let mut build_plugin_base = Command::new("cargo");
let mut build_plugin =
build_plugin_base.arg("build").arg("-p").arg("test_plugin");

This comment has been minimized.

Copy link
@ry

ry Dec 3, 2019

Collaborator

It's annoying that it's not built already when the integration tests run. I wish I knew some cargo expert who could advise us here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
3 participants
You can’t perform that action at this time.