Native library support for Pub

[DartBot]

Originally opened as dart-lang/sdk#3023

This issue was originally filed by sammcca...@google.com

---

Pub should define a layout convention for extensions with native library components (https://github.com/sam-mccall/dart-sqlite is a simple example).

Pub should be able to build the native components as part of installation, on systems with the appropriate toolchain available.

Pub should also be able to share the resulting platform-specific binary package, so that other users don't have to compile it.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/5479?v=3" align="left" width="48" height="48"hspace="10"> Comment by sethladd

---

Agreed!

---

cc @munificent.
Removed Type-Defect label.
Added Type-Enhancement, Area-Pub, Triaged labels.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/5479?v=3" align="left" width="48" height="48"hspace="10"> Comment by sethladd

---

cc @whesse.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/4905639?v=3" align="left" width="48" height="48"hspace="10"> Comment by whesse

---

We have adopted the convention of putting shared libraries in the same directory as the extension, without thinking too much about it. We also support relative paths and absolute paths.

Soon we should reevaluate where the native libs should go, thinking about deployment and security issues as well. Do you think the current solution is the right one?

[DartBot]

This comment was originally written by sammcca...@google.com

---

.so-in-the-same-directory has been very convenient for me, easy to understand, and keeps the library directory self-contained.

I'd like to keep it, if it doesn't complicate security or pub too much.

[DartBot]

This comment was originally written by pastrana.lo...@gmail.com

---

A "Dartify" step could be nice at pull time in Pub ..

[DartBot]

<img src="https://avatars.githubusercontent.com/u/46275?v=3" align="left" width="48" height="48"hspace="10"> Comment by munificent

---

Added this to the Later milestone.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/46275?v=3" align="left" width="48" height="48"hspace="10"> Comment by munificent

---

Removed this from the Later milestone.
Added this to the M1 milestone.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/46275?v=3" align="left" width="48" height="48"hspace="10"> Comment by munificent

---

Removed this from the M1 milestone.
Added this to the Later milestone.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/46275?v=3" align="left" width="48" height="48"hspace="10"> Comment by munificent

---

This looks very cool, Andrew. We aren't working on native extension support right now, but when we get to it, I'll take a closer look at what you have here. Thanks!

[DartBot]

<img src="https://avatars.githubusercontent.com/u/5479?v=3" align="left" width="48" height="48"hspace="10"> Comment by sethladd

---

Issue #250 has been merged into this issue.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/46275?v=3" align="left" width="48" height="48"hspace="10"> Comment by munificent

---

Issue #69 has been merged into this issue.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/348532?v=3" align="left" width="48" height="48"hspace="10"> Comment by tomaszkubacki

---

Any status change for this issue ?

[DartBot]

<img src="https://avatars.githubusercontent.com/u/46275?v=3" align="left" width="48" height="48"hspace="10"> Comment by munificent

---

Sorry, but there's no update here.

We've been primarily focused on client-side Dart applications which (naturally) do not allow native extensions. When the larger Dart team makes a shift towards server-side applications, we'll be able to start looking into this.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/46275?v=3" align="left" width="48" height="48"hspace="10"> Comment by munificent

---

Can you provide some information about any specific timeline?

Unlike many companies, Google isn't very timeline oriented. We try to stay flexible so that we can respond quickly to changing circumstances. This means that we have a rough idea of what we're working on in the next few months, but little beyond that. Even within that shorter time span, we are rarely committed to work on something. Priorities change quickly.

Right now, our focus is getting the transformer and development story solid and fully integrated into the Editor and the other corners of the Dart developer experience. That will keep us busy at least for the next few months as we add features to make transformers more expressive and chase down bugs.

Once that settles down, I expect we'll put a bit more focus on server-side Dart. Right now, it isn't easy to run a command line application exposed by a package, so we intend to fix that.

So far, we haven't had many requests for native library support in pub compared to other things users are asking for, but once other aspects of supporting server-side apps in pub are working smoothly, that will be a natural time to start thinking about native libraries.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/2156198?v=3" align="left" width="48" height="48"hspace="10"> Comment by kasperl

---

Removed this from the Later milestone.
Added Oldschool-Milestone-Later label.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/2156198?v=3" align="left" width="48" height="48"hspace="10"> Comment by kasperl

---

Removed Oldschool-Milestone-Later label.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/444270?v=3" align="left" width="48" height="48"hspace="10"> Comment by seaneagan

---

Adding a use case:

issue dart-lang/sdk#21337

[azenla]

@nex3 If I can get an idea of how you think this should be done, I would be more than happy to do a long term contribution of this. I'm rather convinced that full integration of native extensions support into Pub is something that could seriously impact the number of native extensions out in the wild.

At the bare minimum, we can have pre/post install/upgrade hooks, and maybe put out some community Dart scripts that do all the work for you as far as building extensions.

[nex3]

This is going to take a lot of careful thought to design. My instinct is that something like Ruby's solution is the right way to go—allowing a package with an extension to be uploaded as source that must be compiled, but also allowing users who want to to upload pre-compiled versions for different operating systems. This would involve some pretty serious changes to the server, too, since it violates the one-tarball-per-version invariant.

If you're interested in diving into this, I think the first step would be to write up a proposed design with a lot of detail—basically in the form of a DEP. Then you could iterate on that with me and Bob, and once we're all in agreement about it the coding could start.

[azenla]

@nex3 Sounds good, I'll see what I can do :)

[Scorpiion]

Hi, I have been following this issue a little bit on distance, got reminded of it when I read an article on Python packaging today that linked to what I think is Pythons equivalent feature. I remembered @nex3 mentioning a DEP would be a good start here, I thought the Python PEP could be a good inspiration for the person who takes this on:

• PEP 427 -- The Wheel Binary Package Format 1.0

I have not read in the details how Ruby and Python handles this differently, just thought Pythons PEP was nice reference either way.

[sigurdm]

Closing this very old issue in favour of #3693.