How to resolve missed platform classifications?

[isoos]

We have a couple of interesting cases open:

• miscategorized dart_to_js_script_rewriter #598, package:polymerize is confusingly marked as server #634 - build tools for apps targeting the web, classified as server.
• json_serializer says it's only for server? probably works with Flutter? #790, Platform analysis is wrong if package has bin-directory pana#188 build tools for all kind of apps, classified as server
• Packages which depend on NodeJS runtime are categorized under Web #709 "server" library for NodeJs apps, to be used with dart2js, classified as web

The common theme in all of these is the difference between the use of the library and the target platform of the app that is built using it.

I can see a couple of ways to go ahead:

1. No action in the classification.

There are packages that do provide build and code generation utilities, and are not mis-categorized (e.g. pwa generates code, but all of that lives in bin/, so there is no hint that it would be a server package).

If we identify these patterns, and give guidance to package authors, eventually most of these will be corrected.

Related questions:

• What is the ideal pattern (besides hiding the dart:io code in bin/)?
• Shall we add a suggestion for it in pana (e.g. detecting somebody relies on source_gen)?

2. Metadata from pubspec.yaml.

Keywords (#368), categories (#367) or a property for platforms could be coming from pubspec.yaml, and the UI could display it along the ones we are detecting.

Related questions:

• How to display them if they are not the same? (colors, format, order, icon.. ?)
• Which of it takes preference (e.g. in search)?
• Should this affect the score or ranking in any way?

3. Hand-coded exception in the pana analysis.

We could assume that if a library uses certain helper libraries (through transitive dependencies), than it is a build-utility or a nodejs library. If/when dart:io is used only through them, the package could be classified as generic, unless other API use restrict it.

Related questions:

• Is it the case really that if something is a build tool, it can be applied to every platform? I can't think of a counter-example, but there may be one.
• Shall we put NodeJs support in a different category? It is an outlier compared to the build tools.

4. New platform for build utilities.

We could also introduce a new platform, that is exclusively build- or other toolchain-related utilities. These packages could be listed for Flutter or web too, but would be bearing the tool or build or something similar platform label.

Related questions:

• Is it really a platform, or a sub-platform / category?
• NodeJs is still an outlier here, same question for a different category.

[isoos]

/cc @kevmoo @mkustermann @kwalrath @matanlurey @jakemac53 @natebosch @mit-mit @filiph @sethladd @nex3 @MikeMitterer @zoechi @pulyaevskiy @mbullington

[jakemac53]

My feeling on this is we allow package authors to explicitly state their platforms via pubspec.yaml and that would override whatever auto-detection we do by default.

The impact of bad actors is minimal imo, and if it really became an issue we could add some sort of mechanism that would result in a negative impact to the packages score if they improperly declare their platform support.

Fwiw - this is what we do in bazel. We have a platforms attribute on dart_library that is contagious - you can't depend on packages with incompatible platforms and your platforms (if not specified) become the intersection of the platforms from the packages you depend on. We do not attempt any auto-detection logic for platforms by default, although there are some tools which try to do that for basic cases.

[natebosch]

The impact of bad actors is minimal imo

I think mistakes are more likely than bad intent. I think we need to resolve the open questions around what it means to publish a "tool" which can be used on a web or vm or flutter project. Today that shows up as vm/flutter because that's where it runs, but that doesn't mean that web projects won't be interested in discovering it.

The risk, I think, is that someone publishing a "tool" meant to help with web stuff will tag their package as web platform, but someone writing web code can't safely import anything from it.

[jakemac53]

I think mistakes are more likely than bad intent

I agree, but I also think that these issues will resolve themselves quickly. For any package with sufficient visibility somebody will notice the error and file an issue or maybe even send pull requests to update the field since it would be trivial to do so even entirely through github itself.

[matanlurey]

Woooo. Long thread.

@jakemac53:

My feeling on this is we allow package authors to explicitly state their platforms

👍. Let's just let folks do what they wanted to do for a long time. We already have the concept of platforms in the Bazel rules_dart (dart-archive/rules_dart@6bc0a57, compliments of @grouma), having a similar concept in pubspec.yaml can only be useful, even if we don't act on it for anything other than tagging today.

(Ultimately we will need something like "targets", but let's punt on that conversation)

---

My 2 cents: The easiest thing to do without needing consensus is add the ability to label packages as "tools", i.e. neither for the server, web, or flutter specifically (though the role of the tool might lead itself to not be useful on certain platforms).

One way to do that might be detecting executables: ..., which sort of leads that the package is meant to be used via pub run or pub global activate. This might mis-classify some packages like dart_style or build_runner, which does both, but those tend to be in the minority, right?

@isoos:

Shall we put NodeJs support in a different category?

I don't know that this is important today. NodeJs specific packages are <0.1% of all packages, and we don't formally support this consistently anyway. If we ever add the concept of "platforms", I'd be supportive of adding "node" as a platform. Until then, I think we should punt.

@natebosch:

The risk, I think, is that someone publishing a "tool" meant to help with web stuff will tag their package as web platform, but someone writing web code can't safely import anything from it.

This is a bug in our development story, not pub-dartlang-dart. The analyzer should know you are building a web project and prevent you from importing packages that won't build, instead of being VM centric.

[kevmoo]

+1 to the idea of labeling something as a tool

…

On Fri, Dec 22, 2017, 12:13 Matan Lurey ***@***.***> wrote: Woooo. Long thread. @jakemac53 <https://github.com/jakemac53>: My feeling on this is we allow package authors to explicitly state their platforms 👍. Let's just let folks do what they wanted to do for a long time. We already have the concept of platforms in the Bazel rules_dart ( ***@***.*** <dart-archive/rules_dart@6bc0a57>, compliments of @grouma <https://github.com/grouma>), having a similar concept in pubspec.yaml can only be useful, even if we don't act on it for anything other than tagging today. (Ultimately we will need something like "targets", but let's punt on that conversation) ------------------------------ My 2 cents: The easiest thing to do without needing consensus is add the ability to label packages as "tools", i.e. neither for the server, web, or flutter specifically (though the role of the tool might lead itself to not be useful on certain platforms). One way to do that *might* be detecting executables: ..., which sort of leads that the package is meant to be used via pub run or pub global activate. This might mis-classify some packages like dart_style or build_runner, which does both, but those tend to be in the minority, right? @isoos <https://github.com/isoos>: Shall we put NodeJs support in a different category? I don't know that this is important today. NodeJs specific packages are <0.1% of all packages, and we don't formally support this consistently anyway. If we ever add the concept of "platforms", I'd be supportive of adding "node" as a platform. Until then, I think we should punt. @natebosch <https://github.com/natebosch>: The risk, I think, is that someone publishing a "tool" meant to help with web stuff will tag their package as web platform, but someone writing web code can't safely import anything from it. This is a bug in our development story, not pub-dartlang-dart. The analyzer should know you are building a web project and prevent you from importing packages that won't build, instead of being VM centric. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#858 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABCisE6XQLn7VtE-YUptKY2R03CE1tLks5tDA1MgaJpZM4RLBCm> .

[pulyaevskiy]

If we ever add the concept of "platforms", I'd be supportive of adding "node" as a platform.

What are the chances of getting there?

If I understand it correctly at least test package already supports node as a platform (we can pub run test -p node).
It also seems like the language itself is aware of it, looking at this file in Dart SASS:
https://github.com/sass/dart-sass/blob/master/lib/src/io.dart
(though I'm really not sure what's the status of this feature, it doesn't seem to be documented anywhere).

Regardless, for the NodeJS packages, I feel like categorizing them under "server" would be a good first step.

I had a suggestion to check if there is a dependency on node_preamble which seems to be used in many NodeJS-related packages.

One more things we can try is to change classification rules for "web" to require dart:html in dependencies. As I understood from #709 it currently only tracks dart:js and dart:js_util which both can be used on the server side.

[matanlurey]

If we ever add the concept of "platforms", I'd be supportive of adding "node" as a platform.

What are the chances of getting there?

I wouldn't count on it in the immediate future. There's just so much more to do right now.

[mbullington]

I'm also on-board with being able to classify a package's platforms and it's purpose as a tool in the pubspec. 👍 Since being a tool really isn't a platform, there could be a platforms list and a tool boolean?

If you choose to implement this in the future (going off of @matanlurey , this is reasonably a lower priority than other things), I think "node" should be a valid platform, especially if it's already recognized by the language as shown by @pulyaevskiy .

Even though these packages are an incredibly small part of the pub ecosystem, in the issue's spirit of increasing overall clarity I think a "node" platform is the best option. Labeling these packages as "web" has definite problems, but I feel "server" may mislead new users into associating it with the standalone Dart VM.

In the mean time, from my knowledge as node_preamble maintainer, most if not all of the libraries that target Node.js use the preamble. An issue I've seen is that some packages copy and modify it slightly. However, this is really negligible, and on occasion I try to reach out to these devs to see how we can work their changes upstream. I wouldn't see much issue with checking the immediate dependency tree for node_preamble.

[matanlurey]

Small correction, Dart doesn't know anything about node.

Dart SASS just uses a "definition variable" for this purpose at compile-time:
https://github.com/sass/dart-sass/blob/master/tool/grind.dart#L78

You can use any arbitrary string.

[pulyaevskiy]

Just for educational purposes are there any docs about this feature (conditional export?) and what’s the status of it (experimental, stable, deprecated)?

I’d definitely like to use it in node_interop so that it can be used to swap io implementation from Dart vm to node and vice versa.

[matanlurey]

It is an experimental language feature without official support at this time. There are complications in implementing it in AOT-environments that require incremental recompilation, so there is no guarantee the current implementation will officially ship.

[isoos]

One more things we can try is to change classification rules for "web" to require dart:html in dependencies. As I understood from #709 it currently only tracks dart:js and dart:js_util which both can be used on the server side.

pana tracks many packages to associate something as web-only. package:js exports dart:js, and with that, we classify it as web-only.

[pulyaevskiy]

@isoos
In this case it would mean removing dart:js and dart:js_util from the _webPackages list. Not sure if this would cause issues with any web-only packages, but in general if a package provides js bindings it doesn’t mean web-only necessarily.

Maybe there needs to be a separate js-only category.

@matanlurey
Thanks for the details. Hope it finds its way to stable release in some form.

[isoos]

@pulyaevskiy: Maybe there needs to be a separate js-only category.

This is something to consider, thanks for bringing it up.