Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[key packages] Pilot Packages #142

Open
Eomm opened this issue Jan 29, 2019 · 31 comments

Comments

Projects
None yet
@Eomm
Copy link
Member

commented Jan 29, 2019

Split 1/3 of #105

We can use to start collaborating with the authors to learn and iterate with.

Edit more info:
to reach the WG goals we need to start to help some pilot packages.
These modules will be helped by this WG adding support tools, documentation, fixes and all things to archive our goals and carry benefit to the package itself.
This WG will improve his guidelines, will learn different requirements and will be able to find patterns and solutions, test all the process and tools that are gonna to be built in order to evangelize the community and let the maintenance of an OSS package will be easier for maintainers.
What this WG ask to the pilot packages is:

  • explain the biggest problems and their priorities (as talk about #113 )
  • give feedback to our improvements/suggestion/PR

Express and all its direct and transient dependencies like pillarjs and jshttp are the most mentioned package to start with.
@wesleytodd has discussed a bit of this possibility on the last Express TC meeting so I think the Express team would be on board to participating.

What to discuss here

  • We need to agree or not to start with Express as a pilot package. If "no" what else?
  • To define if we can and should add more "friendly" pilot packages

After that we could start to get "Directions of Help": the way package maintainers wish to receive help and define the first tasks.

Ex:
Express needs help with repo cleanup, automation, documentation and, user support.

@mhdawson

This comment has been minimized.

Copy link
Member

commented Jan 31, 2019

Starting with Express sounds good to me, given that they are a key part of the ecosystem and it sounds like they are open to working with us.

@wesleytodd

This comment has been minimized.

Copy link
Member

commented Jan 31, 2019

While clearly I am a fan of working with Express, I think we need at least a few others to make sure we have a variety of input. Anyone have other suggestions?

@mcollina

This comment has been minimized.

Copy link
Member

commented Feb 1, 2019

I think https://github.com/mqttjs/MQTT.js and dependencies are another good place to start.

@mhdawson

This comment has been minimized.

Copy link
Member

commented Feb 1, 2019

I agree we need to have several and MQTT sounds like a good one to add to the list.

@Eomm

This comment has been minimized.

Copy link
Member Author

commented Feb 1, 2019

I don't know other friendly packages that could collaborate with us in this pilot phase.

Appling manually the basic criteria we talk about, I think that request could be a candidate: it is widely used and have a lot of issue and PR: it seems they need help to break all that work!

I looked here, there is a nice bubble schema that show the libs per usage, I took the first one with many issue&PR opened. Nothing personal 😁

@wesleytodd

This comment has been minimized.

Copy link
Member

commented Feb 1, 2019

I agree request seems like a good target. @reconbot & @mikeal Does this program seem like something you would be willing to participate in? I know you two are involved in request, but if there is anyone else we should ping please feel free!

@mcollina Are you the primary maintainer for MQTT? If not, are there others we should reach out to to see if they are willing to participate?

@nschonni

This comment has been minimized.

Copy link

commented Feb 1, 2019

Maybe some of the plumbing packages that are depended on for the native packages like node-pre-gyp
I'm hoping to move node-sass to that in the next major version, which would bump it up close to request numbers (which is also a dependency for both)

@mcollina

This comment has been minimized.

Copy link
Member

commented Feb 1, 2019

@wesleytodd I'm the only active one with publish access. I can add more if it is needed, but I'm probably the only one that knows how all the pieces fits together.

@reconbot

This comment has been minimized.

Copy link

commented Feb 2, 2019

I've been doing some bug triage for request, and have landed some patches but haven't yet done a release. The future of that package is a discussion. There is a desire to leave it as is and encourage the use of more modern packages even, however keeping it supported is obviously desirable. I've read the linked issue and the blog post, but I'm not sure what the work would be to be part of this project.

@wesleytodd

This comment has been minimized.

Copy link
Member

commented Feb 2, 2019

I think that is a great stage for this WG to look at. I think we should be taking into consideration the entire life cycle of maintenance, including EOL like you say request is (although I feel like this might come as a shock so some users). That is to say, I don't believe this should disqualify it from consideration here.

@wesleytodd

This comment has been minimized.

Copy link
Member

commented Feb 2, 2019

Also, @reconbot, maybe you will have some input on #139. Does that proposal seem like something which would help signal to users the state the maintainers think the project is in?

@chrkaatz

This comment has been minimized.

Copy link
Contributor

commented Feb 3, 2019

One other candidate might be nedb as the creator of it asked for help himself but I know that the package is not used very much anymore these days.

@mikeal

This comment has been minimized.

Copy link

commented Feb 4, 2019

It’s not clear to me from the thread here and the links exactly what this pilot program means and what the goals are.

When engaging projects it would be great to have a concise write up of:

  • The problems this program is designed to address.
  • What is being asked for from “pilot” projects and what may be expected from projects in the future beyond the pilot.
  • What is being offered by this group to these projects.
@wesleytodd

This comment has been minimized.

Copy link
Member

commented Feb 4, 2019

Great points. @Eomm maybe we could edit the OP and add some of this?

The problems this program is designed to address.

The goals of this WG can be found here.

What is being asked for from “pilot” projects and what may be expected from projects in the future beyond the pilot.

A pilot project would be the first group we try to help. The requirement would be a willingness to try some ideas and provide feedback on how they work.

What is being offered by this group to these projects.

This is in the process of being defined. And the hope is that the pilot projects help us refine and decide on what specific things would be best for us to provide.

@Eomm

This comment has been minimized.

Copy link
Member Author

commented Feb 4, 2019

@wesleytodd edited 👍
I hope to have clarified the benefit and the effort in play.

@Eomm

This comment has been minimized.

Copy link
Member Author

commented Feb 10, 2019

Hi, I would like to recap the topic to speed up the meeting tomorrow:

Candidate Approved by candidate Main Contact
express yes @wesleytodd
mqtt yes @mcollina
request need information @mikeal
plumbing packages that are depended on for the native packages - -
node-sass - @nschonni
nedb yes @chrkaatz

I hope I have correctly summarized 😁

@Eomm

This comment has been minimized.

Copy link
Member Author

commented Feb 11, 2019

It’s not clear to me from the thread here and the links exactly what this pilot program means and what the goals are.

When engaging projects it would be great to have a concise write up of:

  • The problems this program is designed to address.
  • What is being asked for from “pilot” projects and what may be expected from projects in the future beyond the pilot.
  • What is being offered by this group to these projects.

Hi @mikeal, to reach the WG goals we need to start to help some pilot packages.
These modules will be helped by this WG adding support tools, documentation, fixes and all things to archive our goals and carry benefit to the package itself.
This WG will improve his guidelines, will learn different requirements and will be able to find patterns and solutions, test all the process and tools that are gonna to be built in order to evangelize the community and let the maintenance of an OSS package will be easier for maintainers.
What this WG ask to the pilot packages is:

  • explain the biggest problems and their priorities (as talk about #113 )
  • give feedback to our improvements/suggestion/PR

I hope to be exhaustive, let us know what do you think about it
Thank you

@JamesMGreene

This comment has been minimized.

Copy link
Contributor

commented Feb 20, 2019

I would also be interested in being a maintainer of NeDB. I try to respond to most of the newly opened issues in the existing repository, and have also forked the project as NestDB to implement some requested new features, bug fixes, and performance improvements that its sole maintainer was not keen on merging or able to give time to (a problem with maintainership that so many of us know all too well).

I've done my best to track the relationships of my changes to the original NeDB issues that most of them originated from in the hopes that, someday, they might make it back into the NeDB core codebase.

There have also been other forks of NeDB, as nedb2, nedb3, nedbhq/nedb-core, and even the now unrecognizable TeDB (TypeScript-based), so the popularity of NeDB -- and its consumers seeking for actual maintenance -- definitely has no doubt in my mind.

@mhdawson

This comment has been minimized.

Copy link
Member

commented Feb 25, 2019

@wesleytodd We touched on this briefly and we wondered if we are ready to take the next steps with either express or mqtt as we continue to build out the pilot list?

@wesleytodd

This comment has been minimized.

Copy link
Member

commented Mar 1, 2019

Hey, yes any time. Do we have a concrete list of the first steps? I seem to remember an issue regarding this, but I have so many unread messages at the moment (👶 time). The start was a survey right? If so I can get that posted up in the express repo so we can fill it in.

@wesleytodd

This comment has been minimized.

Copy link
Member

commented Mar 1, 2019

Ok, I posted this on the express discussion repo. I will hopefully get other contributor feedback and start working on those answers soon.

expressjs/discussions#77

@Eomm

This comment has been minimized.

Copy link
Member Author

commented Mar 4, 2019

I've done my best to track the relationships of my changes to the original NeDB issues that most of them originated from in the hopes that, someday, they might make it back into the NeDB core codebase.

@JamesMGreene could you ping the maintainer of NeDB to join the discussion?
Because I think we can't help that package if the maintainer can't collaborate for whatever reason.

@Eomm

This comment has been minimized.

Copy link
Member Author

commented Apr 10, 2019

In last meeting we talked to find more packages

We could start digging from this list:

The top 20, to save you a click:
debug
kind-of
supports-color
readable-stream
source-map
yargs
camelcase
minimist
strip-ansi
chalk
commander
@types/node
punycode
ansi-regex
ms
glob
define-property
semver
async
string_decoder

https://twitter.com/seldo/status/1115725616433602560?s=19

@mcollina

This comment has been minimized.

Copy link
Member

commented Apr 12, 2019

https://docs.google.com/spreadsheets/d/1lZDNYsLntwD2q9XaTw-XLuG0VpHH6cOV-Uoa7Y1aTSM/edit#gid=1745448509 contains a list of the top 1000 most downloaded modules.
It would be good to do some sort of data analysis of the dependants and dependencies of these, as I guess there are a lot of overlaps and macro-trends.

@dominykas

This comment has been minimized.

Copy link
Contributor

commented Apr 13, 2019

Here's some percentiles on last update age of top 1000 packages:

percentile last commit last publish last update to .travis.yml
5% 1 days old 7 days old 10 days old
10% 4 days old 11 days old 34 days old
15% 5 days old 18 days old 56 days old
20% 5 days old 37 days old 92 days old
25% 10 days old 58 days old 126 days old
30% 19 days old 88 days old 129 days old
35% 33 days old 131 days old 129 days old
40% 57 days old 157 days old 160 days old
45% 81 days old 200 days old 196 days old
50% 119 days old 246 days old 259 days old
55% 150 days old 300 days old 324 days old
60% 194 days old 369 days old 387 days old
65% 256 days old 438 days old 501 days old
70% 328 days old 559 days old 639 days old
75% 418 days old 669 days old 723 days old
80% 563 days old 777 days old 891 days old
85% 706 days old 944 days old 1036 days old
90% 887 days old 1098 days old 1214 days old
95% 1194 days old 1372 days old 1501 days old
100% 2201 days old 2792 days old 2620 days old

This means that roughly half of them have not been published since last node LTS... Now, that in itself does not mean they weren't tested in that LTS (even it travis.yml wasn't update, although that's a good indicator), but it gives perspective on how much of just the top 1000 is either "abandoned" or "done".

What kind of dependents / dependencies data did you want to see @mcollina?

@dominykas

This comment has been minimized.

Copy link
Contributor

commented Apr 13, 2019

And here's some stats from versions extracted from .travis.yml (didn't bother cleaning those up, just yet):

node version count in travis.yml
version count
1 3
2 3
3 3
4 269
5 128
6 507
7 112
8 459
9 85
10 381
11 53
0.1 6
0.10 257
0.10.12 1
0.11 29
0.12 244
0.12.0 1
0.4 1
0.6 31
0.8 98
0.8.6 1
0.9 2
1.7.1 1
1.8 37
10.0 2
10.11 2
10.12 5
10.13 2
10.14 3
10.15 25
10.2 1
10.4 1
10.6 1
10.7 2
10.8 1
11.0 2
11.10 8
11.12 1
11.13 1
11.14 1
11.2 1
11.4 2
11.6 5
11.7 3
11.8 1
11.9 2
2.5 37
3.3 37
4.0 6
4.0.0 3
4.1 4
4.2 4
4.3 1
4.4 3
4.4.0 1
4.5 2
4.7 1
4.8 7
4.9 48
5.0 2
5.0.0 1
5.1 1
5.10 3
5.11 1
5.12 55
5.2 1
5.3 1
5.4 1
5.5 1
5.6 2
5.7 2
5.8 1
5.9 1
6.0 4
6.0.0 1
6.1 1
6.10 1
6.11 1
6.12 2
6.13 2
6.14 18
6.15 4
6.16 21
6.17 5
6.2 1
6.3 1
6.4 1
6.5 2
6.6 1
6.9 1
7.10 53
7.7 1
8.0 1
8.11 9
8.12 8
8.13 2
8.14 3
8.15 26
8.6 1
8.9 5
9.11 46
9.3 1
9.5 2
iojs 50
iojs-1 1
iojs-2 1
iojs-3 1
iojs-v1.0 1
iojs-v1.1 1
iojs-v1.2 1
iojs-v1.3 1
iojs-v1.4 1
iojs-v1.5 1
iojs-v1.6 1
iojs-v1.7 1
iojs-v1.8 17
iojs-v2.0 1
iojs-v2.1 1
iojs-v2.2 1
iojs-v2.3 1
iojs-v2.4 1
iojs-v2.5 17
iojs-v3.0 1
iojs-v3.1 1
iojs-v3.2 1
iojs-v3.3 17
lts/* 18
node 196
stable 52
v0.12 1
v10.* 1
v4 3
v5 2
v6 2
v6.* 1
v8.* 1
@mcollina

This comment has been minimized.

Copy link
Member

commented Apr 14, 2019

Scavenging that data is very interesting.
I’d like to see if there are some dependency relationship. For example, we know that express uses debug. This info is helpful because we can take some macro-family.

Another data that would be useful are the number of issues and prs open on those repo to gauge activity.

@ljharb

This comment has been minimized.

Copy link
Contributor

commented Apr 14, 2019

tbh what I’d really like to see is top packages on npm with download counts of dependents and devDependents subtracted out

@dominykas

This comment has been minimized.

Copy link
Contributor

commented Apr 20, 2019

This info is helpful because we can take some macro-family.

Sorry, not sure I'm following. You'd like to have the top1k somehow clustered into a group of deps that belong to a single dependent or smth? Or to put it differently - try to find a group that would likely get downloaded together? I can try to look into the dependency links in the top1k, but we don't have an easy, publically available method to go beyond that...

Another data that would be useful are the number of issues and prs open on those repo to gauge activity.

So getting these numbers is easy, but gaining insight is hard... Do you have any specific ideas or questions in mind?

A number of open issues on its own does not say anything (some maintainers close everything, others let issues linger, esp. support ones), the issue/pr rate over time is also meaningless - low activity could just be an indicator of done-ness and stability and I'm not sure how that is helpful?


What problems are we looking to solve with this? What problems can be discovered? I'm usually a fan of looking for data that can help make decisions, e.g. the numbers I posted can be used to determine which packages need some touching up to at least have CI run with newer node versions. Are there any other specific things we could be looking for?

@Eomm

This comment has been minimized.

Copy link
Member Author

commented Apr 23, 2019

Are there any other specific things we could be looking for?

I think that this question is more related to this issue #143 (comment) and here (pilot packages) I think we are looking for collaboration mainly from the maintainers of the modules and download numbers.

Adding issues/PR numbers will be useful for sure, but we could proceed step by step building the tool that we will use to list the modules to find them out so.. who are those packages in the >=50% from last travis update?

@dominykas

This comment has been minimized.

Copy link
Contributor

commented Apr 24, 2019

who are those packages in the >=50% from last travis update?

I'll open a separate issue with the list shortly and we can take it from there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.