Integrate npm

[chadwhitacre]

---

✈️ This is the flight deck for the Integrate npm project. ✈️

---

Current open-source crowdfunding options (Kickstarter, Patreon, Gratipay, OpenCollective, etc.) are consumer-grade. Our hunch is that a business-grade product with better aggregation can better serve the companies that want to pay for open source, because companies use hundreds or thousands of open source packages, not just a few.

Picking up from #4135 (comment) and gratipay/inside.gratipay.com#852 (comment) ...

For wider context see:

• The fundamental idea is to get a viral loop going again like we had under Gittipay 1.0.
• The burn I feel

JavaScript is the most popular language in open source and npm is the most popular package manager for JavaScript. A good first concrete step towards helping companies pay for open source (#4135), therefore, will be to add the ability to pay for any package on npm. Once we have npm deployed, we will have enough experience to inform a partnership with Libraries.io for the rest of the package managers.

Target

Our goal is to announce this feature in my lightning talk on Thursday, October 26 at Red Hat's All Things Open conference (gratipay/inside.gratipay.com#757).

Our goal is to incrementally improve this feature throughout the first half of 2017, with an eye towards OSCON and $ustain in May.

Package names to test with

From #4135 (comment):

http://localhost:8537/on/npm/async/
http://localhost:8537/on/npm/iframe-resizer/
http://localhost:8537/on/npm/mongoose/
http://localhost:8537/on/npm/nodemon/
http://localhost:8537/on/npm/react/
http://localhost:8537/on/npm/react-helmet/
http://localhost:8537/on/npm/react-modal/
http://localhost:8537/on/npm/react-redux/
http://localhost:8537/on/npm/react-router/
http://localhost:8537/on/npm/react-router-redux/
http://localhost:8537/on/npm/redux/
http://localhost:8537/on/npm/redux-thunk/
http://localhost:8537/on/npm/webpack/

Todo

Prerequisites

• Land the Decouple milestone—Relax Open Work Requirement inside.gratipay.com#432
• Finish Project Basics—https://github.com/gratipay/gratipay.com/milestone/18

Checkpoint 1: Inert /on/npm/foo/ Pages

• wrap marky-markdown (Wrap marky-markdown #4154)
• load up npm (Load up npm #4153, Fix npm sync bug with empty descriptions #4158)
• load up npm readmes (Here's a first crack at syncing readmes #4159, Install self in requirements.txt #4160, Avoid ijson if we can help it #4161, Add a package.json #4162, Modify first h1/p just like npm does #4164, Fix npm sync regression #4176, Improve sync_npm code #4179, Fix regression in sync-npm cli #4180, Delete 404 (unpublished) packages #4181, Fix bugs in fetch_readmes #4182)
• stop storing readmes after all (Rip out READMEs after all #4211)
• add inert (can't actually give to them) /on/npm/foo pages (Stub out /on/npm/foo/ pages #4212)

Checkpoint 2: Giving to Packages

• link packages to teams (Stub out Package model #4155)
• opt in or out from /on/npm/foo/ pages via email verification
• add payment widget on /on/npm/foo/ pages

Checkpoint 3: Easy Sign-up

• bulk manage payments from /on/npm/ pages

Nice to Have

• move search into {% content %} (leaving social network jump in sidebar)
• merge current result types together (~user via username, ~user via profile statement)
• add Teams to local index for inclusion in results
• mix npm packages into search (Add npm to /search #4147)
• pledging to as-yet-unclaimed packages
• load up npm incrementally (reticket from Load up npm #4153)
• deal intelligently with unpublished packages (reticket from Integrate npm #4148 (comment))
• we seem to get two H1s on https://www.npmjs.com/package/resize-img
• syntax highlight

Promotion

• update the "★ announcement ★" link (update the "★ announcement ★" link #4149)
• Publish a blog post.
• Promote the heck out of it!

---

✈️ This is the flight deck for the Integrate npm project. ✈️

[chadwhitacre]

@mattbk You wanna draft a blog post?

[chadwhitacre]

@aandis How about I start hacking on this today and tomorrow, and I will leave notes for you over the weekend based on how far I get. Ya?

[chadwhitacre]

I've started a task list in the ticket description.

[aandis]

@aandis How about I start hacking on this today and tomorrow, and I will leave notes for you over the weekend based on how far I get. Ya?

Yep. That works. A quickstart on #4147 will help in implementing the exact functionality you have in mind.

[chadwhitacre]

@aandis What's your availability? I was assuming Saturday and Sunday would be the time you could work on this but maybe I'm wrong?

[aandis]

@whit537 I am free over weekend, yes. I'll be keeping a close eye on this ticket for the next couple of days.

[chadwhitacre]

I've reached out to npm, Inc. in private email to let them know about our plans and invite them to give feedback and/or cross-promote with us next week.

[chadwhitacre]

Is there a rate limit on npm registry API requests? Can it be up'd with authentication? I find no docs on this.

[chadwhitacre]

We should think about possibly merging existing teams into these.

[chadwhitacre]

The code that npm uses to talk to the registry.

https://github.com/npm/npm-registry-client

[aandis]

http://nipstr.com

[aandis]

@whit537 I am not seeing any official well documented api for searching packages over npm. Best I could find is https://github.com/wires/npm-keywordsearch through this.

[chadwhitacre]

@aandis Let's discuss search on #4147.

[chadwhitacre]

Hashed this out a bit with @kaguillera. The problem we're solving is companies funding open source software. That's a market. The two sides are companies (buyers) and projects (sellers). We need to make it as easy as possible for companies to give to open source software; that's #4135. This is the other side, making it as easy as possible for developers to receive payments for open source.

/on/npm/

One thing we realized is that a single developer will have multiple projects. "Easy as possible" means letting them configure multiple projects at once. Doug Wilson is maintaining 103 packages. We can't make him go through 103 separate verification workflows. He needs one worfklow.

We're thinking that that workflow should live at /on/npm/.

-anon—Sign in to configure payments for your npm packages.
-auth—Verify the email you use on npm to configure payments your npm packages.
-no packages—We didn't find any npm packages with alice@example.com as an author or maintainer.
-packages—We found 103 npm packages with alice@example.com as an author or maintainer.

We should account for the case where another ~user has already claimed the package.

Added to todo. ^^^ Bring this over to a PR when the time comes.

Going all in?

The other thing we realized is that the 400,000 packages on npm dwarf the 200 projects currently on Gratipay. We should consider redoing the homepage to contain a search box (for givers and the curious) and a link to /on/npm/ (for developers ready to start receiving payments). We don't need to list "our Teams" on the homepage. With this change, "our Teams" instantly explodes to the size of npm.

[chadwhitacre]

Oh! And we should bring back pledging: givers should be able to set up payment instructions to projects before the projects opt-in. This is where opt-out starts to matter, because it prevents pledging.

[chadwhitacre]

We're getting back to the same dynamic as Gittipay 1.0, but with a focus on packages on open source software repositories (→ Teams [Projects]) rather than users of social media networks (→ ~users).

[chadwhitacre]

Also, it seems unlikely that we'll get this all done before next Thursday. Let's do what we can ... and have fun doing it! :-)

[chadwhitacre]

Alright, @aandis, I've stubbed out a few PRs and updated the todo in the description here. Hopefully that gives you something to go on before I get online tomorrow!

!m @aandis

[chadwhitacre]

@aandis Can we use this thread to communicate and coordinate about what specifically we're each working on? I'm seeing work from you on #4154 and #4155 (!m @aandis). What are you working on now? Do you think the todo on this ticket is right and in the right order or do we need to modify it?

I'm going to start at the top of the todo with #4117 and then move down to join you on the actual coding bits. :-)

Let's do this! 💃 🌻

[aandis]

@whit537 I am gonna wrap up #4154 and then work on #4153 and #4155 to make #4151 possible. I think we should move pledging out of #4155 into it's own ticket and focus on mvp to be able to give to a package. Pledging is a nice to have feature but only when the groundwork is laid.

[chadwhitacre]

@aandis Cool. Strictly speaking, #4155 isn't necessary for #4151. In other words, we could have /on/npm/foo/ pages without any possibility of giving to them. Is it worth batching #4154 #4153 and #4151 as our first checkpoint, and tackle #4155 after that first checkpoint is deployed?

[chadwhitacre]

{#4154, #4153, #4151} feels like something we could potentially get deployed before the end of the day today. Deployment itself will take some effort because of the need to spin up a worker dyno—something we haven't done before—to handle the continual npm syncing. Sure, inert /on/npm/foo/ pages won't be much to brag about, but it'll at least give us a feeling of win. :)

Thoughts?

[aandis]

Sounds good. :)

[chadwhitacre]

@aandis Cool. I've updated the todo.

[chadwhitacre]

Discussing in slack.

[aandis]

January or February. Or March.

😄

[chadwhitacre]

Ok. I will bless this technical debt with my fairy wand but request a deadline for fixing it.
12 weeks after feature launch

@JessaWitzel at slack

[chadwhitacre]

We have reached Checkpoint 1: Inert /on/npm/foo/ Pages! 💃

https://gratipay.com/on/npm/react-router/

[nobodxbodon]

Anything I can do to accelarate this? It seems top priority for now.

Some questions:

• is NPM cool with this, and are they willing to partner/coordinate in any way?
• any workload estimate and roadmap with timeline?
• who are main developers and any dividing of tasks?
• any blockers?

[chadwhitacre]

Thanks for bumping this, @nobodxbodon! As mentioned in slack, I hoping to spec this out this week while also bringing Relax Open Work Requirement in for landing (this was blocked on that).

[chadwhitacre]

is NPM cool with this, and are they willing to partner/coordinate in any way?

I emailed them and didn't hear back. I think if we get some traction with this, that will be the time to reapproach a conversation with them.

any workload estimate and roadmap with timeline?

Last month I guesstimated (one, two) that the two projects together would take six weeks of calendar time. It's now been four and a half weeks and we're not done with either yet. We're likely to finish the first by the end of this week (Week 5), with next week being Week 1 on Integrate npm. It seems unlikely to take a week. ;-) I was off by a factor of 2.5 on the Relax Open Work time estimate (I figured two out of the six). That suggests 10 weeks of calendar time for Integrate npm.

Roadmap and further estimation tbd when I can spec this out.

who are main developers and any dividing of tasks?

Me and maybe @aandis? Anybody else wanna volunteer? :-) Division of tasks tbd.

any blockers?

https://github.com/orgs/gratipay/projects/5?fullscreen=true

[chadwhitacre]

I've put both projects on the calendar, with an initial target date of March 17 for Integrate npm.

[chadwhitacre]

Honestly this is probably going to go into April or May, but we can take that as it comes. :-)

[aandis]

I'd really like to see this happening on gratipay so I am ready to help as much as I can. We need a development plan so that things are moving along.

[mattbk]

I've put both projects on the calendar

This would be more useful in Gantt chart format, IMO. The calendar is cool for events, but in terms of lining up work, it's just a bunch of blue bars to me.

ETA: Don't take that as a personal feature request. If I want it bad enough I'll figure it out.

[chadwhitacre]

Conversation about product developement and wire-framing, might try https://gomockingbird.com/.

[chadwhitacre]

Closing per gratipay/inside.gratipay.com#987 (comment).