Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Publish sourcecred on npm #1232

Open
decentralion opened this issue Jul 16, 2019 · 9 comments

Comments

@decentralion
Copy link
Member

commented Jul 16, 2019

We should make depending on SourceCred a lot easier, by publishing it on npm. Users should be able to install the sourcecred CLI; once that's widely available, we can start making it easier to use (e.g. use the sourcecred CLI to create and update an instance, rather than needing build_static_site.sh).

Ideally, we'll also make it possible to take code dependencies on pieces of the SourceCred API--e.g. importing the SourceCred graph and cred calculator.

I'm not sure how difficult this will be, or what the best practices are for publishing webpack-ified tools to npm.

Also, we can consider whether we want to publish everything to sourcecred, or use namespacing like @sourcecred/cli, @sourcecred/core. (Would that also imply splitting up sourcecred/sourcecred into sub repos?)

cc @Beanow and @wchargin

@Beanow

This comment has been minimized.

Copy link
Member

commented Jul 16, 2019

I believe namespacing would make sense. Especially considering various plugin systems and library dependencies are something you may want. But on top of that a meta-package would be cool.

From the typical user / getting started perspective

From a users perspective who just wants sourcecred added to their project a sourcecred meta package may be useful. This could be a 'typical' use-case including perhaps many dependencies like @sourcecred/core, @sourcecred/cli, @sourcecred/widgets, @sourcecred/themes, @sourcecred/github, @sourcecred/discourse, etc.

On top of that, I would ideally use a configuration file for everything. Weights, repos to load, project definitions, list of widgets to generate, output directories, caching directories, plugins to use... something like .sourcecred.yaml.

Along with that a workflow like npm i -g sourcecred to install, sourcecred init to generate a sample config file and sourcecred update to load, calculate and regenerate everything. Maybe sourcecred preview to have a look at it from localhost.

Many tools now use a similar approach. Flow, hugo, prettier, eslint. All with minimal cli arguments and a lot of declarative config.

A bit more advanced setup

When messing around with manual deployments and multiple versions of npm i -g sourcecred starts causing trouble. You'll probably want something a bit more contained and automated.

You could use a package.json just to declare exactly what selection of sourcecred packages you're using. Along with your .sourcecred.yaml configuration file. Maybe some cache or static files committed that are referenced by your config. And a CI config file if you're automating it.

An idea of such a package.json.

{
  "name": "my-sourcecred-updater",
  "private": true,
  "dependencies": {
    "@sourcecred/core": "^1.1.5",
    "@sourcecred/cli": "^1.1.0",
    "@sourcecred/widgets": "^1.2.0-beta.1",
    "@sourcecred/github": "^1.1.2"
  },
  "scripts": {
    "sourcecred": "sourcecred update"
  }
}

Something along those lines would allow more specific version constraints and a pretty easy workflow like: npm i && npm run sourcecred or yarn && yarn sourcecred.

Build and export to NPM

Some ideas about that can be drawn from here:
https://medium.com/netscape/shipping-flowtype-definitions-in-npm-packages-c987917efb65
https://github.com/ryyppy/medium-expose-flow-types

@decentralion

This comment has been minimized.

Copy link
Member Author

commented Jul 17, 2019

I believe namespacing would make sense. Especially considering various plugin systems and library dependencies are something you may want. But on top of that a meta-package would be cool.

From the typical user / getting started perspective

From a users perspective who just wants sourcecred added to their project a sourcecred meta package may be useful. This could be a 'typical' use-case including perhaps many dependencies like @sourcecred/core, @sourcecred/cli, @sourcecred/widgets, @sourcecred/themes, @sourcecred/github, @sourcecred/discourse, etc.

I love the idea of a meta package. Refactoring SourceCred (more accurately, the
SourceCred build) to the point where we can implement a clean meta-package
might take a while. But if we are willing to hack, I think we could make a meta
package quite quickly. We could have a meta package which (for now) clones
SourceCred from source, does yarn install and yarn backend, and then is
ready for usage?

Then, for now, the meta package can act as an interface for the existing
SourceCred workflows, and we can continue to iterate on it from there.
Automating things like setting up a SourceCred directory, setting the
GitHub token, etc, will make things a lot easier.

Here's a question for you: do we want a monorepo or multiple repos? From a conceptual
standpoint I like the idea of multirepos. It will allow us to have faster and simpler builds
in individual projects. Just as an example, one PITA in developing in sourcecred is that
you need to install better-sqlite3 and its dependencies, for the benefit of the GitHub plugin.
However, only a small fraction of our development work is focused on that plugin. So our
builds (and new dev onboarding) could both be much simpler if the GitHub plugin were in its
own repo. Also, using multirepos will make it easier for us to do interesting cred analysis (for sourcecred itself), namely giving cred on a per-repo basis.

However, splitting into a multirepo will also create a lot of coordination
costs for synchronizing updates across the repos. I notice for example that
Babel uses a monorepo:

Juggling a multimodule project over multiple repos is like trying to teach a newborn baby how to ride a bike. --the babel docs

Of immediate import: do we want to build the meta tool in sourcecred/sourcecred or
sourcecred/meta?

If we do it in sourcecred/sourcecred, it will probably be easier/more convenient to get it up
and running.

If doing it in sourcecred/meta, it will likely take longer but put more pressure on us to invest
in making sourcecred easy to depend on.

I'm leaning towards sourcecred/meta.

On top of that, I would ideally use a configuration file for everything. Weights, repos to load, project definitions, list of widgets to generate, output directories, caching directories, plugins to use... something like .sourcecred.yaml.

Along with that a workflow like npm i -g sourcecred to install, sourcecred init to generate a sample config file and sourcecred update to load, calculate and regenerate everything. Maybe sourcecred preview to have a look at it from localhost.

This sounds great to me. Among other things, I want
there to be a sourcecred init step so that we can
prompt users to get their GitHub token, and can set up
a persistent $SOURCECRED_DIRECTORY for users along
with information about how to clear it. This will
improve the dev ergonomics.

I'm currently re-writing the sourcecred load command, and as I do so, I'm
following @wchargin's advice in #945 about making a sourcecred API. Specifically,
I've implemented a new module, analysis/load.js] which has all of the behavior, and takes a data structure with all the configuration information.
Then cli/load.js is just a basic wrapper for calling that API. See #1233 for details.

This should make it straightforward to run SourceCred from configuration files
as you're suggesting.

A bit more advanced setup

When messing around with manual deployments and multiple versions of npm i -g sourcecred starts causing trouble. You'll probably want something a bit more contained and automated.

Yep, love this idea. We should never tell people to npm install -g sourcecred. Both because we are changing our data formats pretty quickly, so mixing versions will cause a lot of frustration, and also because new versions may change the cred distribution, and people's cred should not change unexpectedly. (Once we hit 1.0, any change that would change users' cred should be considered a breaking change.)

You could use a package.json just to declare exactly what selection of sourcecred packages you're using. Along with your .sourcecred.yaml configuration file. Maybe some cache or static files committed that are referenced by your config. And a CI config file if you're automating it.

An idea of such a package.json.

{
  "name": "my-sourcecred-updater",
  "private": true,
  "dependencies": {
    "@sourcecred/core": "^1.1.5",
    "@sourcecred/cli": "^1.1.0",
    "@sourcecred/widgets": "^1.2.0-beta.1",
    "@sourcecred/github": "^1.1.2"
  },
  "scripts": {
    "sourcecred": "sourcecred update"
  }
}

Something along those lines would allow more specific version constraints and a pretty easy workflow like: npm i && npm run sourcecred or yarn && yarn sourcecred.

I'm curious what you imaging sourcecred update doing. Could you walk me through
an example workflow that starts from scratch and ends with having an up-to-date
SourceCred instance for (say) sfosc?

Build and export to NPM

Some ideas about that can be drawn from here:
https://medium.com/netscape/shipping-flowtype-definitions-in-npm-packages-c987917efb65
https://github.com/ryyppy/medium-expose-flow-types

Thanks, will look into these.

@Beanow

This comment has been minimized.

Copy link
Member

commented Jul 18, 2019

I'm curious what you imaging sourcecred update doing. Could you walk me through
an example workflow that starts from scratch and ends with having an up-to-date
SourceCred instance for (say) sfosc?

I think it should do basically everything fetching, calculating and generating wise. In a similar fashion to how the yarn or npm install commands work. You find the config file (package.json / sourcecred.yaml), utilize any caches if available (node_modules / sourcecred_data) and reconcile the differences between current state and desired state. If shortcuts can be taken (like using caches when fetching, or not rebuilding the SPA and only updating data) that's great, but otherwise just start from scratch.

Most likely the desired state would describe: the current timestamp as "source data last synced", plugins to use, weights to be used, frontend SPA to be generated, widgets to be generated and so on.

Note that I include widgets and plugins. I think this command should be extensible through the configuration. In the case of widgets, you would describe: I want a top 100 all time cred SVG for each project and each repo, output to this folder. The widgets NPM package might somehow register itself as being capable of producing this aspect of the desired state.

That way, even when you start using more features / 3rd party integrations, the command doesn't really grow in complexity, instead the config does.

@Beanow

This comment has been minimized.

Copy link
Member

commented Jul 18, 2019

Either way, musing about future CLI's isn't the point of this issue. We can probably improve on this over time.

For the first iteration I would perhaps create:
@sourcecred/sourcecred and @sourcecred/widgets. With sourcecred as the meta package to have both. Separating into more packages can be done alongside refactoring the code.

Priority imo is to have a better invocation of the CLI. Probably by specifying a bin in the package.json and making sure it can be executed directly without compiling. One as sourcecred and one as sourcecred-widgets?

Additionally it would be great if we can distribute a pre-compiled SPA to avoid needing webpack and babel dependencies entirely. Relying only on the api data folder.

I would skip library style dependencies entirely right now as there's hardly any code that could be safely exported as a public API.

@decentralion

This comment has been minimized.

Copy link
Member Author

commented Aug 6, 2019

@Beanow would you be interested in taking a stab at setting up the sourcecred CLI command?

@Beanow

This comment has been minimized.

Copy link
Member

commented Aug 8, 2019

I'll be on holiday so it may be two weeks before I get around to any new things.
By the CLI command, do you mean the setup from last comment? #1232 (comment)

Having the 3 packages and exposing a pre-compiled backend CLI with the npm bin feature?

@decentralion

This comment has been minimized.

Copy link
Member Author

commented Aug 8, 2019

What I'm imagining for the CLI command: We create a concept of a sourcecred instance. An instance has the following properties:

  • it is a git repository
  • it has a package.json which pins the versions of sourcecred used
  • it has its own data and cache directories
  • it has zero or more Projects
  • it contains a complete live frontend
  • it has a root index.html file which serves the frontend

Then the flow would go something like:

# initialize a new SourceCred instance
# sets up a git repository, dummy package.json, adds deps, sets up frontend
$ sourcecred init

# Next add a project to the instance. I'm showing a few possible 
# invocations. Not sure if we'll actually use all of these invocations.
$ sourcecred add-project @my-org
$ sourcecred add-project my-org/my-repo
$ sourcecred add-project -f/path/to/project/config
$ sourcecred add-project https://url-of-my-project-config;

# now update the data
$ sourcecred update [project-id]

# to get scores out
$ sourcecred scores project-id

# to inspect the instance frontend
$ sourcecred serve

Then to publish the instance, the user can just serve the whole instance via a web server, or can deploy by e.g. pushing the whole instance to GitHub pages.

I'm not sure how widgets will fit in exactly; maybe if the user has setup widgets, then running sourcecred update also updates the widgets.

Also, I'm not sure if letting an instance own more than one project is a good idea. It's how I've done things so far, but it also adds a lot of complexity. Now that I think about it, the ugly route data complications I mentioned in #1272 would disappear if we had only one project per instance--the frontend wouldn't need any route-structure configuration, it would just have one url to navigate to. Which seems to me like a pretty strong endorsement of a 1-project-per-instance constraint.

@Beanow

This comment has been minimized.

Copy link
Member

commented Aug 8, 2019

I'll be on holiday so it may be two weeks before I get around to any new things.

Had the planning wrong, should have time here and there.

I see what you mean. I would consider add-project nice to have atm because it's essentially adding one line of config.

But the crux is the init > configure > update workflow. Incorporating some of #1272 frontend perhaps and local previewing with serve.

A big prerequisite for that being the configuration based update command.

Something interesting I noticed about your example is that it may warrant a global installation of the sourcecred package after all. Eg npm i -g sourcecred which would act as a more stable proxy, that can create or find an instance and delegate commands to it.

For example:
sourcecred update may search for a sourcecred.yaml file in the workdir and up. If it finds that, it will invoke node_modules/.bin/sourcecred update from the root of the instance so your global and local versions can be different, you just need to occasionally update the global one in case the proxy gets changed.

About updating widgets. I think that might be a reason to merge features of #1272 into update instead of it's own command. As I would imagine the frontend, widgets and data to be part of the output of sourcecred update. You could consider widgets an "output plugin" in this behavior, and won't need to explicitly do anything with the scores cli command for it.

Assuming the project concept will be a mix and match to for example mean, several handpicked repos, not necessarily in one github org and other sources like discourse, I think that would make sense to make 1:1 with an instance.

As for what I could do to help with the CLI. I could make a proof of concept with these global and local packages and structure of an instance.

Or I could work on implementing the update command as an addition to the current CLI to be a configuration based alternative to sourcecred.js load; yarn -s frontend?

The whole thing at once seems like a fairly big chunk of work, so rather split it up.

@decentralion

This comment has been minimized.

Copy link
Member Author

commented Aug 12, 2019

Assuming the project concept will be a mix and match to for example mean, several handpicked repos, not necessarily in one github org and other sources like discourse, I think that would make sense to make 1:1 with an instance.

I imagine each project will be an arbitrary set of data sources. For example, the SourceCred project might include all of the SourceCred GitHub repositories, possibly excluding odyssey-hackathon as its a fork, as well as the SC Discourse instance. Does this match your expectation?

Re what you can do to help: I advise working on the CLI proof of concept. I think that when it comes to getting the sourcecred frontend style command, I have a bigger "comparative advantage" from knowing where all the skeletons are in our build system. Also if the change coincides with making 1-project-per-instance (which makes the system a lot simpler, as we can take out the project-level interpolation in the frontend), then we need to coordinate with changes to the website, which I'm already re-writing. So it makes more sense for me to take that one on.

decentralion added a commit that referenced this issue Aug 13, 2019

Make SourceCred a public package
I'm mostly motivated by wanting to get greenkeeper lockfile
auto-updating working (see #1269) although this is also a first step
towards making SourceCred usable from NPM (#1232).

For now, see this as us making sure we claim the sourcecred package name
on npm (see: https://www.npmjs.com/package/sourcecred).

I also fixed the license spec so that it's valid SPDX.

decentralion added a commit that referenced this issue Aug 13, 2019

Make SourceCred a public package (#1284)
I'm mostly motivated by wanting to get greenkeeper lockfile
auto-updating working (see #1269) although this is also a first step
towards making SourceCred usable from NPM (#1232).

For now, see this as us making sure we claim the sourcecred package name
on npm (see: https://www.npmjs.com/package/sourcecred).

I also fixed the license spec so that it's valid SPDX.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.