New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cli2: add sketch of plugin loading #1810
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
// @flow | ||
|
||
import type {CliPlugin} from "./cliPlugin"; | ||
import {GithubCliPlugin} from "../plugins/github/cliPlugin"; | ||
|
||
/** | ||
* Returns an object mapping owner-name pairs to CLI plugin | ||
* declarations; keys are like `sourcecred/github`. | ||
*/ | ||
export function bundledPlugins(): {[pluginId: string]: CliPlugin} { | ||
return {"sourcecred/github": new GithubCliPlugin()}; | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
// @flow | ||
|
||
import type {PluginDeclaration} from "../analysis/pluginDeclaration"; | ||
import type {WeightedGraph} from "../core/weightedGraph"; | ||
import type {ReferenceDetector} from "../core/references/referenceDetector"; | ||
|
||
export interface CliPlugin { | ||
declaration(): PluginDeclaration; | ||
load(PluginDirectoryContext): Promise<void>; | ||
graph(PluginDirectoryContext, ReferenceDetector): Promise<WeightedGraph>; | ||
referenceDetector(PluginDirectoryContext): Promise<ReferenceDetector>; | ||
} | ||
|
||
export interface PluginDirectoryContext { | ||
configDirectory(): string; | ||
cacheDirectory(): string; | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
// @flow | ||
|
||
import {join as pathJoin} from "path"; | ||
import fs from "fs-extra"; | ||
|
||
import type {PluginDirectoryContext} from "./cliPlugin"; | ||
import {parse as parseConfig, type InstanceConfig} from "./instanceConfig"; | ||
|
||
export async function loadInstanceConfig( | ||
baseDir: string | ||
): Promise<InstanceConfig> { | ||
const projectFilePath = pathJoin(baseDir, "sourcecred.json"); | ||
const contents = await fs.readFile(projectFilePath); | ||
return Promise.resolve(parseConfig(JSON.parse(contents))); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: in an async function There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, but only because of a JavaScript design flaw that has really There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Certainly won't argue with you that Promises have a bunch of design flaws. I've built several client projects with https://github.com/fluture-js/Fluture instead of Promises and there's a lot of great properties there that I'm sorely missing having gone back to Promises. On the other hand, the language integration definitely yields productivity benefits over going against the unfortunate standard, and reduces learning curve. In this case In other words: Promise.resolve(Promise.resolve(123))
> Promise { <state>: "fulfilled", <value>: 123 } We get a My taste preference here is to avoid that surprise, so I generally encourage others not to Anyhow, if you feel like there's a benefit to going the other way, curious to hear about it. But for this PR, I'm fine with either :] There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I prefer the non-redundant style, although haven't invested the effort to really understand why William avoids it. Since we don't have a lint rule to automatically enforce it, I say it's authors pick on which to use. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That I’m happy to discuss in more detail, and I can provide a specific |
||
} | ||
|
||
// Make a directory, if it doesn't exist. | ||
function mkdirx(path: string) { | ||
try { | ||
fs.mkdirSync(path); | ||
} catch (e) { | ||
if (e.code !== "EEXIST") { | ||
throw e; | ||
} | ||
} | ||
} | ||
Comment on lines
+17
to
+26
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Seems like There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That does something else. We don’t want to make all the transitive |
||
|
||
export function makePluginDir( | ||
baseDir: string, | ||
prefix: $ReadOnlyArray<string>, | ||
pluginId: string | ||
): string { | ||
const idParts = pluginId.split("/"); | ||
if (idParts.length !== 2) { | ||
throw new Error(`Bad plugin name: ${pluginId}`); | ||
} | ||
const [pluginOwner, pluginName] = idParts; | ||
const pathComponents = [...prefix, pluginOwner, pluginName]; | ||
let path = baseDir; | ||
for (const pc of pathComponents) { | ||
path = pathJoin(path, pc); | ||
mkdirx(path); | ||
} | ||
Comment on lines
+38
to
+43
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Seems like There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As above. |
||
return path; | ||
} | ||
|
||
export function pluginDirectoryContext( | ||
baseDir: string, | ||
pluginName: string | ||
): PluginDirectoryContext { | ||
const cacheDir = makePluginDir(baseDir, ["cache"], pluginName); | ||
const configDir = makePluginDir(baseDir, ["config"], pluginName); | ||
return { | ||
configDirectory() { | ||
return configDir; | ||
}, | ||
cacheDirectory() { | ||
return cacheDir; | ||
}, | ||
}; | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,55 @@ | ||
// @flow | ||
|
||
import {CliPlugin} from "./cliPlugin"; | ||
import {bundledPlugins as getAllBundledPlugins} from "./bundledPlugins"; | ||
|
||
type PluginName = string; | ||
|
||
export type InstanceConfig = {| | ||
+bundledPlugins: Map<PluginName, CliPlugin>, | ||
|}; | ||
|
||
export type RawInstanceConfig = {| | ||
+bundledPlugins: $ReadOnlyArray<BundledPluginSpec>, | ||
|}; | ||
|
||
// Plugin identifier, like `sourcecred/identity`. Version number is | ||
// implicit from the SourceCred version. This is a stopgap until we have | ||
// a plugin system that admits external, dynamically loaded | ||
// dependencies. | ||
export type BundledPluginSpec = string; | ||
|
||
type JsonObject = | ||
| string | ||
| number | ||
| boolean | ||
| null | ||
| JsonObject[] | ||
| {[string]: JsonObject}; | ||
Comment on lines
+22
to
+28
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Having read your linked: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ This is pretty nice. Previously been irked by the Not sure if I like this library too much, due to fairly opaque babel magic. On the other hand, not having to declare the same type twice (once as a static Flow type, again as a parser) is really handy. Have you tried any generators? Thoughts on them? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We (@decentralion and I) chatted about this a bit. We chose to write the |
||
|
||
export function parse(raw: JsonObject): InstanceConfig { | ||
if (raw == null || typeof raw !== "object" || Array.isArray(raw)) { | ||
throw new Error("bad config: " + JSON.stringify(raw)); | ||
} | ||
const {bundledPlugins: rawBundledPlugins} = raw; | ||
if (!Array.isArray(rawBundledPlugins)) { | ||
console.warn(JSON.stringify(raw)); | ||
throw new Error( | ||
"bad bundled plugins: " + JSON.stringify(rawBundledPlugins) | ||
); | ||
} | ||
const allBundledPlugins = getAllBundledPlugins(); | ||
const bundledPlugins = new Map(); | ||
for (const name of rawBundledPlugins) { | ||
if (typeof name !== "string") { | ||
throw new Error("bad bundled plugin: " + JSON.stringify(name)); | ||
} | ||
const plugin = allBundledPlugins[name]; | ||
if (plugin == null) { | ||
throw new Error("bad bundled plugin: " + JSON.stringify(name)); | ||
} | ||
bundledPlugins.set(name, plugin); | ||
} | ||
const result = {bundledPlugins}; | ||
return result; | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
// @flow | ||
|
||
import type {Command} from "./command"; | ||
import {loadInstanceConfig, pluginDirectoryContext} from "./common"; | ||
|
||
function die(std, message) { | ||
std.err("fatal: " + message); | ||
return 1; | ||
} | ||
|
||
const loadCommand: Command = async (args, std) => { | ||
if (args.length !== 0) { | ||
die(std, "usage: sourcecred load"); | ||
} | ||
const baseDir = process.cwd(); | ||
const config = await loadInstanceConfig(baseDir); | ||
for (const [name, plugin] of config.bundledPlugins) { | ||
const dirContext = pluginDirectoryContext(baseDir, name); | ||
plugin.load(dirContext); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Let's have the task reporter report that load started for the plugin--that way we have consistent measurements for all the plugins, and the plugin can use the provided task reporter to give more detailed information (e.g. timing on sub-portions) |
||
} | ||
return 0; | ||
}; | ||
Comment on lines
+11
to
+22
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Want to suggest that there's some complecting going on here. One part is the transport implementation we just-so-happen to be using (CLI) and the other is a use-case. Since we're taking a greenfield approach at the CLI now, it's a distinction I'd love to see at some point. Doing so would allow exposing a first-class javascript API, similar to the likes of babel and webpack. While making the CLI transport implementation simple. In this case the distinction would look like: const loadCommand: Command = async (args, std) => {
// Positional arguments, transport detail.
if (args.length !== 0) {
die(std, "usage: sourcecred load");
}
// Environment data extracting, transport detail.
const baseDir = process.cwd();
// Call the use-case.
loadUsecase(baseDir);
// Exit codes, transport detail.
return 0;
};
// Suitable to expose as Javascript API.
function loadUseCase(baseDir: string): void {
const config = await loadInstanceConfig(baseDir);
for (const [name, plugin] of config.bundledPlugins) {
const dirContext = pluginDirectoryContext(baseDir, name);
plugin.load(dirContext);
}
} There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Based on the dev call today. If we do want to run a local daemon we can send commands over HTTP, I think this will help out. IPFS and Docker are examples that invested in this structure, as they expose their commands through multiple transports (library, unix sockets and HTTP) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree with Beanow that we'll want to expose a functional style for implementing this logic that isn't hard-wired into the CLI. I don't need that to happen in this commit--I think it will be easy to factor out in the future when we need it. Though it could also be nice to have. I approve either way. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, I of course also agree that it’d be great to have a JS API that |
||
|
||
export default loadCommand; |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,108 @@ | ||
// @flow | ||
|
||
import Database from "better-sqlite3"; | ||
import fs from "fs-extra"; | ||
import {join as pathJoin} from "path"; | ||
|
||
import fetchGithubRepo, {fetchGithubRepoFromCache} from "./fetchGithubRepo"; | ||
import type {CacheProvider} from "../../backend/cache"; | ||
import type {CliPlugin, PluginDirectoryContext} from "../../cli2/cliPlugin"; | ||
import type {PluginDeclaration} from "../../analysis/pluginDeclaration"; | ||
import type {ReferenceDetector} from "../../core/references/referenceDetector"; | ||
import type {WeightedGraph} from "../../core/weightedGraph"; | ||
import {Graph} from "../../core/graph"; | ||
import {RelationalView} from "./relationalView"; | ||
import {createGraph} from "./createGraph"; | ||
import {declaration} from "./declaration"; | ||
import {fromRelationalViews as referenceDetectorFromRelationalViews} from "./referenceDetector"; | ||
import {parse as parseConfig, type GithubConfig} from "./config"; | ||
import {validateToken, type GithubToken} from "./token"; | ||
import {weightsForDeclaration} from "../../analysis/pluginDeclaration"; | ||
|
||
const TOKEN_ENV_VAR_NAME = "SOURCECRED_GITHUB_TOKEN"; | ||
|
||
async function loadConfig( | ||
dirContext: PluginDirectoryContext | ||
): Promise<GithubConfig> { | ||
const dirname = dirContext.configDirectory(); | ||
const path = pathJoin(dirname, "config.json"); | ||
const contents = await fs.readFile(path); | ||
return Promise.resolve(parseConfig(JSON.parse(contents))); | ||
} | ||
|
||
// Shim to interface with `fetchGithubRepo`; TODO: refactor that to just | ||
// take a directory. | ||
class CacheProviderImpl implements CacheProvider { | ||
Comment on lines
+33
to
+35
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Actually, I'm a huge fan of using these interfaces. It decouples the rest of the code from the implementation details of using filesystems. Would recommend applying the same idea to the configuration and tokens too like On the flipside, I would rather not see To give a common example where this practice pays off. Many applications use ENV for secrets. However interesting when using Docker Swarm, ENV is not suitable as it will be passed around in plaintext. Instead it should use "Docker Secrets", which most application implement by accepting a Having a central There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I think we're in agreement that stuff like IMO, losing the CacheProvider is not a big deal because we aren't actually supporting other filesystems, so as developers we aren't getting feedback on whether we're (for example) using CacheProvider consistently across the codebase. Without that feedback, there's no particular reason to believe that we will actually be using it consistently when we need that flexibility, or even that this is giving us the right flexibility compared to what we'll actually need when we need it. So we don't benefit much from it, and it does add extra indirection and cognitive overhead. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I’m not really familiar with the way in which you’re using the term Currently, |
||
_dirContext: PluginDirectoryContext; | ||
constructor(dirContext: PluginDirectoryContext) { | ||
this._dirContext = dirContext; | ||
} | ||
database(id: string): Promise<Database> { | ||
const path = pathJoin(this._dirContext.cacheDirectory(), `${id}.db`); | ||
return Promise.resolve(new Database(path)); | ||
} | ||
} | ||
|
||
function getTokenFromEnv(): GithubToken { | ||
const rawToken = process.env[TOKEN_ENV_VAR_NAME]; | ||
if (rawToken == null) { | ||
throw new Error(`No GitHub token provided: set ${TOKEN_ENV_VAR_NAME}`); | ||
} | ||
return validateToken(rawToken); | ||
} | ||
|
||
export class GithubCliPlugin implements CliPlugin { | ||
declaration(): PluginDeclaration { | ||
return declaration; | ||
} | ||
|
||
async load(ctx: PluginDirectoryContext): Promise<void> { | ||
const cache = new CacheProviderImpl(ctx); | ||
const token = getTokenFromEnv(); | ||
const config = await loadConfig(ctx); | ||
for (const repoId of config.repoIds) { | ||
await fetchGithubRepo(repoId, {token, cache}); | ||
} | ||
} | ||
|
||
async graph( | ||
ctx: PluginDirectoryContext, | ||
rd: ReferenceDetector | ||
): Promise<WeightedGraph> { | ||
const _ = rd; // TODO(#1808): not yet used | ||
const cache = new CacheProviderImpl(ctx); | ||
const token = getTokenFromEnv(); | ||
const config = await loadConfig(ctx); | ||
|
||
const repositories = []; | ||
for (const repoId of config.repoIds) { | ||
repositories.push(await fetchGithubRepoFromCache(repoId, {token, cache})); | ||
} | ||
const graph = Graph.merge( | ||
repositories.map((r) => { | ||
const rv = new RelationalView(); | ||
rv.addRepository(r); | ||
return createGraph(rv); | ||
}) | ||
); | ||
const weights = weightsForDeclaration(declaration); | ||
return {graph, weights}; | ||
} | ||
|
||
async referenceDetector( | ||
ctx: PluginDirectoryContext | ||
): Promise<ReferenceDetector> { | ||
const cache = new CacheProviderImpl(ctx); | ||
const token = getTokenFromEnv(); | ||
const config = await loadConfig(ctx); | ||
|
||
const rvs = []; | ||
for (const repoId of config.repoIds) { | ||
const repo = await fetchGithubRepoFromCache(repoId, {token, cache}); | ||
const rv = new RelationalView(); | ||
rv.addRepository(repo); | ||
rvs.push(rv); | ||
} | ||
return referenceDetectorFromRelationalViews(rvs); | ||
} | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
// @flow | ||
|
||
import {type RepoId, stringToRepoId} from "./repoId"; | ||
|
||
export type GithubConfig = {| | ||
+repoIds: $ReadOnlyArray<RepoId>, | ||
|}; | ||
|
||
// eslint-disable-next-line no-unused-vars | ||
type SerializedGithubConfig = {| | ||
+repositories: $ReadOnlyArray<string>, | ||
|}; | ||
// (^ for documentation purposes) | ||
|
||
type JsonObject = | ||
| string | ||
| number | ||
| boolean | ||
| null | ||
| JsonObject[] | ||
| {[string]: JsonObject}; | ||
|
||
export function parse(raw: JsonObject): GithubConfig { | ||
if (raw == null || typeof raw !== "object" || Array.isArray(raw)) { | ||
throw new Error("bad config: " + JSON.stringify(raw)); | ||
} | ||
const {repositories} = raw; | ||
if (!Array.isArray(repositories)) { | ||
throw new Error("bad repositories: " + JSON.stringify(repositories)); | ||
} | ||
const repoIds = repositories.map((x) => { | ||
if (typeof x !== "string") { | ||
throw new Error("bad repository: " + JSON.stringify(x)); | ||
} | ||
return stringToRepoId(x); | ||
}); | ||
return {repoIds}; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's provide a TaskReporter to each of these methods.