Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automate filing third-party content review requests #51

Closed
mickaelistria opened this issue Feb 2, 2021 · 17 comments
Closed

Automate filing third-party content review requests #51

mickaelistria opened this issue Feb 2, 2021 · 17 comments

Comments

@mickaelistria
Copy link
Contributor

I happily run dash-license tool and get a clear and useful output

$ java -jar ~/git/dash-licenses/core/target/org.eclipse.dash.licenses-0.0.1-SNAPSHOT.jar package-lock.json 
Feb 02, 2021 10:29:29 AM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport queryLicenseData
INFO: Querying Eclipse Foundation for license data for 145 items.
Feb 02, 2021 10:29:30 AM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport queryLicenseData
INFO: Found 71 items.
Feb 02, 2021 10:29:30 AM org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport queryLicenseData
INFO: Querying ClearlyDefined for license data for 74 items.
Feb 02, 2021 10:29:32 AM org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport queryLicenseData
INFO: Found 74 items.
License information could not be automatically verified for the following content:

npm/npmjs/-/copy-anything/2.0.1
npm/npmjs/-/less/3.13.1
npm/npmjs/-/sass/1.32.6
npm/npmjs/-/typescript/4.1.3
npm/npmjs/@angular/language-service/11.1.1

This content is either not correctly mapped by the system, or requires review.

Now, I'd like the extra step forward, just like pushing on GitHub suggests a link to directly create a pull request: a prepopulated link to open a CQ for those

@waynebeaton
Copy link
Collaborator

Now, I'd like the extra step forward, just like pushing on GitHub suggests a link to directly create a pull request: a prepopulated link to open a CQ for those

I have good news. I've actually already committed the code that does this (0ffba5e).

It doesn't actually create a CQ; rather, it creates a GitLab Issue (e.g.). I haven't yet written down how to actually use the feature yet, primarily because I'm not quite ready yet for the deluge of requests that might come (we haven't written the backend code yet, so processing these is entirely manual).

It's still all very experimental, so there will be changes. But it should get the job done for your immediate need.

Short version:

  • Get an authentication token from gitlab.eclipse.org
  • Include the "-review" option
  • Pass that token via the "-token" option
  • Pass the project id via the "-project" option

Please do not run this if you have more than dozen or so dependencies identified as requiring review. At least not yet.

@mickaelistria
Copy link
Contributor Author

Great! Then I suggest it's mostly a matter of adding such info in the standard output to make it more accessible. I'll make a PR that copies your comment here.

@mickaelistria
Copy link
Contributor Author

Submitted #53 to help people leveraging this great capability.

@waynebeaton
Copy link
Collaborator

Great! Then I suggest it's mostly a matter of adding such info in the standard output to make it more accessible. I'll make a PR that copies your comment here.

Actually... it's mostly a matter of finding cycles to put some automation support on the backend before somebody runs it on their list of 4,000 NPM dependencies.

At this point, my primary interest is whether or not this actually works for you. There are no new issues in the GitLab repository. Did you try it?

I haven't fully tested the privileges on the repository. Ideally the ability to create issues should be available (and limited) to all committers.

@mickaelistria
Copy link
Contributor Author

I already opened the CQ yesterday, so I didn't use it.

@waynebeaton
Copy link
Collaborator

Try it anyway. It would be helpful.

@mickaelistria
Copy link
Contributor Author

This works like a charm!

$ java -jar ~/git/dash-licenses/core/target/org.eclipse.dash.licenses-0.0.1-SNAPSHOT.jar -project technology.wildwebdeveloper -review -token XXXXXXXXXXXX package-lock.json 
Feb 03, 2021 2:15:44 PM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport queryLicenseData
INFO: Querying Eclipse Foundation for license data for 32 items.
Feb 03, 2021 2:15:45 PM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport queryLicenseData
INFO: Found 11 items.
Feb 03, 2021 2:15:45 PM org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport queryLicenseData
INFO: Querying ClearlyDefined for license data for 21 items.
Feb 03, 2021 2:15:46 PM org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport queryLicenseData
INFO: Found 21 items.
License information could not be automatically verified for the following content:

npm/npmjs/-/copy-anything/2.0.1
npm/npmjs/-/less/3.13.1
npm/npmjs/-/sass/1.32.6
npm/npmjs/-/typescript/4.1.3
npm/npmjs/@angular/language-service/11.1.1

This content is either not correctly mapped by the system, or requires review.


Setting up a review for npm/npmjs/-/less/3.13.1.
 - Created: https://gitlab.eclipse.org/eclipsefdn/iplab/iplab/-/issues/22
Setting up a review for npm/npmjs/-/typescript/4.1.3.
 - Created: https://gitlab.eclipse.org/eclipsefdn/iplab/iplab/-/issues/23
Setting up a review for npm/npmjs/-/copy-anything/2.0.1.
 - Created: https://gitlab.eclipse.org/eclipsefdn/iplab/iplab/-/issues/24
Setting up a review for npm/npmjs/@angular/language-service/11.1.1.
 - Created: https://gitlab.eclipse.org/eclipsefdn/iplab/iplab/-/issues/25
Setting up a review for npm/npmjs/-/sass/1.32.6.
 - Created: https://gitlab.eclipse.org/eclipsefdn/iplab/iplab/-/issues/26

@waynebeaton
Copy link
Collaborator

I'm going to leave this issue open to provide a place to discuss this functionality while we evolve the implementation.

@waynebeaton
Copy link
Collaborator

Note that I've added some code in place to throttle creation to five requests. My intention is to open this up, but only after we have some proper automation supporting the backend.

@waynebeaton waynebeaton changed the title CLI output providing a link to create CQ Automate filing third-party content review requests Mar 11, 2021
@marcdumais-work
Copy link
Contributor

before somebody runs it on their list of 4,000 NPM dependencies.

hehe :)

Please do not incorporate this feature into your automated builds at this time

Please ping me when you think this is ready for use in a "friendly" project's CI. In the meantime I'll consider adding it as an information step, that scans and informs about suspicious dependencies, but does not automatically submits them for review.

@waynebeaton
Copy link
Collaborator

before somebody runs it on their list of 4,000 NPM dependencies.

hehe :)

Don't laugh too hard; I was thinking of you when I typed that. :-)

@marcdumais-work
Copy link
Contributor

As an experiment I did a run, using Theia's yarn.lock directly (like Example: Yarn via yarn.lock), instead of using the helper tool (like Example: Yarn via yarn) or using synp to translate yarn.lock to package-lock.json, and using that as input.

This new run seem to mis-parse packages that start with a @, like the @babel/* bunch and many others. dash-licenses seem to consider them separate packages, form the already known ones that properly start with a @. Looking at automatically opened issues (full list below), I notice that:

  • these packages are not found on npm or clearlydefined
  • the source code is still correctly found
  • issues were created as if they were new packages, many auto-approved

Full results:

License information could not be automatically verified for the following content:

npm/npmjs/-/ajv/6.12.6
npm/npmjs/-/autoprefixer/6.7.7
npm/npmjs/-/babel-polyfill/6.26.0
npm/npmjs/-/big.js/3.2.0
npm/npmjs/-/coa/1.0.4
npm/npmjs/-/eslint-plugin-deprecation/1.2.1
npm/npmjs/-/esprima/4.0.1
npm/npmjs/-/esquery/1.4.0
npm/npmjs/-/extsprintf/1.4.0
npm/npmjs/-/from/0.1.7
npm/npmjs/-/fs-extra/4.0.3
npm/npmjs/-/gauge/2.7.4
npm/npmjs/-/gitconfiglocal/1.0.0
npm/npmjs/-/glob/7.1.3
npm/npmjs/-/js-yaml/3.7.0
npm/npmjs/-/jschardet/2.3.0
npm/npmjs/-/jsdom/11.12.0
npm/npmjs/-/jsmin/1.0.1
npm/npmjs/-/json-schema/0.2.3
npm/npmjs/-/json5/0.5.1
npm/npmjs/-/less-loader/2.2.3
npm/npmjs/-/npmlog/4.1.2
npm/npmjs/-/oniguruma/7.2.3
npm/npmjs/-/parse-json/2.2.0
npm/npmjs/-/postcss-reduce-initial/1.0.1
npm/npmjs/-/q/1.5.1
npm/npmjs/-/rc/1.2.8
npm/npmjs/-/readable-stream/2.3.7
npm/npmjs/-/rechoir/0.6.2
npm/npmjs/-/rx/2.3.24
npm/npmjs/-/shelljs/0.8.4
npm/npmjs/-/source-list-map/2.0.1
npm/npmjs/-/spdx-correct/3.1.1
npm/npmjs/-/spdx-license-ids/3.0.9
npm/npmjs/-/style-loader/0.13.2
npm/npmjs/-/through/2.3.8
npm/npmjs/-/ts-md5/1.2.9
npm/npmjs/-/tweetnacl/0.14.5
npm/npmjs/-/typescript/3.9.10
npm/npmjs/-/uc.micro/1.0.6
npm/npmjs/-/uri-js/4.4.1
npm/npmjs/-/wcwidth/1.0.1
npm/npmjs/babel/code-frame/7.12.11
npm/npmjs/babel/code-frame/7.14.5
npm/npmjs/babel/compat-data/7.14.7
npm/npmjs/babel/core/7.14.8
npm/npmjs/babel/generator/7.14.8
npm/npmjs/babel/helper-annotate-as-pure/7.14.5
npm/npmjs/babel/helper-builder-binary-assignment-operator-visitor/7.14.5
npm/npmjs/babel/helper-compilation-targets/7.14.5
npm/npmjs/babel/helper-create-class-features-plugin/7.14.8
npm/npmjs/babel/helper-create-regexp-features-plugin/7.14.5
npm/npmjs/babel/helper-define-polyfill-provider/0.2.3
npm/npmjs/babel/helper-explode-assignable-expression/7.14.5
npm/npmjs/babel/helper-function-name/7.14.5
npm/npmjs/babel/helper-get-function-arity/7.14.5
npm/npmjs/babel/helper-hoist-variables/7.14.5
npm/npmjs/babel/helper-member-expression-to-functions/7.14.7
npm/npmjs/babel/helper-module-imports/7.14.5
npm/npmjs/babel/helper-module-transforms/7.14.8
npm/npmjs/babel/helper-optimise-call-expression/7.14.5
npm/npmjs/babel/helper-plugin-utils/7.14.5
npm/npmjs/babel/helper-remap-async-to-generator/7.14.5
npm/npmjs/babel/helper-replace-supers/7.14.5
npm/npmjs/babel/helper-simple-access/7.14.8
npm/npmjs/babel/helper-skip-transparent-expression-wrappers/7.14.5
npm/npmjs/babel/helper-split-export-declaration/7.14.5
npm/npmjs/babel/helper-validator-identifier/7.14.8
npm/npmjs/babel/helper-validator-option/7.14.5
npm/npmjs/babel/helper-wrap-function/7.14.5
npm/npmjs/babel/helpers/7.14.8
npm/npmjs/babel/highlight/7.14.5
npm/npmjs/babel/parser/7.14.8
npm/npmjs/babel/plugin-bugfix-v8-spread-parameters-in-optional-chaining/7.14.5
npm/npmjs/babel/plugin-proposal-async-generator-functions/7.14.7
npm/npmjs/babel/plugin-proposal-class-properties/7.14.5
npm/npmjs/babel/plugin-proposal-class-static-block/7.14.5
npm/npmjs/babel/plugin-proposal-dynamic-import/7.14.5
npm/npmjs/babel/plugin-proposal-export-namespace-from/7.14.5
npm/npmjs/babel/plugin-proposal-json-strings/7.14.5
npm/npmjs/babel/plugin-proposal-logical-assignment-operators/7.14.5
npm/npmjs/babel/plugin-proposal-nullish-coalescing-operator/7.14.5
npm/npmjs/babel/plugin-proposal-numeric-separator/7.14.5
npm/npmjs/babel/plugin-proposal-object-rest-spread/7.14.7
npm/npmjs/babel/plugin-proposal-optional-catch-binding/7.14.5
npm/npmjs/babel/plugin-proposal-optional-chaining/7.14.5
npm/npmjs/babel/plugin-proposal-private-methods/7.14.5
npm/npmjs/babel/plugin-proposal-private-property-in-object/7.14.5
npm/npmjs/babel/plugin-proposal-unicode-property-regex/7.14.5
npm/npmjs/babel/plugin-syntax-class-properties/7.12.13
npm/npmjs/babel/plugin-syntax-class-static-block/7.14.5
npm/npmjs/babel/plugin-syntax-dynamic-import/7.8.3
npm/npmjs/babel/plugin-syntax-export-namespace-from/7.8.3
npm/npmjs/babel/plugin-syntax-private-property-in-object/7.14.5
npm/npmjs/babel/plugin-syntax-top-level-await/7.14.5
npm/npmjs/babel/plugin-transform-arrow-functions/7.14.5
npm/npmjs/babel/plugin-transform-async-to-generator/7.14.5
npm/npmjs/babel/plugin-transform-block-scoped-functions/7.14.5
npm/npmjs/babel/plugin-transform-block-scoping/7.14.5
npm/npmjs/babel/plugin-transform-classes/7.14.5
npm/npmjs/babel/plugin-transform-computed-properties/7.14.5
npm/npmjs/babel/plugin-transform-destructuring/7.14.7
npm/npmjs/babel/plugin-transform-dotall-regex/7.14.5
npm/npmjs/babel/plugin-transform-duplicate-keys/7.14.5
npm/npmjs/babel/plugin-transform-exponentiation-operator/7.14.5
npm/npmjs/babel/plugin-transform-for-of/7.14.5
npm/npmjs/babel/plugin-transform-function-name/7.14.5
npm/npmjs/babel/plugin-transform-literals/7.14.5
npm/npmjs/babel/plugin-transform-member-expression-literals/7.14.5
npm/npmjs/babel/plugin-transform-modules-amd/7.14.5
npm/npmjs/babel/plugin-transform-modules-commonjs/7.14.5
npm/npmjs/babel/plugin-transform-modules-systemjs/7.14.5
npm/npmjs/babel/plugin-transform-modules-umd/7.14.5
npm/npmjs/babel/plugin-transform-named-capturing-groups-regex/7.14.7
npm/npmjs/babel/plugin-transform-new-target/7.14.5
npm/npmjs/babel/plugin-transform-object-super/7.14.5
npm/npmjs/babel/plugin-transform-parameters/7.14.5
npm/npmjs/babel/plugin-transform-property-literals/7.14.5
npm/npmjs/babel/plugin-transform-regenerator/7.14.5
npm/npmjs/babel/plugin-transform-reserved-words/7.14.5
npm/npmjs/babel/plugin-transform-runtime/7.14.5
npm/npmjs/babel/plugin-transform-shorthand-properties/7.14.5
npm/npmjs/babel/plugin-transform-spread/7.14.6
npm/npmjs/babel/plugin-transform-sticky-regex/7.14.5
npm/npmjs/babel/plugin-transform-template-literals/7.14.5
npm/npmjs/babel/plugin-transform-typeof-symbol/7.14.5
npm/npmjs/babel/plugin-transform-unicode-escapes/7.14.5
npm/npmjs/babel/plugin-transform-unicode-regex/7.14.5
npm/npmjs/babel/preset-env/7.14.8
npm/npmjs/babel/preset-modules/0.1.4
npm/npmjs/babel/runtime/7.14.8
npm/npmjs/babel/template/7.14.5
npm/npmjs/babel/traverse/7.14.8
npm/npmjs/babel/types/7.14.8
npm/npmjs/dabh/diagnostics/2.0.2
npm/npmjs/discoveryjs/json-ext/0.5.3
npm/npmjs/electron/get/1.12.4
npm/npmjs/eslint/eslintrc/0.4.3
npm/npmjs/humanwhocodes/config-array/0.5.0
npm/npmjs/humanwhocodes/object-schema/1.2.0
npm/npmjs/improved/node/1.1.1
npm/npmjs/nodelib/fs.scandir/2.1.5
npm/npmjs/nodelib/fs.stat/2.0.5
npm/npmjs/nodelib/fs.walk/1.2.8
npm/npmjs/phosphor/algorithm/1.2.0
npm/npmjs/phosphor/collections/1.2.0
npm/npmjs/phosphor/commands/1.7.2
npm/npmjs/phosphor/coreutils/1.3.1
npm/npmjs/phosphor/disposable/1.3.1
npm/npmjs/phosphor/domutils/1.1.4
npm/npmjs/phosphor/dragdrop/1.4.1
npm/npmjs/phosphor/keyboard/1.1.3
npm/npmjs/phosphor/messaging/1.3.0
npm/npmjs/phosphor/properties/1.1.3
npm/npmjs/phosphor/signaling/1.3.1
npm/npmjs/phosphor/virtualdom/1.2.0
npm/npmjs/phosphor/widgets/1.9.3
npm/npmjs/primer/octicons-react/9.6.0
npm/npmjs/sindresorhus/df/1.0.1
npm/npmjs/sindresorhus/df/2.1.0
npm/npmjs/sindresorhus/is/0.14.0
npm/npmjs/stroncium/procfs/1.2.1
npm/npmjs/szmarczak/http-timer/1.1.2
npm/npmjs/theia/compression-webpack-plugin/3.0.0
npm/npmjs/theia/monaco-editor-core/0.23.0
npm/npmjs/theia/node-pty/0.9.0-theia.6
npm/npmjs/types/base64-arraybuffer/0.1.0
npm/npmjs/types/bent/7.3.2
npm/npmjs/types/caseless/0.12.2
npm/npmjs/types/chai-string/1.4.2
npm/npmjs/types/chai/4.2.21
npm/npmjs/types/cookie/0.3.3
npm/npmjs/types/decompress/4.2.4
npm/npmjs/types/diff/3.5.4
npm/npmjs/types/dompurify/2.2.3
npm/npmjs/types/escape-html/0.0.20
npm/npmjs/types/eslint-scope/3.7.1
npm/npmjs/types/eslint/7.28.0
npm/npmjs/types/estree/0.0.50
npm/npmjs/types/fs-extra/4.0.12
npm/npmjs/types/highlight.js/10.1.0
npm/npmjs/types/highlight.js/9.12.4
npm/npmjs/types/jsdom/11.12.0
npm/npmjs/types/linkify-it/3.0.2
npm/npmjs/types/lodash.clonedeep/4.5.6
npm/npmjs/types/lodash.debounce/4.0.3
npm/npmjs/types/lodash.throttle/4.1.6
npm/npmjs/types/lodash/4.14.171
npm/npmjs/types/markdown-it-anchor/4.0.4
npm/npmjs/types/markdown-it/12.0.3
npm/npmjs/types/mdurl/1.0.2
npm/npmjs/types/mime-types/2.1.0
npm/npmjs/types/mime/1.3.2
npm/npmjs/types/minipass/2.2.1
npm/npmjs/types/mkdirp/0.5.2
npm/npmjs/types/mocha/5.2.7
npm/npmjs/types/node/12.20.17
npm/npmjs/types/p-queue/2.3.2
npm/npmjs/types/proxy-from-env/1.0.1
npm/npmjs/types/ps-tree/1.1.1
npm/npmjs/types/puppeteer/2.1.6
npm/npmjs/types/react-dom/16.9.14
npm/npmjs/types/react-virtualized/9.21.12
npm/npmjs/types/react/16.14.11
npm/npmjs/types/react/17.0.15
npm/npmjs/types/request/2.48.7
npm/npmjs/types/requestretry/1.12.8
npm/npmjs/types/rimraf/2.0.5
npm/npmjs/types/route-parser/0.1.3
npm/npmjs/types/safer-buffer/2.1.0
npm/npmjs/types/sanitize-html/2.3.2
npm/npmjs/types/scheduler/0.16.2
npm/npmjs/types/semver/5.5.0
npm/npmjs/types/showdown/1.9.4
npm/npmjs/types/sinon/2.3.7
npm/npmjs/types/tar-fs/1.16.3
npm/npmjs/types/tar-stream/2.2.1
npm/npmjs/types/tar/4.0.5
npm/npmjs/types/temp/0.8.34
npm/npmjs/types/tough-cookie/4.0.1
npm/npmjs/types/trusted-types/2.0.2
npm/npmjs/types/uuid/7.0.5
npm/npmjs/types/which/1.3.2
npm/npmjs/types/write-json-file/2.2.1
npm/npmjs/types/ws/5.1.2
npm/npmjs/types/yargs-parser/20.2.1
npm/npmjs/typescript-eslint/eslint-plugin-tslint/4.28.5
npm/npmjs/typescript-eslint/eslint-plugin/4.28.5
npm/npmjs/typescript-eslint/experimental-utils/3.10.1
npm/npmjs/typescript-eslint/experimental-utils/4.28.5
npm/npmjs/typescript-eslint/parser/4.28.5
npm/npmjs/typescript-eslint/scope-manager/4.28.5
npm/npmjs/typescript-eslint/types/3.10.1
npm/npmjs/typescript-eslint/types/4.28.5
npm/npmjs/typescript-eslint/typescript-estree/3.10.1
npm/npmjs/typescript-eslint/typescript-estree/4.28.5
npm/npmjs/typescript-eslint/visitor-keys/3.10.1
npm/npmjs/typescript-eslint/visitor-keys/4.28.5
npm/npmjs/vscode/codicons/0.0.21
npm/npmjs/webassemblyjs/ast/1.11.1
npm/npmjs/webassemblyjs/floating-point-hex-parser/1.11.1
npm/npmjs/webassemblyjs/helper-api-error/1.11.1
npm/npmjs/webassemblyjs/helper-buffer/1.11.1
npm/npmjs/webassemblyjs/helper-numbers/1.11.1
npm/npmjs/webassemblyjs/helper-wasm-bytecode/1.11.1
npm/npmjs/webassemblyjs/helper-wasm-section/1.11.1
npm/npmjs/webassemblyjs/ieee754/1.11.1
npm/npmjs/webassemblyjs/leb128/1.11.1
npm/npmjs/webassemblyjs/utf8/1.11.1
npm/npmjs/webassemblyjs/wasm-edit/1.11.1
npm/npmjs/webassemblyjs/wasm-gen/1.11.1
npm/npmjs/webassemblyjs/wasm-opt/1.11.1
npm/npmjs/webassemblyjs/wasm-parser/1.11.1
npm/npmjs/webassemblyjs/wast-printer/1.11.1
npm/npmjs/webpack-cli/configtest/1.0.4
npm/npmjs/webpack-cli/info/1.3.0
npm/npmjs/webpack-cli/serve/1.5.1

This content is either not correctly mapped by the system, or requires review.

Setting up a review for npm/npmjs/babel/helper-hoist-variables/7.14.5.

More content needs to be reviewed.
For now, however, this experimental feature only submits the first 10.

@waynebeaton
Copy link
Collaborator

This new run seem to mis-parse packages that start with a @, like the @babel/* bunch and many others. dash-licenses seem to consider them separate packages, form the already known ones that properly start with a @.

It's been a while since I wrote that code, but I swear that I observed the ClearlyDefined ids omitted the '@'... Regardless, that does not appear to be the case. I'll fix it (and the issues that were created).

Thanks for point this out.

@waynebeaton
Copy link
Collaborator

I've pushed an update and have kicked off a build. The update should be reflected in repo.eclipse.org in a few minutes.

@marcdumais-work
Copy link
Contributor

Awesome! I'll try this offline first

@marcdumais-work
Copy link
Contributor

I've pushed an update and have kicked off a build. The update should be reflected in repo.eclipse.org in a few minutes.

Confirmed. I now obtain exactly the same "unverified dependencies" list locally, using yarn.lock and using package-lock.json generated from yarn.lock using synp.

@mickaelistria
Copy link
Contributor Author

This is alive and working.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants