New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refine and simplify Package models #275

Closed

pombredanne opened this issue Jun 26, 2016 · 6 comments

Assignees

Labels

Milestone

Member

pombredanne commented Jun 26, 2016 •

edited

Things that need to be corrected:

the use of subclassing for Package does not make sense as implemented. We need instead some composition IMHO: the current way smells funny.
And we need a plugin approach such that new package formats are implement as external libraries, eventually using different languages than Python to reuse the many parsers natively available (Ruby, Js, Java, etc). This would likely make the pure subclassing likely naturally vanish.
schematics is likely not the right library for our use case: attrs or marshmallow need to be reviewed. The schematics-based code smells funny too

We need a few more types and change some types:

Project: several packages are part of a larger project that defines many packages attributes
Repo: a package name/version is typically unique only within the confine of a repo
FQN: fully qualified name such as in repo:name:version (and possibly os/arch for many packages beyond deb and rpms) and defining some kind of global identification
Grouping: several packages are often derived for different architecture/platforms (Java for Ruby, egg/wheels/sdist/bdist on Python, various os/arch on RPMs and debs, etc.) or different artifacts are available for a package (src/bin/doc/pom for Maven)
Dependencies: while they live in some group, a flat sequence is likely better than a by-group tree. Also there is always a notion of potential deps vs resolved, locked deps. This needs to be more simply modelled.
Downloads: we have a 1-to-1 between a download and a package (even though each download can be several urls, this is for the same exact file). This could be refined in one package -> many downloads where each download is for the same package version but packaged differently (arch/os/platform/source/binary/etc)

Some extra data fields needed too:

documentation_url
the web page if any for a package release, as is typical for many package repos
beyond keywords, support for Trove-like classifiers (in sf.net or Pypi)
download stats when provided?
file size and filename
release or upload date

Or should be retired:

keywords_doc_url

pombredanne added the package scan label

pombredanne self-assigned this

pombredanne modified the milestone: v2.0

pombredanne mentioned this issue

Enforce the correctness of Package objects #241

Closed

Member Author

pombredanne commented Aug 30, 2016

From #241
So now that we have used schematics, I am not super happy with the way this works. In particular anything that would be stored in the DB in a near future would require having yet another schema being defined in yet another library. I made a few local tests storing things as raw JSON in a DB but that raise another problem of schema migrations.
I am considering using Django's ORM to define the models instead. I will try out some tests in a branch. The semantics are essentially the same as schematics or marshmallow (or the other way around) and there could be some side benefits to store an-flight scan in a temp sqlite DB. I will work out something.
Feedback welcomed

pombredanne added a commit that referenced this issue


          #275 Add Django dep and a minimal configuration

ac96689

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          #275 Add aboutcode namespaced module and django abstract models

05b30cf

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          #275 Ignore aboutcode/django.py from tests

449d2f5

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Add Django dep and a minimal configuration for #275

4b837f2

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          #275 Add aboutcode namespaced module and django abstract models

90f86b0

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          #275 Ignore aboutcode/django.py from tests

ea6463d

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Add Django dep and a minimal configuration for #275

d1335db

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          #275 Add aboutcode namespaced module and django abstract models

0f5f6f3

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          #275 Ignore aboutcode/django.py from tests

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Add Django dep and a minimal configuration for #275

ffa1846

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          #275 Add aboutcode namespaced module and django abstract models

fc7a471

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          #275 Ignore aboutcode/django.py from tests

052f2d1

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Add Django dep and a minimal configuration for #275

eb078da

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          #275 Add aboutcode namespaced module and django abstract models

92b8b2d

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne mentioned this issue

WIP: Use a generator to reduce memory consumption when filtering results #491

Closed

pombredanne added a commit that referenced this issue


          Add Django dep and a minimal configuration for #275

dc4cc89

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          #275 Add aboutcode namespaced module and django abstract models

a10bf82

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne mentioned this issue

Option to write scan result directly into a database #273

Open

pombredanne modified the milestones: v2.0, v2.1

pombredanne mentioned this issue

#699 Post scan plugins #704

Merged

yashdsaraf mentioned this issue

Efficiently navigate up and down the results "tree" of files without loading scanned results into memory #717

Closed

pombredanne added a commit that referenced this issue


          Add code_type attribute #275

f37ff8c

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Refine Package models #275

fdabded

 * Add new PackageIdentifier class for #805 as Package property and as
   discrete type:namespace/name@version?qualifiers#path fields
 * Improved DependentPackage definitions using a package idenitifier
   and simpler flags. Do not use a mapping per scope anymore.
 * Improve related packages definitions with a PackageRelationship
   class using from/to package identifiers
 * Add OrderedDictType for schematics
 * Remove unused Package methods for versions

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Update Package models types and tests #275

579e7bf

 * Support PackageIdentifier class for #805 as Package property and as
   discrete type:namespace/name@version?qualifiers#path fields
 * Improved DependentPackage definitions using a package idenitifier
   and simpler flags. Do not use a mapping per scope anymore.
 * Improve related packages definitions with a PackageRelationship
   class using from/to package identifiers

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Add tests for scoped NPM pcakges #275

c6fbf4a

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Set defaults for DependentPackage is_* flags #275

385b194

 * is_runtime defaults to True
 * is_resolved and is_optional default to False
 * also improve scope handling for maven poms and other cosmetics

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Add basic Win DLL parsing support #275

cc9149b

 * migrated from scancode-toolkit-contrib

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Add basic Gemfile.lock parsing support #275

40f61c4

 * migrated from scancode-toolkit-contrib

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Add basic gemspec and Rubygems parsing support #275

c71c067

 * migrated from scancode-toolkit-contrib

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Add basic Godeps parsing support #275

050d395

 * migrated from scancode-toolkit-contrib

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Add missing test file for Win PE #275

2441d87

 * this is otherwsie gitignored

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Return null, not empty strings in win_pe #275

813bf67

* also renamed the info function to pe_info

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Use puurl for package identification #805 and #275

aba0f7a

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Use the purl Package URL #275

97a6ed9

 * as specified at https://github.com/package-url/purl-spec
 * at the package, dependencies and related levels
 * rename path to subpath to conform with purl
 * not directly tested for now

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Add purl-python library #275

8647cc9

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Use purl library. Remove old code #275

ddf3e3a

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Add purl-python library to setup #805 #275

e3dc995

 * This is a first rough implmentation using
   https://github.com/package-url/packageurl-python
 * Based on package-url/purl-spec#1

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Update maven tests expectations #275

ec224d2

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Upgrade packageurl library to 0.5.0 #275

29f57af

 * this is an early release that implements the to-be spec
  at package-url/purl-spec#31
 * update tests accordingly

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Add namespace to npm. Compute defaults URL #275

927273b

 * handle namespace for npm scoped packages properly
 * add missing description to npm
 * Add new package attributes default_web_baseurl,
   default_download_baseurl and default_api_baseurl and the
   corresponding methods to override in subclasses
 * Improve npm URLs computation and implement these methods.
 * Other minor refinements.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Update test expectations #275

1c23225

 * these come from the latest rebase on develop

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added a commit that referenced this issue


          Merge latest develop #275

3dc1abc

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne mentioned this issue

Distinguish package manifests from actual packages #1027

Closed

pombredanne added a commit that referenced this issue


          Rename asserted_license to declared_licensing #275

c7fd59a

 Using "licensing" rather than license makes it clear that this is not a
 normalized scancode "license" but is instead the original, as
 "asserted" or "declared" licensing in a package manifest.
 Declared is the term used by SPDX.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Member Author

pombredanne commented May 16, 2018

This is completed and merged in develop.

pombredanne closed this as completed

pombredanne mentioned this issue

Normalize package types in packagedcode #1066

Closed

jhgoebbert commented Sep 19, 2018

This commit adds in setup.py
'pefile == 1.2.10-132',
which makes it impossible to install on Linux with pip, as there is no version like this in PyPi:
https://pypi.org/project/pefile/#history

Member Author

pombredanne commented Sep 19, 2018 •

edited

@jhgoebbert good point. That's never been a released version on Pypi!
Do you mind to create a separate ticket specifically for this issue?
thank you!

Member Author

pombredanne commented Sep 25, 2018

Thanks ... This is now tracked in #1183

pombredanne mentioned this issue

Document if a dependency is required or not #634

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment