Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add system packages support in the new packages model #2909

Merged
merged 36 commits into from
Apr 25, 2022
Merged

Add system packages support in the new packages model #2909

merged 36 commits into from
Apr 25, 2022

Conversation

AyanSinhaMahapatra
Copy link
Member

@AyanSinhaMahapatra AyanSinhaMahapatra commented Apr 5, 2022 •
edited by pombredanne
Loading

Tasks

  • Reviewed contribution guidelines
  • PR is descriptively titled 📑 and links the original issue above 🔗
  • Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
    Run tests locally to check for errors.
  • Commits are in uniquely-named feature branch and has no merge conflicts 📁

@AyanSinhaMahapatra AyanSinhaMahapatra force-pushed the syspacfiles branch 2 times, most recently from 58692f8 to 6a1cdc4 Compare April 7, 2022 23:06
@pombredanne
Copy link
Member

@AyanSinhaMahapatra the latest jinja2 have some import changes sphinx-doc/sphinx#10291 which makes some test fail.
IMHO we should:

  1. use Python 3.9 for doc generation
  2. bump all the doc deps.

@AyanSinhaMahapatra
Copy link
Member Author

AyanSinhaMahapatra commented Apr 11, 2022 •
edited
Loading

@pombredanne I bumped doc build tests to run on python3.9 here: #2896

We don't use pinned requirements for doc builds anymore, using https://github.com/nexB/scancode-toolkit/blob/develop/setup.cfg#L133 instead. The above mentioned PR makes some changes and fixes the doc builds (updated skeleton with UI from @johnmhoran and --docs option in configure directly).

We can merge this first in develop imho. (Not merging here because we need to amend and squash commits in this branch)

@AyanSinhaMahapatra AyanSinhaMahapatra force-pushed the syspacfiles branch 2 times, most recently from 87ebb9a to 350d4ab Compare April 18, 2022 14:36
@pombredanne
Copy link
Member

@AyanSinhaMahapatra let's leave the tests/summarycode/ aside as this is is WIP from @JonoYang in #2914

@AyanSinhaMahapatra
Copy link
Member Author

let's leave the tests/summarycode/

Yes, makes sense.

@pombredanne
Collect package_data_fields correctly in CSV
89efd09 
We now collect package_data_fields corectly for use in CSVs

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Streamline get_package_data() API
3558de4 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Pass the Resource.id to scanner functions
affa5a9 
Also apply minor formatting and cosmetic changes

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Remove package and dependency uuid from tests
f3cdfb4 
In tests scans the UUID are best left out for stability of expected
results

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Refine package models
6ab3565 
* Rename PackageFile to FileReference and update definition
* Rename PackageDataFile to DatafileHandler
* Streamline class hierarchy
* Move PackageDataFile subclasses to misc.py

In Dependency:
* add for_package_uid, datafile_path, datasource_id
* rename dependency_uuid to dependency_uid

In Package
* add package_uid, datafile_paths, datasource_ids

In PackageData:
* remove contains_source_code from
* add file_references using FileReference, and remove installed_files
* all *_url computed URLs are now stored
* remove extra_key_files and extra_root_files properties

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@JonoYang JonoYang mentioned this pull request Apr 22, 2022
Closed
@pombredanne
Adopt new handler design for ABOUT files
2dd374f 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Update testing utilities for package handlers
472fc31 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for Alpine Linux
fbdba2e 
We now have separate handlers:
* AlpineApkArchiveHandler
* AlpineInstalledDatabaseHandler
* AlpineApkbuildHandler

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for Bower
0752827 
We now have one handler: BowerJsonHandler

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for build scripts
ba0ed20 
We now have separate handlers:
* AutotoolsConfigureHandler
* BazelBuildHandler
* BuckPackageHandler
* BuckMetadataBzlHandler

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for Gradle build scripts
229d945 
We now have this handler to parse Gradle files:
* BuildGradleHandler

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for Rust Cargo
66216d4 
We now have separate handlers:
* CargoTomlHandler
* CargoLockHandler

Adn we assemble these in a Package correctly

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for chef cookbooks
2813a7f 
We now have separate handlers:
* ChefCookbookHandler
* ChefMetadataJsonHandler
* ChefMetadataRbHandler

Adn we assemble these in Packages correctly

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for Cocoapods
acd10b6 
We now have these data file handlers:
* PodspecHandler
* PodfileHandler
* PodfileLockHandler
* PodspecJsonHandler

And we assemble these in Packages correctly

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for Conda
c8ec5d1 
We now have these data file handlers:
* CondaYamlHandler
* CondaMetaYamlHandler

And we assemble these in Packages correctly

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for CRAN
f1a2cfa 
We now have this data file handler:
* CranDescriptionFileHandler

And we assemble it in a Package correctly

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for Debian
2c416c0 
This is a major refactoring of how we handle Debian packages in
archives, extracted or installed layouts.
In particular copyright files are treated as a package data file.

We now have these data file handlers:
* DebianSourcePackageMetadataTarballHandler
* DebianSourcePackageTarballHandler
* DebianControlFileInExtractedDebHandler
* DebianControlFileInSourceHandler
* DebianDscFileHandler
* DebianInstalledStatusDatabaseHandler
* DebianDistrolessInstalledDatabaseHandler
* DebianInstalledFilelistHandler
* DebianInstalledMd5sumFilelistHandler
* DebianMd5sumFilelistInPackageHandler
* DebianCopyrightFileInSourceHandler
* DebianCopyrightFileInPackageHandler
* StandaloneDebianCopyrightFileHandler (no active for now)

And we assemble these in Packages correctly.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for FreeBSD
068b86d 
We now have this data file handler:
* CompactManifestHandler

And we assemble it in a Package correctly

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for Conda
78523fc 
We now have these data file handlers:
* GoModHandler
* GoSumHandler
* GodepsHandler (legacy)

And we assemble these in Packages correctly

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Modernize code for parsing Gemfile.lock
8e84c6f 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for Haxe
2ccba8a 
We now have this data file handler:
* HaxelibJsonHandler

And we assemble it in a Package correctly

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for Maven and JARS
2ad6761 
We now have these data file handlers:
* JavaJarManifestHandler
* MavenPomXmlHandler
* MavenPomPropertiesHandler

And we assemble these in Packages correctly

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for Windows packages
b131074 
We now have these data file handlers:
* MsiInstallerHandler
* NugetNupkgHandler
* NugetNuspecHandler
* WindowsExecutableHandler
* MicrosoftUpdateManifestHandler
* InstalledProgramFromDockerSoftwareDeltaHandler
* InstalledProgramFromDockerFilesSoftwareHandler
* InstalledProgramFromDockerUtilityvmSoftwareHandler

And we assemble these in Packages correctly

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for Rubygems packages
fb93891 
We now have these data file handlers:
* GemArchiveHandler
* GemMetadataArchiveExtractedHandler
* GemspecHandler
* GemspecInExtractedGemHandler
* GemspecInInstalledVendorBundleSpecificationsHandler
* GemfileHandler
* GemfileInExtractedGemHandler
* GemfileLockHandler
* GemfileLockInExtractedGemHandler

And we assemble these in Packages correctly
Also improve minimally gemspec parsing

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for npm packages
494b87d 
We now have these data file handlers:
* NpmPackageJsonHandler
* NpmPackageLockJsonHandler
* NpmShrinkwrapJsonHandler
* YarnLockV2Handler
* YarnLockV1Handler

And we assemble these in Packages correctly

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for OCaml OPAM
5a0650c 
We now have this data file handler:
* OpamFileHandler

And we assemble this in a Package correctly

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for npm packages
8d67fd8 
We now have these data file handlers:
* PhpComposerJsonHandler
* PhpComposerLockHandler

And we assemble these in Packages correctly

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for RPMs
1af5648 
We now have these data file handlers:
* RpmInstalledNdbDatabaseHandler
* RpmInstalledSqliteDatabaseHandler
* RpmInstalledBdbDatabaseHandler
* RpmSpecfileHandler
* RpmArchiveHandler

And we assemble these in Packages correctly.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for PyPI packages
3dc5dd8 
We now have these data file handlers:
* PythonEggPkgInfoFile
* PythonEditableInstallationPkgInfoFile
* PythonSdistPkgInfoFile
* PythonInstalledWheelMetadataFile
* PyprojectTomlHandler
* PypiWheelHandler
* PypiEggHandler
* PypiSdistArchiveHandler
* PythonSetupPyHandler
* SetupCfgHandler
* PipfileHandler
* PipfileLockHandler
* PipRequirementsFileHandler

And we assemble these in Packages correctly.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for Dart pub packages
adc0862 
We now have these data file handlers:
* DartPubspecYamlHandler
* DartPubspecLockHandler

And we assemble these in Packages correctly.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new handler design for README files
2cd448c 
We now have this data file handler:
* ReadmeHandler

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Add new handler for etc/os-release files
0db6803 
* Add new dependency on container-inspector
* Also cleanup and sync setup files

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Improve package handling
296176a 
* Adopt new handler style in plugin
* improve utilities
* test plugin and package recognition

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Align CLI and summary code with packagedcode
b5a126a 
* Adopt new handler style and top level packages

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Remove skeleton test suite
ba7dd22 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Add doc on expected test file regeneration
8e07301 
Using the new SCANCODE_REGEN_TEST_FIXTURES

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Copy link
Member

All green at last.....
Merging now!

@pombredanne pombredanne merged commit 2c9b4ab into develop Apr 25, 2022
@pombredanne pombredanne deleted the syspacfiles branch April 25, 2022 06:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@AyanSinhaMahapatra @pombredanne