Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create new model for dependencies #447

Closed
JonoYang opened this issue Jun 14, 2022 · 10 comments
Closed

Create new model for dependencies #447

JonoYang opened this issue Jun 14, 2022 · 10 comments
Milestone

Comments

@JonoYang
Copy link
Contributor

scancode-toolkit has a new model called DependentPackage, which models the dependencies detected for a Package from a package manifest. The DependentPackage data shows up in the scancode JSON results in the top-level Dependencies field. We should create a new model, DiscoveredDependency (or some other name) to represent this in scancode.io

@tdruez
Copy link
Member

tdruez commented Jun 14, 2022

@JonoYang I remember you had committed a new model for the dependencies but we decided to postpone after the toolkit upgrade.
Could you start a new branch with this new model? This would be a good base for discussion.

JonoYang added a commit that referenced this issue Jun 14, 2022
Signed-off-by: Jono Yang <jyang@nexb.com>
@JonoYang
Copy link
Contributor Author

@tdruez I started a new branch with the DiscoveredDependency model I had from earlier. The only change I made from my original commit is that I increased the size of the package_uid related fields to 1024.

JonoYang added a commit that referenced this issue Jul 21, 2022
    * Update scan_for_application_packages to save detected Package data to the CodebaseResource it is from, then iterate through the CodebaseResources with Package data and use the proper Package handler to process the Package data
    * Create DiscoveredDependency model
    * Add package_data JSON field to CodebaseResource

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jul 21, 2022
    * Increase field sizes in DiscoveredDependency

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jul 21, 2022
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jul 22, 2022
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jul 23, 2022
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jul 26, 2022
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jul 26, 2022
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jul 27, 2022
    * We never run into the situation where we are updating a DiscoveredDependency when scanning packages
    * Clean up migrations

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jul 27, 2022
    * We never run into the situation where we are updating a DiscoveredDependency when scanning packages
    * Clean up migrations

Signed-off-by: Jono Yang <jyang@nexb.com>
@pombredanne pombredanne added this to the v32.0.0 milestone Jul 28, 2022
JonoYang added a commit that referenced this issue Jul 28, 2022
    * Update scancode-toolkit to 31.0.0rc3 #447
    * Update expected test results

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jul 28, 2022
    * Update scancode-toolkit to 31.0.0rc3 #447
    * Update expected test results
    * Update code formatting

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jul 28, 2022
    * Update scancode-toolkit to 31.0.0rc3 #447
    * Update expected test results
    * Update code formatting

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jul 28, 2022
    * This is to reflect the changes made to the Package model in scancode-toolkit
    * Update test expectations

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jul 28, 2022
    * This is to reflect the changes made to the Package model in scancode-toolkit
    * Update test expectations

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jul 30, 2022
    * purl and dependency_uid are the only required fields on DiscoveredDependency
    * Update migrations
    * Check for created DiscoveredDependency in tests

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Jul 30, 2022
    * Create DiscoveredDependencies in load_codebase pipeline
    * Update tests to check for DiscoveredDependencies
    * Update test expectations

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Aug 1, 2022
    * Update test expectations

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Aug 2, 2022
    * This will fail until scancode-toolkit 31.0.0rc4 is released
    * Update test expectations

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Aug 2, 2022
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Aug 2, 2022
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Aug 2, 2022
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Aug 2, 2022
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Aug 26, 2022
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Aug 26, 2022
    * Update DiscoveredDependency ordering
    * Update daglib test expectations

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Aug 27, 2022
    * Update test expectations

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Aug 27, 2022
    * Update test expectations

Signed-off-by: Jono Yang <jyang@nexb.com>
tdruez added a commit that referenced this issue Aug 29, 2022
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Aug 29, 2022
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Aug 29, 2022
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Aug 29, 2022
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Aug 29, 2022
Signed-off-by: Thomas Druez <tdruez@nexb.com>
JonoYang added a commit that referenced this issue Aug 30, 2022
    * Only show links in dependency for_package tab or dependency datafile_resource tab if there is a value

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Aug 30, 2022
    * Use UpdateFromDataMixin in DiscoveredDependency
    * Create test for DiscoveredDependency.update_from_data()

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Aug 30, 2022
    * Use UpdateFromDataMixin in DiscoveredDependency
    * Create test for DiscoveredDependency.update_from_data()

Signed-off-by: Jono Yang <jyang@nexb.com>
tdruez added a commit that referenced this issue Aug 31, 2022
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Aug 31, 2022
* Implement package assembly in scancode.io #447

Signed-off-by: Jono Yang <jyang@nexb.com>

* Minor formatting changes for consistency #447

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Create DiscoveredPackages before other models #447

Signed-off-by: Jono Yang <jyang@nexb.com>

* Revert "Create DiscoveredPackages before other models #447"

This reverts commit c9b8bed.

Sorting Packages, Dependencies, and Resources from DatafileHandler.assemble() will never work. The code needs to be changed in scancode-toolkit.

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update migration #444

Signed-off-by: Jono Yang <jyang@nexb.com>

* Return package_uids in for_packages #444

    * This is so we are consistent with scancode-toolkit JSON output
    * Update expected test results

Signed-off-by: Jono Yang <jyang@nexb.com>

* Create directory Resources in docker pipeline #485

    * Update test expectations

Signed-off-by: Jono Yang <jyang@nexb.com>

* Implement package assembly in scancode.io #447

Signed-off-by: Jono Yang <jyang@nexb.com>

* Implement package assembly in scancode.io #447

Signed-off-by: Jono Yang <jyang@nexb.com>

* Create DiscoveredDependency model #447

    * Create new dependency list and detail views
    * Update assemble_packages() to create DiscoveredDependencies
    * Update test expectations

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update fields on DiscoveredDependency #447

    * Remove for_package_uid and replace with ForeignKey for_package
    * Remove datafile_path and replace with ForeignKey datafile_resource
    * Create properties for the two removed fields
    * Update dependency views to link to datafile_resource
    * Update expected test results

Signed-off-by: Jono Yang <jyang@nexb.com>

* Properly pluralize verbose name #447

Signed-off-by: Jono Yang <jyang@nexb.com>

* Create new argument for create_from_data #447

    * Add strip_datafile_path_root to DiscoveredDependency.create_from_data
    * This argument strips the root path segment from `datafile_path` before using the path to look up the corresponding CodebaseResource
    * This is used in the case where we are importing a scan from scancode-toolkit, where the root path segments are not stripped by default
    * Update expected test results

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update prefetch_related #447

    * Used cached_property for DiscoveredDependency properties

Signed-off-by: Jono Yang <jyang@nexb.com>

* Prefetch related models in output code #447

Signed-off-by: Jono Yang <jyang@nexb.com>

* Import scancode.io 30.2.0 scans in load_codebase

    * Order DiscoveredDependencies by is_runtime, is_optional, is_resolved, and dependency_uid
    * Do not show dependency_uid value in DiscoveredDependency list view

Signed-off-by: Jono Yang <jyang@nexb.com>

* Revert changes for importing old scancode.io scans

Signed-off-by: Jono Yang <jyang@nexb.com>

* Regen migrations for DiscoveredDependency #447

Signed-off-by: Jono Yang <jyang@nexb.com>

* Migrate DiscoveredPackage.dependencies #447

    * Create migrations to generate new DiscoveredDependency objects from DiscoveredPackage.dependencies before removing the dependencies field

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update test expectations #447

Signed-off-by: Jono Yang <jyang@nexb.com>

* Remove accidentally committed files #447

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update migration logic #447

    * Remove unnecessary else from DiscoveredDependency properties

Signed-off-by: Jono Yang <jyang@nexb.com>

* Add PackageURLMixin to DiscoveredDependency #447

Signed-off-by: Jono Yang <jyang@nexb.com>

* Set DiscoveredDependencies purl fields #447

    * Create migration that populates purl fields for existing DiscoveredDependencies

Signed-off-by: Jono Yang <jyang@nexb.com>

* Store purl values in purl fields #447

    * Do not store dependency_uid in purl fields

Signed-off-by: Jono Yang <jyang@nexb.com>

* Remove purl field from DiscoveredDependency #447

    * We are already storing this info in the purl fields
    * Create purl property on DiscoveredDependency for compatibility

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update DependencyFilterSet #447

    * Add search and purl fields

Signed-off-by: Jono Yang <jyang@nexb.com>

* Don't show DiscoveredDependencies purl fields #447

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update package detail view dependencies tab #447

Signed-off-by: Jono Yang <jyang@nexb.com>

* Add package_type to dependency serializer #511

    * Update test expectations

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update expected test results

Signed-off-by: Jono Yang <jyang@nexb.com>

* Add dependency table column #447

Signed-off-by: Jono Yang <jyang@nexb.com>

* Use tabset in dependency detail view #447

    * Add package_type property to DiscoveredDependency

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update dependency list view #447

    * Use updated table header include
    * Update dependency presentation in package detail view
    * Show package uid on hover on for package tab

Signed-off-by: Jono Yang <jyang@nexb.com>

* Set DiscoveredDependency serializer fields #511

    * Update DiscoveredDependency ordering

Signed-off-by: Jono Yang <jyang@nexb.com>

* Create donut chart for package type #447

Signed-off-by: Jono Yang <jyang@nexb.com>

* Consolidate migrations #447

    * Update DiscoveredDependency ordering
    * Update daglib test expectations

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update dependency JSON ordering #447

    * Update test expectations

Signed-off-by: Jono Yang <jyang@nexb.com>

* Set proper discovereddependencies related_name #447

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Fix template indentation #447

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Refactor update_from_data method into a UpdateFromDataMixin #447

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Fix the ProjectSerializer fields #447

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Fix test_scanpipe_api_project_detail unit test #447

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Add HTML title for list views #506

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Update dependency tabs #447

    * Only show links in dependency for_package tab or dependency datafile_resource tab if there is a value

Signed-off-by: Jono Yang <jyang@nexb.com>

* Use UpdateFromDataMixin #447

    * Use UpdateFromDataMixin in DiscoveredDependency
    * Create test for DiscoveredDependency.update_from_data()

Signed-off-by: Jono Yang <jyang@nexb.com>

* Fix formatting #447

Signed-off-by: Thomas Druez <tdruez@nexb.com>

Signed-off-by: Jono Yang <jyang@nexb.com>
Signed-off-by: Thomas Druez <tdruez@nexb.com>
Co-authored-by: Thomas Druez <tdruez@nexb.com>
@tdruez
Copy link
Member

tdruez commented Aug 31, 2022

PR merged.
Thanks @JonoYang for the implementation 👍

@tdruez tdruez closed this as completed Aug 31, 2022
@tdruez tdruez reopened this Aug 31, 2022
@tdruez
Copy link
Member

tdruez commented Aug 31, 2022

Actually getting an issue when applying the migrations on a production database.

web_1     | Running migrations:
web_1     |   Applying scanpipe.0022_rename_dependencies_discoveredpackage_dependencies_data_and_more... OK
web_1     | Traceback (most recent call last):
web_1     |   File "/app/./manage.py", line 6, in <module>
web_1     |     command_line()
web_1     |   File "/app/scancodeio/__init__.py", line 45, in command_line
web_1     |     execute_from_command_line(sys.argv)
web_1     |   File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
web_1     |     utility.execute()
web_1     |   File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 440, in execute
web_1     |     self.fetch_command(subcommand).run_from_argv(self.argv)
web_1     |   File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 402, in run_from_argv
web_1     |     self.execute(*args, **cmd_options)
web_1     |   File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 448, in execute
web_1     |     output = self.handle(*args, **options)
web_1     |   File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 96, in wrapped
web_1     |     res = handle_func(*args, **kwargs)
web_1     |   File "/usr/local/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 349, in handle
web_1     |     post_migrate_state = executor.migrate(
web_1     |   File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 135, in migrate
web_1     |     state = self._migrate_all_forwards(
web_1     |   File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
web_1     |     state = self.apply_migration(
web_1     |   File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
web_1     |     state = migration.apply(state, schema_editor)
web_1     |   File "/usr/local/lib/python3.9/site-packages/django/db/migrations/migration.py", line 130, in apply
web_1     |     operation.database_forwards(
web_1     |   File "/usr/local/lib/python3.9/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
web_1     |     self.code(from_state.apps, schema_editor)
web_1     |   File "/app/scanpipe/migrations/0023_migrate_dependencies.py", line 33, in migrate_dependencies_to_discovereddependencies
web_1     |     DiscoveredDependency.objects.create(
web_1     |   File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
web_1     |     return getattr(self.get_queryset(), name)(*args, **kwargs)
web_1     |   File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 669, in create
web_1     |     obj = self.model(**kwargs)
web_1     |   File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 585, in __init__
web_1     |     raise TypeError(
web_1     | TypeError: DiscoveredDependency() got unexpected keyword arguments: 'requirement'

tdruez added a commit that referenced this issue Aug 31, 2022
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Aug 31, 2022
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Aug 31, 2022
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Aug 31, 2022
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Aug 31, 2022
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Aug 31, 2022


Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Aug 31, 2022
Signed-off-by: Thomas Druez <tdruez@nexb.com>
@tdruez
Copy link
Member

tdruez commented Aug 31, 2022

Migration issues fixed in #514

@JonoYang
Copy link
Contributor Author

Closing this now that #486 and #514 has been merged.

tdruez added a commit that referenced this issue Oct 27, 2022
Signed-off-by: Thomas Druez <tdruez@nexb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants