Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add D2D for ELFs and Go binaries #1113 #1114 #1170

Merged
merged 18 commits into from
Apr 23, 2024
Merged

Add D2D for ELFs and Go binaries #1113 #1114 #1170

merged 18 commits into from
Apr 23, 2024

Conversation

TG1999
Copy link
Member

@TG1999 TG1999 commented Apr 18, 2024

This is for these two issues:

Testing steps:

  • Install latest Scancode.io
  • Open http://127.0.0.1:8001/project/add/ in your browser, Create a new project: Name this “test”
  • Add the elf or executable binary in a zip file and add it to the to side of the D2D, by adding to- prefix on the zip file. For example to-data.zip and add the files like builder.go in a zip file and add it to the from side of the D2D, by adding from- prefix on the zip file.
  • Select map_deploy_to_develop pipeline.
  • Click create
  • Press F5
  • Click on Relations tab and you will find the relation.

Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
@TG1999 TG1999 changed the title D2d for collecting dwarf paths for elfs and go file paths for executable binaries Add D2d for collecting dwarf paths for elfs and go file paths for executable binaries Apr 18, 2024
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
scanpipe/pipelines/deploy_to_develop.py Outdated Show resolved Hide resolved
@@ -1662,3 +1664,148 @@ def _match_purldb_resources_post_process(
package.add_resources(unmapped_resources)

return interesting_codebase_resources.count()


def _map_paths_resource(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Too much imbrications, please refactor in smaller testable pieces.

from_resources.get(id=rid) for rid in match.resource_ids
]
matched_from_resources.sort(
key=lambda res: (len(res.path.strip("/").split("/")), res.path)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A proper well-named function would be better for readability and testing.

Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
@pombredanne pombredanne changed the title Add D2d for collecting dwarf paths for elfs and go file paths for executable binaries Add D2D for ELFs and Go binaries #1113 #1114 Apr 23, 2024
@@ -189,6 +191,16 @@ def map_javascript(self):
"""
d2d.map_javascript(project=self.project, logger=self.log)

@group("Elf")
def map_elf(self):
"""Map DWARF paths using similarities of path suffixes."""
Copy link
Member

@pombredanne pombredanne Apr 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"""Map DWARF paths using similarities of path suffixes."""
"""Map ELF binaries to their sources."""

Copy link
Member

@tdruez tdruez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are 9 new functions introduced in those changes and I see only 2 unit tests.
We need to improve this test coverage.

)


def process_relations(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function name and the docstring are too generic.

"""
return sorted(
matched_from_resources,
key=lambda res: (len(res.path.strip("/").split("/")), res.path),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's the lambda "key" that needed to be replaced by a function.


@group("Go")
def map_go(self):
"""Map GO file paths using similarities of path suffixes."""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"""Map GO file paths using similarities of path suffixes."""
"""Map Go binaries to their sources."""

Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
@TG1999 TG1999 merged commit 868b0bf into main Apr 23, 2024
9 checks passed
@TG1999 TG1999 deleted the elf_d2d branch April 23, 2024 12:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants