Create "client" matching pipeline #1042
Conversation
JonoYang
force-pushed
the
matching-update
branch
2 times, most recently
from
78a04e8
to
9cc1a6e
Compare
|
The pipeline has been renamed from |
|
@JonoYang @pombredanne I'm wondering if we should split the logic into pipeline steps or have a single step: Single step:
OR Multiple step:
|
|
Also, missing CHANGELOG entry and adding the new pipeline in the docs. |
I ended up splitting |
JonoYang
force-pushed
the
matching-update
branch
from
ab45e4a
to
e78dd3d
Compare
|
|
||
| def poll_matching_results(self): | ||
| """Wait until the match results are ready by polling the match run status.""" | ||
| self.match_results = purldb.poll_until_success(self.run_url) |
There was a problem hiding this comment.
I'm surprised this works as the poll_until_success expects 2 arguments run_url and results_url.
There was a problem hiding this comment.
We should have a unit test for the whole pipeline to catch this.
There was a problem hiding this comment.
Do we have an existing test that tests an entire pipeline? I just see tests where we test the individual pipeline steps.
scanpipe/pipes/purldb.py
Outdated
| response = request_get(run_url) | ||
| if response: | ||
| status = response["status"] | ||
| if status == "success": |
There was a problem hiding this comment.
In case there's a failure (status != "success") on the PurlDB side, how do we handle it on the pipeline side?
There was a problem hiding this comment.
@tdruez My first thought would be to raise an Exception and stop the pipeline there and print the log message from the run response. Do you have a suggestion on what should be done?
There was a problem hiding this comment.
I think we should separate the results_url = project_url + "results/" return request_get(results_url) logic from the polling function to simplify this. We can better handle report potential polling issues this way.
JonoYang
force-pushed
the
matching-update
branch
2 times, most recently
from
2d9bcd7
to
a577df2
Compare
Signed-off-by: Jono Yang <jyang@nexb.com>
Signed-off-by: Jono Yang <jyang@nexb.com>
Signed-off-by: Jono Yang <jyang@nexb.com>
Signed-off-by: Jono Yang <jyang@nexb.com>
* Update test Signed-off-by: Jono Yang <jyang@nexb.com>
Signed-off-by: Jono Yang <jyang@nexb.com>
Signed-off-by: Jono Yang <jyang@nexb.com>
Signed-off-by: Jono Yang <jyang@nexb.com>
Signed-off-by: Jono Yang <jyang@nexb.com>
* Create tests for new functions Signed-off-by: Jono Yang <jyang@nexb.com>
Signed-off-by: Jono Yang <jyang@nexb.com>
Signed-off-by: Jono Yang <jyang@nexb.com>
Signed-off-by: Jono Yang <jyang@nexb.com>
* Update docstrings Signed-off-by: Jono Yang <jyang@nexb.com>
* Update match_to_purldb description Signed-off-by: Jono Yang <jyang@nexb.com>
Signed-off-by: Jono Yang <jyang@nexb.com>
* Update poll_until_success to return True when a run is successful, instead of returning the match results Signed-off-by: Jono Yang <jyang@nexb.com>
Signed-off-by: Jono Yang <jyang@nexb.com>
* Create new test cases for poll_until_success Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang
force-pushed
the
matching-update
branch
2 times, most recently
from
febbc6b
to
8d23184
Compare
* Fix indent in send_project_json_to_matchcode
* Update docstring
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang
force-pushed
the
matching-update
branch
from
8d23184
to
2c6a388
Compare
This PR introduces a new pipeline
matching, which uses the new matching API endpoint from purldb/matchcode for Package matching on a codebase. This pipeline takes in an archive input, creates CodebaseResources, sends a JSON scan to purldb/matchcode, then processes the matched package results when complete.