Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create a code matcher for approximate files matching #342

Closed
1 task done
Tracked by #239
pombredanne opened this issue Mar 12, 2024 · 1 comment
Closed
1 task done
Tracked by #239

Create a code matcher for approximate files matching #342

pombredanne opened this issue Mar 12, 2024 · 1 comment
Assignees
@JonoYang
Labels
high priority High Priority

Comments

@pombredanne
Copy link
Member

pombredanne commented Mar 12, 2024 •
edited by JonoYang
Loading

See also:
@pombredanne pombredanne mentioned this issue Mar 12, 2024
Closed
3 tasks
@pombredanne pombredanne added the high priority High Priority label Apr 3, 2024
@JonoYang JonoYang mentioned this issue Apr 9, 2024
Closed
@JonoYang JonoYang mentioned this issue Apr 23, 2024
Closed
JonoYang added a commit that referenced this issue Apr 24, 2024
@JonoYang
Create ApproximateResourceContentIndex model #342
ca88dd4 
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Apr 24, 2024
@JonoYang
Add step to perform approximate matching #342
d44c06f 
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue Apr 25, 2024
@JonoYang
Add missing migration #342
c89e918 
    * Fix tests

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue May 10, 2024
@JonoYang
Create ApproximateResourceContentIndex model #342
3462680 
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue May 10, 2024
@JonoYang
Add step to perform approximate matching #342
62e32a9 
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue May 10, 2024
@JonoYang
Add missing migration #342
1f37508 
    * Fix tests

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue May 10, 2024
@JonoYang
Remove unused matchcode management commands #342
95f09ee 
    * Move match.py to match_test_utils.py as we are now just using those functions for testing match functionality than actually using it for matching

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue May 11, 2024
@JonoYang
Add match test for approximate file matching #342
c030f37 
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue May 14, 2024
@JonoYang
Ignore all variation of DS_Store files in .gitignore #342
bb19767 
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue May 14, 2024
@JonoYang
Update minecode.tests.test_indexing #342
6743044 
    * Use new test data

Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue May 14, 2024
@JonoYang
Add ApproximateResourceContentIndex model match test #342
ae69966 
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue May 14, 2024
@JonoYang
Replace test files #342
8f1f16d 
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue May 14, 2024
@JonoYang
Move minecode.management.indexing to minecode.indexing #342
95d5238 
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue May 14, 2024
@JonoYang
Add tests for matchcode model index methods #342
9711357 
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue May 14, 2024
@JonoYang
Sort imports #342
a567d48 
Signed-off-by: Jono Yang <jyang@nexb.com>
JonoYang added a commit that referenced this issue May 15, 2024
@JonoYang
Perform approximate directory matching after exact directory matching #…
52d49b0 
…342

Signed-off-by: Jono Yang <jyang@nexb.com>
@JonoYang
Copy link
Contributor

JonoYang commented May 16, 2024 •
edited
Loading

This has been merged into main. We have created a new table for storing approximate file fingerprints and updated the indexing functions to index those values from scans. We have also updated the fingerprinting functions in scancode-toolkit to also generate fingerprints for text files. With these changes, we now able to perform approximate file matching in the matching pipeline run by matchcode. https://github.com/nexB/purldb/blob/main/matchcode_pipeline/pipelines/matching.py#L109

Test instructions after installing and running PurlDB with it's accompanying ScanCode.io worker, and MatchCode.io:

  1. Index the package pkg:npm/deep-equal@1.0.1 using the api/collect/index_packages endpoint
  2. Using the api/matching/ endpoint, upload the test file https://github.com/nexB/purldb/blob/main/matchcode/tests/testfiles/match/approximate-file-matching/index-modified.js
  3. The package data for pkg:npm/deep-equal@1.0.1 should be present at api/matching/<uuid>/results

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JonoYang JonoYang

Labels
high priority High Priority
Projects
02-MatchCode - More matching
Status: Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pombredanne @JonoYang