deltacode doesn't support the new json format of scancode over 3.0

[macrovve]

When I used the json file generate by scancode 3.0.2, It raise an exception.

$ ./deltacode -n samples/samples.json -o samples/samples-of-scancode-3.0.json -j output.json
Traceback (most recent call last):
  File "/Users/macrovve/Workspace/deltacode-1.0.0/bin/deltacode", line 11, in <module>
    load_entry_point('deltacode', 'console_scripts', 'deltacode')()
  File "/Users/macrovve/Workspace/deltacode-1.0.0/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/macrovve/Workspace/deltacode-1.0.0/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/macrovve/Workspace/deltacode-1.0.0/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/macrovve/Workspace/deltacode-1.0.0/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/macrovve/Workspace/deltacode-1.0.0/src/deltacode/cli.py", line 89, in cli
    deltacode = DeltaCode(new, old, options)
  File "/Users/macrovve/Workspace/deltacode-1.0.0/src/deltacode/__init__.py", line 51, in __init__
    self.old = Scan(old_path)
  File "/Users/macrovve/Workspace/deltacode-1.0.0/src/deltacode/models.py", line 46, in __init__
    if not self.is_valid_scan(path):
  File "/Users/macrovve/Workspace/deltacode-1.0.0/src/deltacode/models.py", line 90, in is_valid_scan
    raise ScanException(msg)
deltacode.models.ScanException: JSON file 'samples/samples-of-scancode-3.0.json' is missing the 'scancode_version' attribute.

And here are the head of two sample file:

# the sample json provide by deltacode
{
  "scancode_notice": "Generated with ScanCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nScanCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nScanCode is a free software code scanning tool from nexB Inc. and others.\nVisit https://github.com/nexB/scancode-toolkit/ for support and download.",
  "scancode_version": "2.2.1.post259.4c5233dcb",
  "scancode_options": {
    "input": "samples/",
    "--copyright": true,
    "--info": true,
    "--json-pp": "/Users/sesser/Desktop/samples.json",
    "--license": true,
    "--package": true,
    "--processes": "3"
  },
  "files_count": 33,

# new sample json generated by scancode 3.0.2
{
  "headers": [
    {
      "tool_name": "scancode-toolkit",
      "tool_version": "3.0.2",
      "options": {
        "input": "samples/zlib/",
        "--copyright": true,
        "--info": true,
        "--json-pp": "-",
        "--license": true,
        "--package": true,
        "--processes": "3"
      },
      "notice": "Generated with ScanCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nScanCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nScanCode is a free software code scanning tool from nexB Inc. and others.\nVisit https://github.com/nexB/scancode-toolkit/ for support and download.",
      "start_timestamp": "2019-04-07T024626.059587",
      "end_timestamp": "2019-04-07T024635.259788",
      "message": null,
      "errors": [],
      "extra_data": {
        "files_count": 16
      }
    }
  ],

And here are the source code. The reason why raise exception is the scancode_version has been change to tool_version in scancode 3.0+.

def is_valid_scan(self, location):
        """
        In conjunction with test_deltacode.py, test the incoming files to
        ensure that they are well-formed JSON and otherwise satisfy the
        requirements to be run in DeltaCode (e.g., ScanCode version, proper
        ScanCode options etc.).
        """
        # TODO: handle this exception during #171
        try:
            scan = json.loads(open(location).read())
        except IOError:
            return

        scan = json.loads(open(location).read())

        if not scan.get('scancode_version'):
            msg = ('JSON file \'' + location + '\' is missing the \'scancode_version\' attribute.')
            raise ScanException(msg)

        if int(scan.get('scancode_version').split('.').pop(0)) < 2:
            msg = ('JSON file \'' + location + '\' was created with an old version of ScanCode.')
            raise ScanException(msg)

        if not scan.get('scancode_options').get('--info'):
            msg = ('JSON file \'' + location + '\' is missing the \'scancode_options/--info\' attribute.')
            raise ScanException(msg)

        return True

but I cannot find #171 information in TODO. it seems easy to fix. Can I work on it?

[pombredanne]

@macrovve excellent catch! You surely can fix it and that will be mucho welcomed!

[steven-esser]

#123 merged, closing this.

Sorry if I stepped on your toes here @macrovve, but I needed these changes quickly for an internal project.

[Pratikrocks]

@pombredanne @MaJuRG is there any way to use deltacode for comparing the scans of two json from scancode over 3.0??

[steven-esser]

@Pratikrocks it should. What errors are you getting?

[Pratikrocks]

this is the error i was receiving deltacode.models.ScanException: JSON file 'samples/test1.json' is missing the 'scancode_version' attribute.

and i used delta code via the command deltacode -n samples/test1.json -o samples/char_modifier.json -j output.json

the headers for -n file is
headers": [ { "tool_name": "scancode-toolkit", "tool_version": "3.1.1", "options": { "input": [ "test.cpp" ], "--copyright": true, "--info": true, "--json-pp": "test1.json", "--license": true, "--package": true }, "notice": "Generated with ScanCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nScanCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nScanCode is a free software code scanning tool from nexB Inc. and others.\nVisit https://github.com/nexB/scancode-toolkit/ for support and download.", "start_timestamp": "2020-03-12T042208.876901", "end_timestamp": "2020-03-12T042211.042702", "message": null, "errors": [], "extra_data": { "files_count": 1 } } ]

and the header for the -o file path is
"headers": [ { "tool_name": "scancode-toolkit", "tool_version": "3.1.1", "options": { "input": [ "char_modifier.ipynb" ], "--copyright": true, "--info": true, "--json-pp": "char_modifier.json", "--license": true, "--package": true }, "notice": "Generated with ScanCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nScanCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nScanCode is a free software code scanning tool from nexB Inc. and others.\nVisit https://github.com/nexB/scancode-toolkit/ for support and download.", "start_timestamp": "2020-03-12T042221.164276", "end_timestamp": "2020-03-12T042223.422432", "message": null, "errors": [], "extra_data": { "files_count": 1 } } ]

[Pratikrocks]

@MaJuRG Upon scanning with scancode 3.X the scancode attribute is missing

[steven-esser]

ill take a look, thanks for this

[Pratikrocks]

@MaJuRG then i need to use scancode version 2.x for testing the functionalities of deltacode?

[steven-esser]

@Pratikrocks I have just tested DeltaCode and it works fine on Scancode v3 scans. I am not sure what issue you are experiencing. Perhaps you have an old version of deltacode? I would run off the develop branch.

[Pratikrocks]

@MaJuRG sir I downloaded the .zip file from
https://github.com/nexB/deltacode/releases/tag/v1.0.0

[steven-esser]

@Pratikrocks Good point. That release is a bit out of date. I will cut an updated release shortly.

For now, you should clone this repo via git and run deltacode off the latest develop branch.

[Pratikrocks]

@MaJuRG okay sir