Add ubuntu usn importer and it's tests

[sbs2001]

Fixes #212
Data is imported from https://usn.ubuntu.com/usn-db/database-all.json.bz2
Signed-off-by: Shivam Sandbhor shivam.sandbhor@gmail.com

[haikoschol]

I was wondering what the status of this PR is. Is it WIP? I just ran the importer and got the following exception:

(venv) hs@x280 vulnerablecode $ ./manage.py import ubuntu_usn
Importing data from ubuntu_usn
Traceback (most recent call last):
  File "./manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/home/hs/code/vulnerablecode/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/home/hs/code/vulnerablecode/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/hs/code/vulnerablecode/venv/lib/python3.8/site-packages/django/core/management/base.py", line 328, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/hs/code/vulnerablecode/venv/lib/python3.8/site-packages/django/core/management/base.py", line 369, in execute
    output = self.handle(*args, **options)
  File "/home/hs/code/vulnerablecode/vulnerabilities/management/commands/import.py", line 74, in handle
    self.import_data(sources, options['cutoff_date'])
  File "/home/hs/code/vulnerablecode/vulnerabilities/management/commands/import.py", line 98, in import_data
    self._import_data(importers, cutoff_date)
  File "/home/hs/code/vulnerablecode/vulnerabilities/management/commands/import.py", line 104, in _import_data
    ImportRunner(importer, batch_size).run(cutoff_date=cutoff_date)
  File "/home/hs/code/vulnerablecode/vulnerabilities/import_runner.py", line 78, in run
    _process_updated_advisories(data_source)
  File "/home/hs/code/vulnerablecode/vulnerabilities/import_runner.py", line 109, in _process_updated_advisories
    vuln, _ = _get_or_create_vulnerability(advisory)
  File "/home/hs/code/vulnerablecode/vulnerabilities/import_runner.py", line 153, in _get_or_create_vulnerability
    vuln, created = models.Vulnerability.objects.get_or_create(**query_kwargs)
  File "/home/hs/code/vulnerablecode/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/hs/code/vulnerablecode/venv/lib/python3.8/site-packages/django/db/models/query.py", line 559, in get_or_create
    return self.get(**kwargs), False
  File "/home/hs/code/vulnerablecode/venv/lib/python3.8/site-packages/django/db/models/query.py", line 419, in get
    raise self.model.MultipleObjectsReturned(
vulnerabilities.models.MultipleObjectsReturned: get() returned more than one Vulnerability -- it returned more than 20!

[sbs2001]

@haikoschol

Can you check now ? For some reason I thought I had ran that importer.

For now I've fixed the issue by passing an empty string as the summary(USN summaries anyway don't fit with CVE summary). Do you remember awhile back I had similar problems and that fixed by doing passing empty string as you had suggested ? My point is, I don't really understand what's causing this issue.

[haikoschol]

@sbs2001

For now I've fixed the issue by passing an empty string as the summary(USN summaries anyway don't fit with CVE summary). Do you remember awhile back I had similar problems and that fixed by doing passing empty string as you had suggested ? My point is, I don't really understand what's causing this issue.

Works now. I haven't looked into it, but your description makes me think of the issue with nullable string columns. It sounds like for one vulnerability, multiple Vulnerability rows are written with null values in a column that has a unique constraint.