Improve ABOUT file handling in d2d pipeline

[pombredanne]

1. It may take a significant time to map now that we are using patterns.
   Using a query regex and possibly tow conditions with ignores/excludes is likely to be slow. I think we may end up having multiple full table scans in a loop. This main explain why this pipeline step now runs about 5 to 12 times slower than before. I suggest to invert the processing. Right now, we process here this way: https://github.com/nexB/scancode.io/blob/c29c62884a383ae00cb67f5c1e03166e548e4056/scanpipe/pipes/d2d.py#L856C1-L856C1

• for each ABOUT file
  • for each resource matching ABOUT file paths and excludes
    • .....
  • create/update Package

Instead we could do this, ensuring we ever do a single pass on the resources:

• for each ABOUT file
  • collect path patterns and translate and compile to regex
• for each resource:
  • for each ABOUT file pattern
    • if resource matching ABOUT file paths and excludes
      • accumulate Resource for a later bulk operation
    • .....
• for each mapped ABOUT file:
  • create/update Package
  • bulk create the Package -> Resources relationships
• for each non-mapped:
  • save warning

2. Companion files should be taken from the ABOUT file, not implied based on naming conventions this should be using the ABOUT file content

   scancode.io/scanpipe/pipes/d2d.py

   Line 859 in c29c628

   about_file_companions = (

[AyanSinhaMahapatra]

#982 addresses this issue partially, i.e. the About file handling is much more improved and takes less time.
See the comparison for improvement in time taken (for a project with ~800k files, ~550k in deployed, ~220 about files) to process About files:
Before About paths were patterns:

2023-12-09 10:26:55.30 Step [map_about_files] starting
2023-12-09 10:26:55.33 Mapping 221 .ABOUT files found in the from/ codebase.
2023-12-09 11:11:58.06 Step [map_about_files] completed in 2703 seconds (45.0 minutes)

After About paths were patterns this increased to 4 hours, sometimes even 6-7 hours.

2023-11-15 16:20:07.50 Step [map_about_files] starting
2023-11-15 16:20:07.50 Mapping 220 .ABOUT files found in the from/ codebase.
2023-11-15 22:57:41.66 Step [map_about_files] completed in 23854 seconds (6.6 hours)

After the optimizations used here, this is now ~13 minutes:

2024-01-12 18:58:56.50 Step [map_about_files] starting
2024-01-12 18:58:56.55 Mapping 221 .ABOUT files found in the from/ codebase.
2024-01-12 19:11:39.79 Step [map_about_files] completed in 763 seconds (12.7 minutes)

Remaining: Companion files should be taken from the ABOUT file, not implied based on naming conventions this should be using the ABOUT file content
This has to be implemented in aboutcode-toolkit and then added here, could be done when #834 is implemented.

[pombredanne]

Remaining: Companion files should be taken from the ABOUT file, not implied based on naming conventions this should be using the ABOUT file content
This has to be implemented in aboutcode-toolkit and then added here, could be done when #834 is implemented.

@AyanSinhaMahapatra Why the wait? The attributes are already supported ... see https://github.com/nexB/aboutcode-toolkit/blob/89c16a5b762d38c5e7f4ba25097659bd60a0a08c/src/attributecode/model.py#L903