PURLDB: Ensure Maven matching works

[pombredanne]

We need toensure that we have Maven mining and matching for exact JAR working:

• Have Java matching working end-to-end integrated in SCIO as an easy to reuse pipe
• using the latest, up-to-date latest Maven central
• possibly using later alternative Maven repos

[JonoYang]

I've set up purldb and scancode.io for Maven matching by:

0. Getting the latest versions of purldb and scancode.io
1. Build purldb/matchcode-toolkit wheel and modify scancode.io Dockerfile to install it
2. Running docker compose build to build docker containers for both repos
3. Setting up the .env files
   /etc/purldb/.env

SECRET_KEY=<key>
ALLOWED_HOSTS=192.168.1.12
CSRF_TRUSTED_ORIGINS=http://192.168.1.12
SCANCODEIO_URL="http://192.168.1.29/api/"

/etc/scancodeio/.env

SECRET_KEY=<key>
ALLOWED_HOSTS=192.168.1.29
CSRF_TRUSTED_ORIGINS=http://192.168.1.29
PURLDB_URL=http://192.168.1.12/api/

4. Run webservers

• For purldb: docker compose --profile visit_and_map up and docker compose --profile scan_queue up
• For scancode.io: docker compose up

Issue experienced:

• Invalid SHA1 values, initial guess is that the values in the sha1 fields are variables

WARNING:minecode.models:ResourceURI.normalize_fields() for URI: "maven-index://repo1.maven.org/xerces/xercesImpl/2.9.0/xercesImpl-2.9.0.jar" - Invalid SHA1 length: "SHA1(xercesImpl-2.9.0.jar)=": SHA1 ignored!
WARNING:minecode.models:ResourceURI.normalize_fields() for URI: "maven-index://repo1.maven.org/org/apache/httpcomponents/jakarta-httpcore-nio/4.0-alpha3/jakarta-httpcore-nio-4.0-alpha3.jar" - Invalid SHA1 length: "83c7dc2ba364d00d4e3f4db150d831ec": SHA1 ignored!
WARNING:minecode.models:ResourceURI.normalize_fields() for URI: "maven-index://repo1.maven.org/org/apache/axis2/axis2-transport-xmpp/1.0.0/axis2-transport-xmpp-1.0.0-sources.jar" - Invalid SHA1 length: "axis2-transport-xmpp-1.0.0-sources.jar:": SHA1 ignored!

• Scan queue needs to be revamped such that we can have multiple scans going, see Overhaul needed for Package scan requests and indexing #49

• Maven seed URI needs to be modified if we want to revisit the maven index, or we set minecode.visitor.maven.MavenSeed.revisit_after to 0 and then rebuild the container

• scancode.io projects would fail because the server would run out of ram. I had to use 20+ gb for 6 cores to get it to run alright.

• Match results are iffy when you start to mix jar contents together