Run dependency track with MySQL.

[preethamnagesh8]

Hi, I am trying to run dependency track using docker. Me being a newbie to docker am not sure of the below points:

1. Run dependency track with MySql Database.
2. The dependency track when run in a docker, currently with the default H2 database, shuts down automatically after some time (10mins).

I have referred to the documentation of Dependency Track and found a way to change the configurations so that MySql DB is used. But where do I change the data. As in, should the data be changed in the downloaded docker image ? (If yes plz specify the physical path of the file) or should the db connection be specified while running the docker.

Please help me with the above things.

[stevespringett]

But where do I change the data

What data?

If you followed the Docker instructions, you'll have a data volume that is independent of the container itself. Thats where the Dependency-Track stores its data.
https://docs.dependencytrack.org/getting-started/deploy-docker/

should the data be changed in the downloaded docker image

Under most situations, there should be no need to change anything in a running container. The majority of configuration uses environment variables which are passed to the container. For example, the variable ALPINE_DATABASE_URL is used to specify the JDBC URL.

A sample docker compose file is provided in the documentation and is checked into version control. It's available here: https://github.com/DependencyTrack/dependency-track/blob/master/src/main/docker/docker-compose.yml You can use this file as a starting point, uncomment and change the variable to match your environment. That's likely the easiest way to get a production-ready instance deployed.

[preethamnagesh8]

Hi Steve,
Thank you for the reply. I understand that the yml file needs to be modified as per the requirements. But where do I find the yml file ? or how do I create one and pass it to the docker container ?

[stevespringett]

1. Create a directory called dtrack (or similar) on your Docker host machine
2. Download the raw compose file:
   https://raw.githubusercontent.com/DependencyTrack/dependency-track/master/src/main/docker/docker-compose.yml to the directory you just created.
3. Modify it to suite your environment
4. Execute docker-compose up & in the dtrack directory to start the container. Initial startup results in the database being seeded. This usually takes between 10 and 30 minutes. Wait for the NPM and NVD mirroring to complete before you attempt to use it or shut it down.
5. Execute docker-compose down & in the dtrack directory to shut down the container

[preethamnagesh8]

Hi Steve,
Thank you very much for the reply. Appreciate the quick response. Will try this out and revert back to you.

[preethamnagesh8]

Hi Steve,
I tried the yml method of providing mysql configs. The yml file is as follows:

Database Properties

 - ALPINE_DATABASE_MODE=external
 - ALPINE_DATABASE_URL=jdbc:mysql://localhost:3306/dtrack?autoReconnect=true&useSSL=false
 - ALPINE_DATABASE_DRIVER=com.mysql.jdbc.Driver
 - ALPINE_DATABASE_DRIVER_PATH=mysql-connector-java-5.1.48-bin.jar
 - ALPINE_DATABASE_USERNAME=root
 - ALPINE_DATABASE_PASSWORD=root
 - ALPINE_DATABASE_POOL_ENABLED=true
 - ALPINE_DATABASE_POOL_MAX_SIZE=10
 - ALPINE_DATABASE_POOL_IDLE_TIMEOUT=600000
 - ALPINE_DATABASE_POOL_MAX_LIFETIME=600000

When I execute docker-compose, I'm getting the following error:

ERROR [Datastore] Exception thrown creating StoreManager : The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.

I have the file 'mysql-connector-java-5.1.48-bin.jar' downloaded. Unable to find where to place this file. Any help would be appreciated on this.

[preethamnagesh8]

Hi Steve,
I tried the yml method of providing mysql configs. The yml file is as follows:

Database Properties

 - ALPINE_DATABASE_MODE=external
 - ALPINE_DATABASE_URL=jdbc:mysql://localhost:3306/dtrack?autoReconnect=true&useSSL=false
 - ALPINE_DATABASE_DRIVER=com.mysql.jdbc.Driver
 - ALPINE_DATABASE_DRIVER_PATH=mysql-connector-java-5.1.48-bin.jar
 - ALPINE_DATABASE_USERNAME=root
 - ALPINE_DATABASE_PASSWORD=root
 - ALPINE_DATABASE_POOL_ENABLED=true
 - ALPINE_DATABASE_POOL_MAX_SIZE=10
 - ALPINE_DATABASE_POOL_IDLE_TIMEOUT=600000
 - ALPINE_DATABASE_POOL_MAX_LIFETIME=600000

When I execute docker-compose, I'm getting the following error:

ERROR [Datastore] Exception thrown creating StoreManager : The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.

I have the file 'mysql-connector-java-5.1.48-bin.jar' downloaded. Unable to find where to place this file. Any help would be appreciated on this.

[preethamnagesh8]

I was able to set the path for the driver. It was mentioned in the documentation website. Although the above mentioned issue was solved, when I started the docker, It starts downloading the NVD and NPM data and after some time, crashes with code 137. Googling this led me to a conclusion that this might be an out of memory issue.

1. Does this issue happen when the program is run on docker, and on a PC ?
2. Is running the program on a more powerful machine the only solution ?
3. Am i missing something ?

[stevespringett]

The Dependency-Track requires a minimum of 4GB and 2CPU cores to run, however, more is better. If the Docker host is not able to provide this, you'll get some unusual Docker errors related to resources.

[preethamnagesh8]

My system has 16GB of Ram and 8 cores. This shouldn't have been an issue. Also I read that the default allocated resources for a docker are as much as the OS provides. So, this shouldn't have been an issue at all. I am running the docker on a PC laptop. I have a couple of questions

1. Does the program initially fetch all the data, or is it possible to fetch them in batches ?
2. Has the same issue been faced by people trying to run dependency check in a laptop ?
3. Any suggestions for this ?

[stevespringett]

1. Dependency-Track seeds its database by iterating in small batches. Multiple batches for NPM Advisories, and multiple batches for the NVD.
2. Dependency-Check? I assume you mean Dependency-Track. No. I am not aware of anyone else having this issue. I develop Dependency-Track on a MacBook Pro with 16GB and 4 cores. I have a similar spec'ed Linux machine that I use for continuous testing as well.
3. You might want to verify the amount of RAM the host OS is allocating to the Docker container. If the host is running other containers or you have any other resource intensive applications open, the container will not get the resources it needs. This error typically occurs when the memory specified by -Xmx (which is 4GB) cannot be allocated even though Docker told the container it could be. It's usually a host OS issue.

[preethamnagesh8]

Hi Steve,
I was able to change the memory allocation to the docker successfully. The docker now doesn't crash with a status 137. But, while the docker executes the codes (the output can be seen in console), I see couple of exceptions. I also downgraded the MySql version to a different one (5.6). Still the errors appear.

1. Error :

com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '"SCHEMAVERSION"' at line 1

2. Error:

07:22:56.399 ERROR [Datastore] An exception was thrown while adding/validating class(es) : Specified key was too long; max key length is 767 bytes dtrack_1 | com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was too long; max key length is 767 bytes

3. Error:

07:24:19.459 ERROR [LoggableUncaughtExceptionHandler] An unknown error occurred in an asynchronous event or notification thread dtrack_1 | java.lang.NullPointerException: null dtrack_1 | at org.dependencytrack.tasks.scanners.BaseComponentAnalyzerTask.isEnabled(BaseComponentAnalyzerTask.java:63).

1. Is this because of the MySql version ?
2. What is the correct version of docker, MySql needed (considering the jdbc connector version : mysql-connector-java-5.1.47.jar) ?
3. Any other suggestions ?

[stevespringett]

MySQL doesn't work out-of-the-box without configuration. By default MySQL does non-standard things that break applications expecting a database to behave in a specific way. Most orgs that I've talked with either use Postgres or Microsoft SQL.

Ensure that the sql_mode is specified for the container. Otherwise it will not work. Not sure about the key length error. That's a new one that I haven't seen. But there's nothing I can do about key length. That might be yet another configuration issue with MySQL. Not entirely sure.

https://docs.dependencytrack.org/getting-started/database-support/

[preethamnagesh8]

Hi Steve,
Thanks for the advice. It was indeed an issue with sql_mode. Now the errors dont appear. But I'm facing another issue.

1. The nvd and npm data is mirrored.
2. Parallely, the logs show an error trying to connect to dependency-check with the following error:

ERROR [LoggableUncaughtExceptionHandler] An unknown error occurred in an asynchronous event or notification thread dtrack_1 | org.owasp.dependencycheck.data.nvdcve.DatabaseException: Unable to connect to the database

Does the dep track application need dependency check installed ?

3. After the appearence of the above error, the nvd, npm mirroring continues and completes, post which I upload a bom file generated from cyclonedx (npm package). On upload of this, an error occurs saying unable to parse the xml file.

Please help me with the above issues. And thanks for the quick reply everytime.

[stevespringett]

Because of the issue with the database and likely several DT restarts, I would highly recommend starting from a clean database and data directory.

Dependency-Check does not need to be installed. DT v3.5.1 and previous includes an embedded Dependency-Check core library for fuzzy NVD matching. This is being removed in v3.6.0. But it's not necessary to do anything. it does however, assume that the database and the data directory have been properly seeded. So starting from a fresh database and data volume would likely solve any remaining issues.

[preethamnagesh8]

Hi Steve,
Your suggestion was right. I started running the project from a clean state, and it worked fine. Facing another issue. Suppose I use a product and want to understand its vulnerabilities, then I create a new component and add the new product there.

Ex: Gimp Version: 2.10.0. This version of Gimp shows vulnerabilities when searched in NVD database, but the same when added as a new component, doesn't show any vulnerabilities associated. The steps followed for the creation of component:

1. Reached components page from left menu.
2. Clicked on Create Component
3. In the General Tab, added name as Gimp
4. Set version as 2.10.0 and clicked create.
5. After reloading the page, and after the triggering a refresh, the vulnerabilities column for that newly created component shows '0' with blue background.

I was expecting a couple of vulnerabilities to show up because the same showed up in NVD database. Could you help me understand what is happening here ?

[stevespringett]

Dependency-Track v3.5.1 and lower excels at discovering library vulnerabilities. Asset vulnerability (applications, operating systems, hardware) support is currently limited. Major work in this area is currently happening on v3.6.0.

With that said, if you're manually adding components, you need to ensure the group, name, version align to what the NVD states in the CPE. The CPE for this version of gimp is: cpe:2.3:a:gimp:gimp:2.10.0:*:*:*:*:*:*:*, so you,ll need to use 'gimp' for both the group and name of the component. It should work if this is done. If not, then you may have to wait for 3.6.0 to track assets and their vulnerabilities.

[preethamnagesh8]

Hi Steve,
Thank you for the reply. The mistake I was doing was that the component name, version and vendor was not as per the CPE. Referred the CPE for tools details and it worked fine.

1. Is there a timeline when dependency track 3.6.0 would be released ?
2. Once the BOM file is uploaded and the output for the project is generated, is there any way to download an Excel or a PDF file of the vulnerabilities ? Is there any open source reporting tool to manage vulnerabilities that integrates with Dependency Track ?
3. In the present version is there a possibility of generating a BOM file for Asset Vulnerabilities, because adding the components manually is quite a tedious task mainly because it has to fall in line with the CPE specification.

[stevespringett]

Follow the 3.6.0 milestone for updates and progress. There isn’t an eta at the moment.
https://github.com/DependencyTrack/dependency-track/milestone/10

Outputting of excel or pdf is not something this project will ever do, as it goes against the goal of providing continuous metrics. You can however use the data from the various APIs to produce whatever document format you want. Integration with BI tools is elementary.

[preethamnagesh8]

Hi Steve,
I am trying to build a reporting tool that exports the vulnerability data. It does so by making an API call to Dep Track and fetch the data and generate report. I am using docker compose file for the same.

- ALPINE_CORS_ENABLED=true - ALPINE_CORS_ALLOW_ORIGIN=* - ALPINE_CORS_ALLOW_METHODS=* - ALPINE_CORS_ALLOW_HEADERS=* - ALPINE_CORS_EXPOSE_HEADERS=* - ALPINE_CORS_ALLOW_CREDENTIALS=true - ALPINE_CORS_MAX_AGE=3600

Above is the code for CORS enabling. Below is the angular code for making an API call:

const header = { headers: new HttpHeaders({ 'X-Api-Key': 'xxxx ....... xxxx' }) } return this.httpClient.get('http://localhost:80/api/v1/project', header);

After this, there is an error thrown which says (failed) to make request in chrome. While in mozilla, it makes an API call to OPTIONS method and says 401 unauthorized.

Mozilla Output:
Request method:OPTIONS Remote address:127.0.0.1:80 Status code: 401 Version:HTTP/1.1 Referrer Policy:no-referrer-when-downgrade

Chrome output:
It just says (failed).

Could you please tell me what could be going wrong here.

[stevespringett]

Dependency-Track provides an API that others can integrate with. The project (and myself) does not have the expertise or capacity to address integration-specific implementation details. I have no idea why a 401 is returned, nor do I know anything about angular.

Your code is specifying httpClient.get with the header, but if the same header doesn't exist when calling options, then yes, you'll likely either get a 401 or similar. I don't think there are any endpoints that support OPTIONS. So you'll likely need to figure out why thats happening and prevent it.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.