CLI: Command Line Interface

Credentialdigger offers some base functionalities with a command line interface. Obviously, you need to install the credentialdigger package first. You can refer to the README.md for this.

All the commands support both sqlite and postgres databases. In order to use sqlite you will need to set the path of the db as argument (--sqlite /path/to/data.db), whereas for postgres you can either export all the credentials as environment variables or pass an .env file as argument (more on this later).

• Download models
• Add rules
• Scan a repository
• Scan a user
• Scan wiki page

Download models

Download and link a machine learning model. Refer to Machine Learning Models for the complete explanation of how machine learning models work.

python -m credentialdigger download model_name

Add rules

Add the rules contained in a file, that will be used to scan a repository.

  path_to_rules       <Required> The path of the file that contains the rules.
  
  --sqlite DB_PATH    <Optional> If specified, use the sqlite client and
                      the db passed as argument (otherwise use postgres)

Sqlite:

# Add the rules to the database using sqlite
python -m credentialdigger add_rules /path/to/rules.yml --sqlite /path/to/mydata.db

Postgres:

# Add the rules to the database using postgres
export POSTGRES_USER=...
export ...
python -m credentialdigger add_rules /path/to/rules.yml

or

# Add the rules to the database using postgres and an environment file
python -m credentialdigger add_rules /path/to/rules.yml --dotenv /path/to/.env

TIP: if your env file is in the current directory and it's named .env, you don't need to specify the --dotenv parameter.

Scan a repository

The scan command allows to scan a git repo directly via the command line. It can accept multiple arguments:

  repo_url              <Required> The URL of the git repository to be
                        scanned.
  -h, --help            show this help message and exit
  --category CATEGORY   <Optional> If specified, scan the repo using all the
                        rules of this category, otherwise use all the rules in
                        the db
  --models MODELS [MODELS ...]
                        <Optional> A list of models for the ML false positives
                        detection. Cannot accept empty lists.
  --exclude EXCLUDE [EXCLUDE ...]
                        <Optional> A list of rules to exclude
  --force               <Optional> Force a complete re-scan of the repository.
                        Without it, in case the repository has already been 
                        scanned, we would consider only the new commits
  --debug               <Optional> Flag used to decide whether to visualize
                        the progressbars during the scan (e.g., during the
                        insertion of the detections in the db)
  --generate_snippet_extractor
                        <Optional> Generate the extractor model to be used in
                        the SnippetModel. The extractor is generated using the
                        ExtractorGenerator. If `False`, use the pre-trained
                        extractor model
  --git_token GIT_TOKEN
                        <Optional> Git personal access token to authenticate to 
                        the git server
  --sqlite DB_PATH      <Optional> If specified, use the sqlite client and
                        the db passed as argument
  --dotenv ENV_PATH     <Optional> If specified, use the postgres client and
                        the credentials contained in the file at `ENV_PATH`

Sqlite:

python -m credentialdigger scan https://github.com/user/repo --sqlite /path/to/mydata.db --models PathModel SnippetModel

Postgres:

export POSTGRES_USER=...  # either export variables or use --dotenv
python -m credentialdigger scan https://github.com/user/repo [--dotenv /path/to/my/.env] --models PathModel SnippetModel

Extras

The scan command also returns an exit status that is equal to the number of discoveries it has made during the scan. Here are two samples on how we can make use of the exit status.

Bash script

python -m credentialdigger scan https://github.com/user/repo
# $? = exit status
if [ $? -gt 0 ]; then
    echo "This repo contains leaks"
else
    echo "This repo contains no leaks"
fi

Java program

public class credentialdigger{
    public static void main(String[] args) {
        String command = "python -m credentialdigger scan https://github.com/user/repo";
        try {
            Process p = Runtime.getRuntime().exec(command);
            p.waitFor();
            int numberOfDiscoveries = p.exitValue();
            if(numberOfDiscoveries>0){
                System.out.println("This repo contains leaks.");
            }
            else{
                System.out.println("This repo contains no leaks.");
            }
        } catch (Exception e) {
            //IGNORE
        }
    }
}

Scan a user

Scan all the public repositories of a user. The arguments are the same as in scan plus the following:

  --forks               Scan also repositories forked by this user
  --api_endpoint API_ENDPOINT
                        API endpoint of the git server

Sqlite:

python -m credentialdigger scan_user username --sqlite /path/to/mydata.db --models PathModel SnippetModel

Postgres:

export POSTGRES_USER=...  # either export variables or use --dotenv
python -m credentialdigger scan_user username [--dotenv /path/to/my/.env] --models PathModel SnippetModel

Scan wiki page

Scan the wiki page of a project. All the arguments are the same as in scan.

Sqlite:

python -m credentialdigger scan_wiki https://github.com/user/repo --sqlite /path/to/mydata.db

Postgres:

export POSTGRES_USER=...  # either export variables or use --dotenv
python -m credentialdigger scan_wiki https://github.com/user/repo [--dotenv /path/to/my/.env]