Scan a repository

How to scan a repository

1. Install the dependencies (possibly using a virtualenv)

2. Instantiate the client (either Postgres or sqlite)

   from credentialdigger import PgClient
   
   c = PgClient(dbhost='xxx.xxx.xxx.xxx', dbport=NUM, dbname='mydbname', dbuser='myusername', dbpassword='mypassword')

   or

   from credentialdigger import SqliteClient
   
   c = SqliteClient(path='/path/to/data.db')

3. [OPTIONAL] Add the repository

   c.add_repo(url='https://github.com/user/repo')

4. Launch the scan of the repo

   new_discoveries = c.scan(repo_url=REPO_URL,
                            category=CATEGORY,
                            scanner=SCANNER,
                            models=MODELS,
                            exclude=EXCLUDE_RULES,
                            force=FORCE,
                            generate_snippet_extractor=GENERATOR,
                            debug=DEBUG)

Arguments

• REPO_URL: the url of the repo we want to scan.
• CATEGORY: the category of rules to be used for the scan. If no category is selected, the scanner uses all the rules that are actually stored in the database
• MODELS: A list of models that we want to apply to auto-classify false positives (the models are applied in cascade, sequentially). If no models are specified, then do not use any. Refer to Models page to know more on models
• EXCLUDE_RULES: A list of ids of rules that we do not want to use. For instance, if we want to use all the keywords rules except the one with id=5, then we set CATEGORY equal to keywords and EXCLUDE_RULES equal to [5]. If this argument is not set, then use all the rules as explained in CATEGORY
• FORCE: True if we want to force the complete scan of a repository. Indeed, in case the repository has already been scanned, we would consider only the new commits
• GENERATOR: True if we want to generate an adapted extractor for the snippet model. This only works if the SnippetModel is in MODELS, and if there are still discoveries to classify when the time for the SnippetModel comes
• DEBUG: True if we want visual feedbacks (progress bars) when the scan is in progress, False otherwise (the default choice)

Output

new_discoveries is a list of ids of discoveries that have automatically been inserted into the db as new. If we set MODELS, then the discoveries classified as false positives are automatically updated in the db (as false_positive) without user intervention, and do not appear in new_discoveries.

5. new_discoveries are supposed to be analyzed manually by the user, and their state will be manually changed by the user.

   for disc_id in new_discoveries:
       this_discovery = c.get_discovery(disc_id)
       # Analyze it
       # Change its state (if needed)
       c.update(disc_id, 'new state')

   Refer to States for the states supported by the system.

Scan user's repositories

Credentialdigger also provides a method to scan all the repositories belonging to a user. The default scanner used for this process is the git scanner.

After instantiating the client, this scan can be run as follows:

new_repos_discoveries = c.scan_user(username=GITHUB_USERNAME,
                                    category=CATEGORY,
                                    models=MODELS,
                                    exclude=EXCLUDE_RULES,
                                    generate_snippet_extractor=GENERATOR,
                                    debug=DEBUG,
                                    forks=FORKS)

Arguments

• CATEGORY, MODELS, EXCLUDE_RULES, GENERATOR, and DEBUG work same as in scan method.
• USERNAME: the username as appearing on GitHub. All the repositories in this account will be considered for the scan
• FORKS: True if we want to scan also forked repositories, False otherwise (the default choice)

Output

new_repos_discoveries is a dictionary, where keys are urls of the repositories scanned, and, for each repository, its value is a list of ids of discoveries that have automatically been inserted into the db as new (i.e., the return value from the scan function).