Scan a repository

How to scan a repository

1. Install the dependencies (possibly using a virtualenv)

2. Instantiate the client (either Postgres or sqlite)

   from credentialdigger import PgClient
   
   c = PgClient(dbhost='xxx.xxx.xxx.xxx', dbport=NUM, dbname='mydbname', dbuser='myusername', dbpassword='mypassword')

   or

   from credentialdigger import SqliteClient
   
   c = SqliteClient(path='/path/to/data.db')

3. [OPTIONAL] Add the repository

   c.add_repo(url='https://github.com/user/repo')

4. Launch the scan of the repo

   new_discoveries = c.scan(repo_url=REPO_URL,
                            category=CATEGORY,
                            scanner=SCANNER,
                            models=MODELS,
                            exclude=NO_RULES,
                            force=FORCE,
                            generate_snippet_extractor=GENERATOR,
                            debug=DEBUG)

   Arguments:

   • REPO_URL: the url of the repo we want to scan.
   • CATEGORY: the category of rules to be used for the scan. If no category is selected, the scanner uses all the rules that are actually stored in the database
   • SCANNER: The name of the class to be used as a scanner. If no scanner is specified, then use HyperscanScanner. Refer to Scanners page to know more on scanners
   • MODELS: A list of models that we want to apply to auto-classify false positives (the models are applied in cascade, sequentially). If no models are specified, then do not use any. Refer to Models page to know more on models
   • NO_RULES: A list of ids of rules that we do not want to use. For instance, if we want to use all the keywords rules except the one with id=5, then we set CATEGORY equal to keywords and NO_RULES equal to [5]. If this argument is not set, then use all the rules as explained in CATEGORY
   • FORCE: True if we want to force the complete scan of a repository. Indeed, in case the repository has already been scanned, we would consider only the new commits
   • GENERATOR: True if we want to generate an adapted extractor for the snippet model. This only works if the SnippetModel is in MODELS, and if there are still discoveries to classify when the time for the SnippetModel comes
   • DEBUG: True if we want visual feedbacks (progress bars) when the scan is in progress, False otherwise (the default choice)

   Output:
   new_discoveries is a list of ids of discoveries that have automatically been inserted into the db as new. If we set MODELS, then the discoveries classified as false positives are automatically updated in the db (as false_positive) without user intervention, and do not appear in new_discoveries.

5. new_discoveries are supposed to be analyzed manually by the user, and their state will be manually changed by the user.

   for disc_id in new_discoveries:
       this_discovery = c.get_discovery(disc_id)
       # Analyze it
       # Change its state (if needed)
       c.update(disc_id, 'new state')

   Refer to States for the states supported by the system.