diff --git a/docs/detectors/README.md b/docs/detectors/README.md new file mode 100644 index 000000000..a08d566e4 --- /dev/null +++ b/docs/detectors/README.md @@ -0,0 +1,12 @@ +# Detectors + +- CocoaPods +- [Go](go.md) +- [Gradle](gradle.md) +- [Linux](linux.md) +- [Maven](maven.md) +- NPM +- NuGet +- [Pip](pip.md) +- Ruby +- Rust diff --git a/docs/detectors/go.md b/docs/detectors/go.md new file mode 100644 index 000000000..51b710e07 --- /dev/null +++ b/docs/detectors/go.md @@ -0,0 +1,34 @@ +# Go Detection + +## Requirements + +Go detection depends on the following to successfully run: + +- Go v1.11+. + +## Detection strategy + +Go detection is performed by parsing output from executing `go mod graph`. +Full dependency graph generation is supported if Go v1.11+ is present on the build agent. +If no Go v1.11+ is present, a fallback detection strategy is performed, dependent on: + +- One or more `go.mod` or `go.sum` files. + +For the fallback strategy: + +Go detection is performed by parsing any `go.mod` or `go.sum` found under the scan directory. + +Only root dependency information is generated instead of full graph. +I.e. tags the top level component or explicit dependency a given transitive dependency was brought by. +Given a dependency tree A -> B -> C, C's root dependency is A. + +## Known limitations + +Dev dependency tagging is not supported. + +Go detection will fallback if no Go v1.11+ is present. +If executing `go mod graph` takes too long (currently if it takes more than 10 seconds), go detection will fall back. +This can happen if modules are not restored before the scan. + +Due to the nature of `go.sum` containing references for all dependencies, including historical, no-longer-needed dependencies; the fallback strategy can result in over detection. +Executing `go mod tidy` before detection via fallback is encouraged. diff --git a/docs/gradle-detection.md b/docs/detectors/gradle.md similarity index 58% rename from docs/gradle-detection.md rename to docs/detectors/gradle.md index 0a38a9d41..0472b5f52 100644 --- a/docs/gradle-detection.md +++ b/docs/detectors/gradle.md @@ -1,16 +1,20 @@ # Gradle Detection + ## Requirements + Gradle detection depends on the following to successfully run: -- Gradle 7 or prior using [Single File lock](https://docs.gradle.org/6.8.1/userguide/dependency_locking.html#single_lock_file_per_project). -- One or more *.lockfile files. +- Gradle 7 or prior using [Single File lock](https://docs.gradle.org/6.8.1/userguide/dependency_locking.html#single_lock_file_per_project) +- One or more `.lockfile` files ## Detection strategy -Gradle detection is performed by parsing any *.lockfile found under the scan directory. + +Gradle detection is performed by parsing any `*.lockfile` found under the scan directory. ## Known limitations + Gradle detection will not work if lock files are not being used. -*Dev dependency* tagging is not supported. +Dev dependency tagging is not supported. -Full dependency graph generation is not supported. \ No newline at end of file +Full dependency graph generation is not supported. diff --git a/docs/detectors/linux.md b/docs/detectors/linux.md new file mode 100644 index 000000000..d6acc11e4 --- /dev/null +++ b/docs/detectors/linux.md @@ -0,0 +1,16 @@ +# Linux Detection + +## Requirements + +Linux detection depends on the following: + +- [Docker](https://www.docker.com/) + +## Detection strategy + +Linux package detection is performed by running [Syft](https://github.com/anchore/syft) and parsing the output. +The output contains the package name, version, and the layer of the container in which it was found. + +## Known limitations + +- Windows container scanning is not supported diff --git a/docs/maven-detection.md b/docs/detectors/maven.md similarity index 52% rename from docs/maven-detection.md rename to docs/detectors/maven.md index 0bd3801a2..3d873b421 100644 --- a/docs/maven-detection.md +++ b/docs/detectors/maven.md @@ -1,17 +1,21 @@ # Maven Detection + ## Requirements + Maven detection depends on the following to successfully run: - Maven CLI as part of your PATH. mvn should be runnable from a given command line. - Maven Dependency Plugin (installed with Maven). -- One or more *pom.xml* files. +- One or more `pom.xml` files. ## Detection strategy -Maven detection is performed by running *mvn dependency:tree -f {pom.xml}* for each pom file and parsing down the results. -Components tagged as a *test* dependency are marked as *development dependencies*. +Maven detection is performed by running `mvn dependency:tree -f {pom.xml}` for each pom file and parsing down the results. + +Components tagged as a test dependency are marked as development dependencies. Full dependency graph generation is supported. ## Known limitations -Maven detection will not run if *mvn* is unavailable. + +Maven detection will not run if `mvn` is unavailable. diff --git a/docs/pip-detection.md b/docs/detectors/pip.md similarity index 54% rename from docs/pip-detection.md rename to docs/detectors/pip.md index e8301ce58..0bb4f3e25 100644 --- a/docs/pip-detection.md +++ b/docs/detectors/pip.md @@ -1,13 +1,16 @@ # Pip Detection + ## Requirements + Pip detection depends on the following to successfully run: -- Python 2 or Python 3. -- Internet connection. -- One or more setup.py or *requirements.txt* files. +- Python 2 or Python 3 +- Internet connection +- One or more `setup.py` or `requirements.txt` files ## Detection strategy -Pip detection is performed by running the following code snippet on every setup.py: + +Pip detection is performed by running the following code snippet on every *setup.py*: ```python import distutils.core; @@ -15,19 +18,20 @@ Pip detection is performed by running the following code snippet on every se print(setup.install_requires); ``` -The code above allows Pip detection to detect any runtime dependendies. +The code above allows Pip detection to detect any runtime dependencies. -*requirements.txt* files are parsed; a Git component is created for every *git+* url. +`requirements.txt` files are parsed; a Git component is created for every `git+` url. -For every top level component, Pip detection makes http calls to Pip in order to determine latest version available, as well as to resolve the dependency tree by parsing the *METADATA* file on a given release's *bdist_wheel* or *bdist_egg*. +For every top level component, Pip detection makes http calls to Pip in order to determine latest version available, as well as to resolve the dependency tree by parsing the `METADATA` file on a given release's `bdist_wheel` or `bdist_egg`. Full dependency graph generation is supported. ## Known limitations -*Dev dependency* tagging is not supported. -Pip detection will not run if *python* is unavailable. +Dev dependency tagging is not supported. + +Pip detection will not run if `python` is unavailable. -If no *bdist_wheel* or *bdist_egg* are available for a given component, dependencies will not be fetched. +If no `bdist_wheel` or `bdist_egg` are available for a given component, dependencies will not be fetched. -If no internet connection or a component cannot be found in Pypi, said component and its dependencies will be skipped. \ No newline at end of file +If no internet connection or a component cannot be found in Pypi, said component and its dependencies will be skipped. diff --git a/docs/go-detection.md b/docs/go-detection.md deleted file mode 100644 index d7a67cde2..000000000 --- a/docs/go-detection.md +++ /dev/null @@ -1,27 +0,0 @@ -# Go Detection -## Requirements -Go detection depends on the following to successfully run: - -- Go v1.11+. - -## Detection strategy -Go detection is performed by parsing output from executing *go mod graph*. - -Full dependency graph generation is supported if Go v1.11+ is present on the build agent. - -If no Go v1.11+ is present, a fallback detection strategy is performed, dependent on: - -- One or more *go.mod* or *go.sum* files. - -For the fallback strategy: - -Go detection is performed by parsing any *go.mod* or *go.sum* found under the scan directory. - -Only root dependency information is generated instead of full graph. Ie. tags the top level component or explicit dependency a given transitive dependency was brought by. Given a dependency tree A -> B -> C, C's root dependency is A. - -## Known limitations -*Dev dependency* tagging is not supported. - -Go detection will fallback if no Go v1.11+ is present. If executing `go mod graph` takes too long (currently if it takes more than 10 seconds), go detection will fall back. This can happen if modules are not restored before the scan. - -Due to the nature of *go.sum* containing references for all dependencies, including historical, no-longer-needed dependencies; the fallback strategy can result in over detection. Executing *go mod tidy* before detection via fallback is encouraged. diff --git a/docs/linux-scanner.md b/docs/linux-scanner.md deleted file mode 100644 index 040ec20c9..000000000 --- a/docs/linux-scanner.md +++ /dev/null @@ -1,41 +0,0 @@ -# Developing on the Linux Scanner - -The Linux scanner uses a custom container image built from [this repository](https://github.com/Microsoft/LinuxScanning) we created to execute [tern](https://github.com/tern-tools/tern) on a target image. - -As of 6/26/2020, tern does not officially support scanning images in Windows so by extension, our Linux scanner does not either. - -There are 2 options for working on the Linux scanner: -* Set up and work in a virtual Linux environment -* Work with a limited set of images in Windows - -## The Linux way (Recommended) - -**NOTE**: Docker + WSL2 is not a shortcut for this :( The bind mount is still not support even if it's created in WSL2. (See the Windows way to udnerstand why it's necessary) - -1. [Using Hyper-V Quick Create](https://blogs.windows.com/windowsdeveloper/2018/09/17/run-ubuntu-virtual-machines-made-even-easier-with-hyper-v-quick-create/), create a Ubuntu virtual machine. Make sure you give it at least 30gb of storage, docker and the images use a lot of space. -2. Clone Component Detection in your VM -3. Install docker -4. Start developing! - -## The Windows way - -There are _some_ things the tern scanner can do on Windows. - -Limitations around scanning in Windows: - -* The image being scanned is a base OS image (eg: Ubuntu, Fedora, CentOS, etc.). That means no node, no python, definitely no custom Dockerfile images. -* Tern cannot attempt to create a bind mount to a folder - -How to do it: - -1. Install docker -2. Comment out the [OS platform check](https://github.com/microsoft/componentdetection-bcde/blob/d5743af1bda5a8f3b4eb96c08517b05513bf29d2/src/Detectors/linux/LinuxContainerDetector.cs#L42-L48) -3. Retarget the [scanner image](https://github.com/microsoft/componentdetection-bcde/blob/d5743af1bda5a8f3b4eb96c08517b05513bf29d2/src/Detectors/linux/LinuxScanner.cs#L22) to point to `tevoinea/ternwrapper:windows-latest` - * 3.1 **Alternatively**: - * Clone [LinuxScanning](https://github.com/microsoft/linuxscanning) - * Change the [default args](https://github.com/microsoft/LinuxScanning/blob/main/index.js#L6) to the set that doesn't contain the `-b` argument. `-b` means bind which we don't want on Windows - * Rebuild the image locally - * Follow step 3 again but point to your local image -4. Remember to only scan base OS images (scan will fail otherwise) -5. Remember to undo step 2 and 3 before PR -