Maven scopes - additional "AND" filter logic

Changelog.md

@@ -1,5 +1,8 @@
 # FOSSA CLI Changelog
 
+## v3.9.19
+- Adds the ability to more granularly control excluded Maven scopes ([#1427](https://github.com/fossas/fossa-cli/pull/1427)).
+
 ## v3.9.18
 - Resolves an issue where `vendored-dependencies` were rescanned locally, but not in the FOSSA service,
   when `forceRescans` was set to `true` ([#1423](https://github.com/fossas/fossa-cli/pull/1423)).

docs/references/files/fossa-yml.md

@@ -16,7 +16,7 @@ project:
   id: github.com/fossas/fossa-cli
   name: fossa-cli
   team: cli-team
-  teams: 
+  teams:
     - cli-team-1
     - cli-team-2
   policy: custom-cli-policy
@@ -36,7 +36,7 @@ releaseGroup:
   releaseGroupProjects:
     - projectLocator: custom+123/git@github.com/fossas/fossa-cli
       projectRevision: "12345"
-      projectBranch: master 
+      projectBranch: master
     - projectLocator: custom+123/git@github.com/example
       projectRevision: "67890"
       projectBranch: master
@@ -142,7 +142,7 @@ Default:
 
 >NOTE:
     A project's ID cannot be modified after a project is created. If you change the ID,
-    you will be interacting with a different project. If the new ID does not exist, 
+    you will be interacting with a different project. If the new ID does not exist,
     a new project will be created for it.
 
 <img src="../../assets/project-id-example.png">
@@ -193,7 +193,7 @@ The releaseGroup field allows you to configure settings for the release group yo
 
 >NOTE: releaseGroup configurations are only allowed for `fossa release-group create` and `fossa release-group add-project`. This is done so that release groups are not mistakenly deleted.
 
->NOTE: release-group command line options will always take precedence over configurations set in `fossa.yml`. 
+>NOTE: release-group command line options will always take precedence over configurations set in `fossa.yml`.
 
 #### `releaseGroup.title:`
 The title of the release group which can be seen in the FOSSA dashboard.
@@ -206,7 +206,7 @@ The release associated with the release group.
 <img src="../../assets//release-example.png">
 
 #### `releaseGroup.releaseGroupProjects:`
-The projects associated with the release group's release. 
+The projects associated with the release group's release.
 
 >NOTE: At least one project must be specified upon creating a release group.
 
@@ -396,31 +396,70 @@ You can provide maven dependency scopes that you would like to filter. You can f
 
 #### scope-only:
 
-The list of `only` scopes that should by scanned. 
+The list of `only` scopes that should by scanned.
+
+When a dependency is multi-scope (i.e. `[compile, runtime]`) ALL of the scopes must be contained in `scope-only` for the dependency to be included in the scan results.
+
+```yaml
+version: 3
+
+maven:
+  scope-only: # Only reports dependencies that are both 'compile' and 'runtime' scoped.
+    - compile
+    - runtime
+```
+
+For example, using the above setting:
+```
+Dependency { name: "A", scopes: [ "compile" ]}                    <- not reported, since it doesn't have both scopes.
+Dependency { name: "B", scopes: [ "compile", "runtime" ]}         <- reported, since it has both scopes.
+Dependency { name: "C", scopes: [ "compile", "runtime", "test" ]} <- reported, since it has both scopes.
+```
 
-When a dependency is multi-scope (i.e. [compile, runtime]) ALL of the scopes must be conatined in `scope-only` for the dependency to be included in the scan results. 
 
 #### scope-exclude:
 
 The list of `exclude` scopes that you would like to exclude from scanning.
 
-When a dependency is multi-scope (i.e. [compile, runtime]), if ANY of the scopes are contained in `scope-exclude` it will be excluded from the scan results.
+When a dependency is multi-scope (i.e. `[compile, runtime]`), by default if ANY of the scopes are contained in `scope-exclude` it will be excluded from the scan results:
 
 
 ```yaml
 version: 3
 
 maven:
-  scope-only:
-    - compile
-    - runtime
-## OR
-  scope-exclude:
+  scope-exclude: # Excludes dependencies that contain any of 'provided', 'system', or 'test' scopes.
     - provided
     - system
     - test
 ```
 
+For example, using the above setting:
+```
+Dependency { name: "A", scopes: [ "compile" ]}           <- reported, because it doesn't match any excluded scope.
+Dependency { name: "B", scopes: [ "test" ]}              <- not reported, because the scope "test" is excluded.
+Dependency { name: "C", scopes: [ "compile", "system" ]} <- not reported, because the scope "system" is excluded.
+```
+
+For more control, the items provided to `scope-exclude` can be arrays; when this is done it only filters the dependency if ALL of the scopes in that item are contained in the dependency.
+
+```yaml
+version: 3
+
+maven:
+  scope-exclude: # Excludes dependencies that contain the 'system' scope, or if they include both 'provided' and 'test' scopes.
+    - [provided, test]
+    - system
+```
+
+For example, using the above setting:
+```
+Dependency { name: "A", scopes: [ "compile" ]}                     <- reported, because it doesn't have any excluded scope.
+Dependency { name: "B", scopes: [ "test" ]}                        <- reported, because it doesn't have "provided" scope.
+Dependency { name: "C", scopes: [ "provided", "test", "compile" ]} <- not reported, because it has both "test" and "provided" scopes.
+Dependency { name: "B", scopes: [ "system" ]}                      <- not reported, because it has the "system" scope.
+```
+
 ## FAQ
 
 ### Why are some configuration settings (name, team, policy, etc.) ignored by the FOSSA API after a project has already been created?
@@ -440,12 +479,12 @@ In the CLI, project ID refers to a specific portion of a project locator. When r
 
 After project ID is set, the CLI will use that value to construct a project locator on first time analysis. A project locator is a unique ID that the FOSSA API will use to reference a project across FOSSA.
 
-> NOTE: Projects uploaded through the CLI will have `custom` embedded into the project locator. 
+> NOTE: Projects uploaded through the CLI will have `custom` embedded into the project locator.
 
 <img src="../../assets/project-locator-example.png">
 
 ### When do I use project ID vs project locator?
 
 Project ID should be used when referencing projects uploaded via the CLI. In most cases, providing just a project ID is sufficient. However, when referencing projects uploaded through avenues outside the CLI (Quick Import, Archive Upload, etc), providing just a project ID fails. This is due to the nature of FOSSA's project naming conventions as project locators are constructed differently depending on how a project was uploaded. There is no way to tell at runtime which project a user is referencing when only given a project ID (i.e. Is this a project uploaded via the CLI or Archive Upload?).
 
-In `fossa release-group` project locator is used to reference projects that will be added to your release group's release. By using project locator, all FOSSA project upload flows are covered and the CLI is able to determine the specific project a user is trying to add to a release. 
+In `fossa release-group` project locator is used to reference projects that will be added to your release group's release. By using project locator, all FOSSA project upload flows are covered and the CLI is able to determine the specific project a user is trying to add to a release.

docs/references/files/fossa-yml.v3.schema.json

@@ -548,11 +548,24 @@
                     }
                 },
                 "scope-exclude": {
+                  "type": "array",
+                  "description": "A dependency is excluded if it matches any of these filters.",
+                  "items": {
                     "type": "array",
-                    "description": "The list of scopes to exclude from scanning in your maven project.",
                     "items": {
-                        "type": "string"
+                      "oneOf": [
+                        {
+                          "type": "string",
+                          "description": "A dependency is excluded if it has this scope."
+                        },
+                        {
+                          "type": "array",
+                          "description": "A dependency is excluded if it has at least all the scopes in this array.",
+                          "items": { "type": "string" }
+                        }
+                      ]
                     }
+                  }
                 }
             },
             "oneOf": [
@@ -572,4 +585,4 @@
     "required": [
         "version"
     ]
-}
+}

integration-test/Analysis/FixtureUtils.hs

@@ -132,7 +132,7 @@ testRunner f env =
     & withDefaultLogger SevWarn
     & runReader (mempty :: OverrideDynamicAnalysisBinary)
     & runReader (mempty :: AllFilters)
-    & runReader (MavenScopeIncludeFilters mempty)
+    & runReader (MavenScopeOnlyFilters mempty)
     & runReader (ExperimentalAnalyzeConfig Nothing GoModulesBasedTactic False)
     & runFinally
     & runStack

src/App/Fossa/Config/Analyze.hs

@@ -97,7 +97,7 @@ import Data.String.Conversion (
   ToText (toText),
  )
 import Data.Text (Text)
-import Discovery.Filters (AllFilters (AllFilters), MavenScopeFilters (MavenScopeIncludeFilters), comboExclude, comboInclude)
+import Discovery.Filters (AllFilters (AllFilters), MavenScopeFilters (MavenScopeOnlyFilters), comboExclude, comboInclude)
 import Effect.Exec (
   Exec,
  )
@@ -548,7 +548,7 @@ collectMavenScopeFilters ::
   Maybe ConfigFile ->
   m MavenScopeFilters
 collectMavenScopeFilters maybeConfig =
-  pure $ maybe (MavenScopeIncludeFilters mempty) collectConfigMavenScopeFilters maybeConfig
+  pure $ maybe (MavenScopeOnlyFilters mempty) collectConfigMavenScopeFilters maybeConfig
 
 collectFilters ::
   ( Has Diagnostics sig m

src/App/Fossa/Config/Common.hs

@@ -68,6 +68,7 @@ import App.Fossa.Config.ConfigFile (
   ConfigTelemetry (telemetryScope),
   ConfigTelemetryScope (..),
   MavenScopeConfig (..),
+  MavenScopePredicate (..),
   mergeFileCmdMetadata,
  )
 import App.Fossa.Config.EnvironmentVars (EnvVars (..))
@@ -109,11 +110,13 @@ import Data.Aeson (ToJSON (toEncoding), defaultOptions, genericToEncoding)
 import Data.Bifunctor (Bifunctor (first))
 import Data.Functor.Extra ((<$$>))
 import Data.Maybe (fromMaybe)
+import Data.Set (Set)
+import Data.Set qualified as Set
 import Data.String (IsString)
 import Data.String.Conversion (ToText (toText))
 import Data.Text (Text, null, strip, toLower)
 import Diag.Result (Result (Failure, Success), renderFailure)
-import Discovery.Filters (AllFilters (AllFilters), MavenScopeFilters (..), comboExclude, comboInclude, setExclude, setInclude, targetFilterParser)
+import Discovery.Filters (AllFilters (..), MavenScopeFilterPredicate (..), MavenScopeFilters (..), comboExclude, comboInclude, targetFilterParser)
 import Effect.Exec (Exec)
 import Effect.Logger (Logger, logDebug, logInfo, renderIt, vsep)
 import Effect.ReadFS (ReadFS, doesDirExist, doesFileExist)
@@ -574,7 +577,14 @@ collectConfigMavenScopeFilters :: ConfigFile -> MavenScopeFilters
 collectConfigMavenScopeFilters configFile = do
   let maybeMavenScopeConfigs = configMavenScope configFile
   case maybeMavenScopeConfigs of
-    Nothing -> MavenScopeIncludeFilters mempty
+    Nothing -> MavenScopeOnlyFilters mempty
     Just mavenScopeConfig -> case mavenScopeConfig of
-      MavenScopeOnlyConfig filters -> MavenScopeIncludeFilters $ setInclude filters
-      MavenScopeExcludeConfig filters -> MavenScopeExcludeFilters $ setExclude filters
+      MavenScopeOnlyConfig filters -> MavenScopeOnlyFilters filters
+      MavenScopeExcludeConfig filters -> MavenScopeExcludeFilters $ translatePredicates filters
+  where
+    translatePredicates :: Set MavenScopePredicate -> Set MavenScopeFilterPredicate
+    translatePredicates preds = Set.fromList . fmap translatePredicate $ Set.toList preds
+
+    translatePredicate :: MavenScopePredicate -> MavenScopeFilterPredicate
+    translatePredicate (MavenScopePredicateSingle a) = MavenScopeFilterPredicateSingle a
+    translatePredicate (MavenScopePredicateCombined a) = MavenScopeFilterPredicateCombined a

src/App/Fossa/Config/ConfigFile.hs

@@ -17,6 +17,7 @@ module App.Fossa.Config.ConfigFile (
   ExperimentalGradleConfigs (..),
   VendoredDependencyConfigs (..),
   MavenScopeConfig (..),
+  MavenScopePredicate (..),
   ReachabilityConfigFile (..),
   ConfigReleaseGroup (..),
   ConfigReleaseGroupProject (..),
@@ -41,12 +42,14 @@ import Control.Effect.Lift (Lift)
 import Data.Aeson (
   FromJSON (parseJSON),
   ToJSON,
+  Value (..),
   withObject,
   withText,
   (.!=),
   (.:),
   (.:?),
  )
+import Data.Aeson.Types (typeMismatch)
 import Data.Error (createBody, renderErrataStack)
 import Data.Foldable (asum)
 import Data.Functor (($>))
@@ -56,6 +59,8 @@ import Data.Set (Set)
 import Data.Set qualified as Set
 import Data.String.Conversion (ToString (toString), ToText (toText))
 import Data.Text (Text, strip, toLower)
+import Data.Vector (Vector)
+import Data.Vector qualified as Vector
 import Diag.Diagnostic (ToDiagnostic (..))
 import Effect.Logger (
   Logger,
@@ -285,7 +290,14 @@ newtype ExperimentalGradleConfigs = ExperimentalGradleConfigs
   {gradleConfigsOnly :: Set Text}
   deriving (Eq, Ord, Show)
 
-data MavenScopeConfig = MavenScopeOnlyConfig (Set Text) | MavenScopeExcludeConfig (Set Text)
+data MavenScopeConfig
+  = MavenScopeOnlyConfig (Set Text)
+  | MavenScopeExcludeConfig (Set MavenScopePredicate)
+  deriving (Eq, Ord, Show)
+
+data MavenScopePredicate
+  = MavenScopePredicateSingle Text
+  | MavenScopePredicateCombined (Set Text)
   deriving (Eq, Ord, Show)
 
 instance FromJSON (Path Abs File -> ConfigFile) where
@@ -366,6 +378,13 @@ instance FromJSON MavenScopeConfig where
     MavenScopeOnlyConfig . Set.fromList <$> (obj .: "scope-only" .!= [])
       <|> MavenScopeExcludeConfig . Set.fromList <$> (obj .:? "scope-exclude" .!= [])
 
+instance FromJSON MavenScopePredicate where
+  parseJSON (String scope) = pure . MavenScopePredicateSingle $ scope
+  parseJSON (Array scopes) = do
+    scopes' :: Vector Text <- traverse parseJSON scopes
+    pure . MavenScopePredicateCombined . Set.fromList . Vector.toList $ scopes'
+  parseJSON invalid = typeMismatch "MavenScopePredicate" invalid
+
 instance FromJSON ConfigTelemetry where
   parseJSON = withObject "ConfigTelemetry" $ \obj ->
     ConfigTelemetry <$> (obj .: "scope")

src/App/Fossa/Container/Sources/DockerArchive.hs

@@ -179,7 +179,7 @@ analyzeLayer systemDepsOnly filters withoutDefaultFilters capabilities osInfo la
       )
   where
     noMavenScopeFilters :: MavenScopeFilters
-    noMavenScopeFilters = MavenScopeIncludeFilters mempty
+    noMavenScopeFilters = MavenScopeOnlyFilters mempty
     noExperimental :: ExperimentalAnalyzeConfig
     noExperimental =
       ExperimentalAnalyzeConfig
@@ -334,7 +334,7 @@ listTargetLayer capabilities osInfo layerFs tarball layerType = do
           GoModulesBasedTactic -- Targets aren't different between package/module centric analysis for Go.
           False -- Targets are not impacted by path dependencies.
       )
-    . runReader (MavenScopeIncludeFilters mempty)
+    . runReader (MavenScopeOnlyFilters mempty)
     . runReader (mempty :: AllFilters)
     $ run
   where

src/App/Fossa/ListTargets.hs

@@ -82,7 +82,7 @@ listTargetsMain ListTargetsConfig{..} = do
     . runAtomicCounter
     . runReader experimental
     -- `fossa list-targets` does not support maven scope filters.
-    . runReader (MavenScopeIncludeFilters mempty)
+    . runReader (MavenScopeOnlyFilters mempty)
     -- The current version of `fossa list-targets` does not support filters.
     -- TODO: support both discovery and post-discovery filtering.
     . runReader (mempty :: AllFilters)