secrets provider: added tests for caching, updated documentation, imp…

…roved environment variables secret provider

docs/security/credentials/password-related-configuration.adoc

@@ -20,6 +20,7 @@ This is configured in Value Policy objects referenced from Security Policy objec
 
 Most of the password-related settings are located in the xref:/midpoint/reference/security/security-policy/[Security Policy object] where also policies for other credential types are located.
 
+* Passwords that can be resolved from external sources are configured in system configuration, see xref:./secrets-provider-configuration.adoc[]
 
 == Password Policy
 
@@ -135,4 +136,4 @@ Use the security policy settings instead.
 
 * xref:/midpoint/reference/security/credentials/password-policy/[Password Policy]
 
-* xref:/midpoint/reference/security/credentials/password-storage-configuration/[Password Storage Configuration]
+* xref:/midpoint/reference/security/credentials/password-storage-configuration/[Password Storage Configuration]

docs/security/credentials/secrets-provider-configuration.adoc

@@ -124,17 +124,17 @@ All providers have common part of configuration:
 | cache
 | xsd:duration
 | Optional cache duration (TTL). Example: `PT1H` for 1 hour.
+If not defined, cache is not used (equivalent of `PT0S`).
 | No
-
 |===
 
 === File provider
 
 File provider is used to obtain secrets from files.
-Where the file is located is defined by `parentDirectoryPath` and key is the name of the file.
-Value of the key is the content of the file.
+Where the file is located is defined by `parentDirectoryPath` and
 
 Optional configuration property is `charset` which defaults to `utf-8`.
+It is used to specify the character encoding of the files.
 
 .Example of file provider
 [source,xml]
@@ -150,20 +150,25 @@ Optional configuration property is `charset` which defaults to `utf-8`.
 Docker provider just specific implementation of <<File provider>> and it's used to obtain secrets from Docker secrets files (documentation available https://docs.docker.com/compose/use-secrets/[here]).
 Secrets directory is defined by Docker and it is not configurable.
 It is always `/run/secrets` on Linux and `C:\ProgramData\Docker\secrets` on Windows.
+Key is the name of the file and value of the key is the content of the file.
 
 Optional configuration property is `charset` which defaults to `utf-8`.
+It is used to specify the character encoding of the files.
 
 .Example of Docker provider
 [source,xml]
 ----
 <docker>
     <identifier>docker-provider</identifier>
+    <cache>PT1H</cache>
 </docker>
 ----
 
 === Properties provider
 
-#TODO#
+Secrets provider that reads secrets from properties file defined by `propertiesFile` element.
+Optional configuration property is `charset` which defaults to `utf-8`.
+It is used to specify the character encoding of the files.
 
 .Example of properties provider
 [source,xml]
@@ -176,20 +181,35 @@ Optional configuration property is `charset` which defaults to `utf-8`.
 
 === Environment variables provider
 
-#TODO#
+Custom implementation of secrets provider that reads secrets from environment variables.
+If `useSystemProperties` is set to `true` (default is `false`), system properties (e.g. `-Dkey=value parameters`) will be used as well.
+Search for key first attempts to find it in environment variables and then in system properties.
+
+If `prefix` is defined, only variables/properties with the given prefix will be used.
 
 .Example of environment variables provider
 [source,xml]
 ----
 <environmentVariables>
     <identifier>env-provider</identifier>
     <prefix>MP_</prefix>
+    <useSystemProperties>true</useSystemProperties>
 </environmentVariables>
 ----
 
+In this example, only environment variables with prefix `MP_` will be used.
+For example `MP_MY_SECRET_VARIABLE=qwe123` has to be referenced in protected string using key `MY_SECRET_VARIABLE`, prefix will be prepended automatically.
+
 === Custom provider
 
-#TODO#
+[NOTE]
+This provider is an advanced experimental feature.
+
+Custom provider allows to plug-in custom implementation of secrets providers available on classpath.
+Required element `className` is the fully qualified name of the class implementing `com.evolveum.midpoint.xml.ns._public.common.common_3.SecretsProvider` interface.
+
+Configuration of custom provider is defined in `configuration` element.
+Each element of `configuration` will be passed to the provider as DOM element, which can be used during initialization of the provider.
 
 .Example of custom provider
 [source,xml]

infra/common/src/main/java/com/evolveum/midpoint/common/secrets/CacheableSecretsProviderDelegate.java

@@ -85,7 +85,7 @@ private <ST> ST getOrResolveSecret(String key, Class<ST> type) throws Encryption
         if (value != null) {
             LOGGER.trace("Cache hit for key {}", key);
 
-            if (Clock.get().currentTimeMillis() - value.ttl <= ttl) {
+            if (value.ttl - Clock.get().currentTimeMillis() >= 0) {
                 LOGGER.trace("Cache entry for key {} is still valid, using cached value", key);
 
                 if (value.value == null) {

infra/common/src/main/java/com/evolveum/midpoint/common/secrets/EnvironmentVariablesSecretsProvider.java

@@ -9,6 +9,7 @@
 
 import com.evolveum.midpoint.prism.crypto.SecretsProvider;
 
+import org.apache.commons.lang3.BooleanUtils;
 import org.apache.commons.lang3.StringUtils;
 import org.jetbrains.annotations.NotNull;
 
@@ -40,6 +41,11 @@ protected <ST> ST resolveSecret(@NotNull String key, @NotNull Class<ST> type) {
         }
 
         String value = System.getenv(finalKey);
+
+        if (value == null && BooleanUtils.isTrue(getConfiguration().isUseSystemProperties())) {
+            value = System.getProperty(finalKey);
+        }
+
         byte[] data = value != null ? value.getBytes() : null;
 
         return mapValue(data, type);

infra/common/src/main/java/com/evolveum/midpoint/common/secrets/SecretsProviderImpl.java

@@ -17,8 +17,6 @@
 
 public abstract class SecretsProviderImpl<T extends SecretsProviderType> implements SecretsProvider<T> {
 
-    private static final String[] EMPTY_DEPENDENCIES = new String[0];
-
     private final T configuration;
 
     public SecretsProviderImpl(@NotNull T configuration) {
@@ -35,11 +33,6 @@ public SecretsProviderImpl(@NotNull T configuration) {
         return configuration.getIdentifier();
     }
 
-    @Override
-    public @NotNull String[] getDependencies() {
-        return EMPTY_DEPENDENCIES;
-    }
-
     @Override
     public String getSecretString(@NotNull String key) throws EncryptionException {
         return resolveSecret(key, String.class);

infra/schema/src/main/resources/xml/ns/public/common/common-security-3.xsd

@@ -3605,6 +3605,13 @@
                             </xsd:documentation>
                         </xsd:annotation>
                     </xsd:element>
+                    <xsd:element name="useSystemProperties" type="xsd:boolean" minOccurs="0" default="false">
+                        <xsd:annotation>
+                            <xsd:documentation>
+                                If true, the provider will also use system properties as a source of secrets.
+                            </xsd:documentation>
+                        </xsd:annotation>
+                    </xsd:element>
                 </xsd:sequence>
             </xsd:extension>
         </xsd:complexContent>

model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/TestSecretProviders.java

@@ -9,17 +9,26 @@
 
 import static org.testng.AssertJUnit.assertEquals;
 
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.evolveum.midpoint.common.Clock;
+
+import org.jetbrains.annotations.NotNull;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.test.annotation.DirtiesContext;
 import org.springframework.test.context.ContextConfiguration;
 import org.testng.AssertJUnit;
 import org.testng.annotations.Test;
 
 import com.evolveum.icf.dummy.resource.DummyResource;
+import com.evolveum.midpoint.common.secrets.CacheableSecretsProviderDelegate;
 import com.evolveum.midpoint.prism.PrismObject;
 import com.evolveum.midpoint.prism.crypto.EncryptionException;
 import com.evolveum.midpoint.prism.crypto.Protector;
+import com.evolveum.midpoint.prism.crypto.SecretsProvider;
+import com.evolveum.midpoint.prism.xml.XmlTypeConverter;
 import com.evolveum.midpoint.xml.ns._public.common.common_3.CredentialsType;
+import com.evolveum.midpoint.xml.ns._public.common.common_3.CustomSecretsProviderType;
 import com.evolveum.midpoint.xml.ns._public.common.common_3.UserType;
 import com.evolveum.prism.xml.ns._public.types_3.ExternalDataType;
 import com.evolveum.prism.xml.ns._public.types_3.ProtectedStringType;
@@ -33,11 +42,9 @@ public class TestSecretProviders extends AbstractInitializedModelIntegrationTest
 
     /**
      * Test that loads a secret from a properties file using propertiesFile secret provider.
-     *
-     * @throws Exception
      */
     @Test
-    public void test100ProtectedStringInResourceConfiguration() throws Exception {
+    public void test100ProtectedStringInResourceConfiguration() {
         DummyResource orange = getDummyResource(RESOURCE_DUMMY_ORANGE_NAME);
         assertEquals("Wrong guarded useless string", "whatever", orange.getUselessGuardedString());
     }
@@ -74,8 +81,7 @@ public void test110ProtectedStringInUser() throws Exception {
     }
 
     @Test
-    public void test120TestResolvingSecrets() throws Exception {
-
+    public void test120TestResolvingSecrets() {
         ProtectedStringType nonExistingProvider = createProtectedString("non-existing-provider", "MP_USER_PASSWORD");
         try {
             protector.decryptString(nonExistingProvider);
@@ -107,5 +113,60 @@ private ProtectedStringType createProtectedString(String provider, String key) {
         return ps;
     }
 
-    // TODO test cacheable secrets provider delegate
+    @Test
+    public void test130CacheableSecretsProvider() throws Exception {
+        final CustomSecretsProviderType custom = new CustomSecretsProviderType();
+        custom.setIdentifier("fake");
+        custom.setCache(XmlTypeConverter.createDuration("PT10S"));
+        custom.setClassName("com.example.FakeSecretsProvider");
+
+        final String value = "example";
+
+        final AtomicInteger counter = new AtomicInteger(0);
+
+        SecretsProvider<CustomSecretsProviderType> provider = new SecretsProvider<>() {
+
+            @Override
+            public @NotNull String getIdentifier() {
+                return custom.getIdentifier();
+            }
+
+            @Override
+            public CustomSecretsProviderType getConfiguration() {
+                return custom;
+            }
+
+            @Override
+            public String getSecretString(@NotNull String key) throws EncryptionException {
+                counter.incrementAndGet();
+
+                return value;
+            }
+        };
+
+        CacheableSecretsProviderDelegate<CustomSecretsProviderType> delegate =
+                new CacheableSecretsProviderDelegate<>(provider, custom.getCache());
+
+        final String key = "key";
+
+        // first attempt
+        AssertJUnit.assertEquals(value, delegate.getSecretString(key));
+        AssertJUnit.assertEquals(1, counter.get());
+
+        // second attempt should be cached
+        AssertJUnit.assertEquals(value, delegate.getSecretString(key));
+        AssertJUnit.assertEquals(1, counter.get());
+
+        Clock.get().overrideOffset(20000L);
+
+        // third attempt should not be cached, because the cache has expired
+        AssertJUnit.assertEquals(value, delegate.getSecretString(key));
+        AssertJUnit.assertEquals(2, counter.get());
+
+        // fourth attempt should be cached
+        AssertJUnit.assertEquals(value, delegate.getSecretString(key));
+        AssertJUnit.assertEquals(2, counter.get());
+
+        Clock.get().resetOverride();
+    }
 }