Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support concurrent refresh of refresh tokens #38382

Merged
merged 21 commits into from
Mar 1, 2019
Merged

Support concurrent refresh of refresh tokens #38382

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
70e620f
allow tokens to be reissued in a given time window
jkakavas Feb 4, 2019
318af42
Concludes work for supporting concurrent refreshes of access tokens i…
jkakavas Feb 4, 2019
541fc34
Merge remote-tracking branch 'origin/master' into support-concurrent-…
jkakavas Feb 4, 2019
55b7428
remove debug logging
jkakavas Feb 4, 2019
27324ab
fix merge woes
jkakavas Feb 5, 2019
f51a8f8
messing up while resolving merge conflicts is my super power
jkakavas Feb 5, 2019
44e0036
Merge remote-tracking branch 'origin/master' into support-concurrent-…
jkakavas Feb 5, 2019
7b70ca5
Handle/not handle deprecation warnings as needed
jkakavas Feb 5, 2019
2b54388
Handle deprecation header-AbstractUpgradeTestCase
jkakavas Feb 5, 2019
6a66302
Revert "Handle deprecation header-AbstractUpgradeTestCase"
jkakavas Feb 5, 2019
7337574
address feedback
jkakavas Feb 11, 2019
e655b0d
Fix versions for master. Will be changed back to V7_1_0 on backport
jkakavas Feb 11, 2019
0fa0f3c
add test with concurrent refreshes
jkakavas Feb 13, 2019
4d1e1dc
Implement suggested modifications
jkakavas Feb 26, 2019
d0971d1
Address feedback
jkakavas Feb 26, 2019
03475ac
Merge remote-tracking branch 'origin/master' into support-concurrent-…
jkakavas Feb 26, 2019
8f804c1
Fix TokenServiceTests
jkakavas Feb 27, 2019
309c8d1
address ffedback
jkakavas Feb 28, 2019
e13fd13
Merge remote-tracking branch 'origin/master' into support-concurrent-…
jkakavas Feb 28, 2019
35eb8ef
address feedback
jkakavas Feb 28, 2019
cbc626d
address feedback
jkakavas Mar 1, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions x-pack/plugin/core/src/main/resources/security-index-template.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,13 @@
"refreshed" : {
"type" : "boolean"
},
"refresh_time": {
"type": "date",
"format": "epoch_millis"
},
"superseded_by": {
"type": "keyword"
},
"invalidated" : {
"type" : "boolean"
},
Expand Down
423 changes: 283 additions & 140 deletions ...ck/plugin/security/src/main/java/org/elasticsearch/xpack/security/authc/TokenService.java
View file
Open in desktop

Large diffs are not rendered by default.

58 changes: 57 additions & 1 deletion ...in/security/src/test/java/org/elasticsearch/xpack/security/authc/TokenAuthIntegTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -330,7 +330,7 @@ public void testRefreshingInvalidatedToken() {
assertEquals("token has been invalidated", e.getHeader("error_description").get(0));
}

public void testRefreshingMultipleTimes() {
public void testRefreshingMultipleTimesFails() throws Exception {
Client client = client().filterWithHeader(Collections.singletonMap("Authorization",
UsernamePasswordToken.basicAuthHeaderValue(SecuritySettingsSource.TEST_USER_NAME,
SecuritySettingsSourceField.TEST_PASSWORD_SECURE_STRING)));
Expand All @@ -343,6 +343,30 @@ public void testRefreshingMultipleTimes() {
assertNotNull(createTokenResponse.getRefreshToken());
CreateTokenResponse refreshResponse = securityClient.prepareRefreshToken(createTokenResponse.getRefreshToken()).get();
assertNotNull(refreshResponse);
// We now have two documents, the original(now refreshed) token doc and the new one with the new access doc
AtomicReference<String> docId = new AtomicReference<>();
assertBusy(() -> {
SearchResponse searchResponse = client.prepareSearch(SecurityIndexManager.SECURITY_INDEX_NAME)
.setSource(SearchSourceBuilder.searchSource()
.query(QueryBuilders.boolQuery()
.must(QueryBuilders.termQuery("doc_type", "token"))
.must(QueryBuilders.termQuery("refresh_token.refreshed", "true"))))
.setSize(1)
.setTerminateAfter(1)
.get();
assertThat(searchResponse.getHits().getTotalHits().value, equalTo(1L));
docId.set(searchResponse.getHits().getAt(0).getId());
});

// hack doc to modify the refresh time to 10 seconds ago so that we don't hit the lenient refresh case
Instant refreshed = Instant.now();
Instant aWhileAgo = refreshed.minus(10L, ChronoUnit.SECONDS);
assertTrue(Instant.now().isAfter(aWhileAgo));
client.prepareUpdate(SecurityIndexManager.SECURITY_INDEX_NAME, "doc", docId.get())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's verify the update response here as that might help with debugging in case of test failure.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point !

.setDoc("refresh_token", Collections.singletonMap("refresh_time", aWhileAgo.toEpochMilli()))
.setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE)
.get();


ElasticsearchSecurityException e = expectThrows(ElasticsearchSecurityException.class,
() -> securityClient.prepareRefreshToken(createTokenResponse.getRefreshToken()).get());
Expand All @@ -351,6 +375,38 @@ public void testRefreshingMultipleTimes() {
assertEquals("token has already been refreshed", e.getHeader("error_description").get(0));
}

public void testRefreshingMultipleTimesWithinWindowSucceeds() throws Exception {
Client client = client().filterWithHeader(Collections.singletonMap("Authorization",
UsernamePasswordToken.basicAuthHeaderValue(SecuritySettingsSource.TEST_USER_NAME,
SecuritySettingsSourceField.TEST_PASSWORD_SECURE_STRING)));
SecurityClient securityClient = new SecurityClient(client);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

by doing it this way, all threads use the same client which hits the same node; maybe it would be worth having each thread create a client before it says that it is ready to run by counting down the new latch

CreateTokenResponse createTokenResponse = securityClient.prepareCreateToken()
.setGrantType("password")
.setUsername(SecuritySettingsSource.TEST_USER_NAME)
.setPassword(new SecureString(SecuritySettingsSourceField.TEST_PASSWORD.toCharArray()))
.get();
assertNotNull(createTokenResponse.getRefreshToken());

CreateTokenResponse refreshResponse = securityClient.prepareRefreshToken(createTokenResponse.getRefreshToken()).get();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should fire a small number of requests in parallel and then check the responses have the same token string.

assertNotNull(refreshResponse);
assertNotNull(refreshResponse.getRefreshToken());
assertNotEquals(refreshResponse.getRefreshToken(), createTokenResponse.getRefreshToken());
assertNotEquals(refreshResponse.getTokenString(), createTokenResponse.getTokenString());

assertNoTimeout(client().filterWithHeader(Collections.singletonMap("Authorization", "Bearer " + refreshResponse.getTokenString()))
.admin().cluster().prepareHealth().get());

CreateTokenResponse secondRefreshResponse = securityClient.prepareRefreshToken(createTokenResponse.getRefreshToken()).get();
assertNotNull(secondRefreshResponse);
assertNotNull(secondRefreshResponse.getRefreshToken());
assertThat(secondRefreshResponse.getRefreshToken(), equalTo(refreshResponse.getRefreshToken()));
assertThat(secondRefreshResponse.getTokenString(), equalTo(refreshResponse.getTokenString()));

assertNoTimeout(
client().filterWithHeader(Collections.singletonMap("Authorization", "Bearer " + secondRefreshResponse.getTokenString()))
.admin().cluster().prepareHealth().get());
}

public void testRefreshAsDifferentUser() {
Client client = client().filterWithHeader(Collections.singletonMap("Authorization",
UsernamePasswordToken.basicAuthHeaderValue(SecuritySettingsSource.TEST_USER_NAME,
Expand Down
184 changes: 0 additions & 184 deletions ...ugin/security/src/test/java/org/elasticsearch/xpack/security/authc/TokenServiceTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,6 @@
import org.elasticsearch.xpack.core.security.authc.Authentication;
import org.elasticsearch.xpack.core.security.authc.Authentication.AuthenticationType;
import org.elasticsearch.xpack.core.security.authc.Authentication.RealmRef;
import org.elasticsearch.xpack.core.security.authc.TokenMetaData;
import org.elasticsearch.xpack.core.security.authc.support.TokensInvalidationResult;
import org.elasticsearch.xpack.core.security.user.User;
import org.elasticsearch.xpack.core.watcher.watch.ClockMock;
Expand Down Expand Up @@ -187,189 +186,6 @@ public void testInvalidAuthorizationHeader() throws Exception {
}
}

public void testRotateKey() throws Exception {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code is still present in the token service, right? If so I don't think we should delete the tests until we remove the code

TokenService tokenService = new TokenService(tokenServiceEnabledSettings, systemUTC(), client, securityIndex, clusterService);
Authentication authentication = new Authentication(new User("joe", "admin"), new RealmRef("native_realm", "native", "node1"), null);
PlainActionFuture<Tuple<UserToken, String>> tokenFuture = new PlainActionFuture<>();
tokenService.createUserToken(authentication, authentication, tokenFuture, Collections.emptyMap(), true);
final UserToken token = tokenFuture.get().v1();
assertNotNull(token);
mockGetTokenFromId(token, false);
authentication = token.getAuthentication();

ThreadContext requestContext = new ThreadContext(Settings.EMPTY);
requestContext.putHeader("Authorization", "Bearer " + tokenService.getUserTokenString(token));

try (ThreadContext.StoredContext ignore = requestContext.newStoredContext(true)) {
PlainActionFuture<UserToken> future = new PlainActionFuture<>();
tokenService.getAndValidateToken(requestContext, future);
UserToken serialized = future.get();
assertAuthentication(authentication, serialized.getAuthentication());
}
rotateKeys(tokenService);

try (ThreadContext.StoredContext ignore = requestContext.newStoredContext(true)) {
PlainActionFuture<UserToken> future = new PlainActionFuture<>();
tokenService.getAndValidateToken(requestContext, future);
UserToken serialized = future.get();
assertAuthentication(authentication, serialized.getAuthentication());
}

PlainActionFuture<Tuple<UserToken, String>> newTokenFuture = new PlainActionFuture<>();
tokenService.createUserToken(authentication, authentication, newTokenFuture, Collections.emptyMap(), true);
final UserToken newToken = newTokenFuture.get().v1();
assertNotNull(newToken);
assertNotEquals(tokenService.getUserTokenString(newToken), tokenService.getUserTokenString(token));

requestContext = new ThreadContext(Settings.EMPTY);
requestContext.putHeader("Authorization", "Bearer " + tokenService.getUserTokenString(newToken));
mockGetTokenFromId(newToken, false);

try (ThreadContext.StoredContext ignore = requestContext.newStoredContext(true)) {
PlainActionFuture<UserToken> future = new PlainActionFuture<>();
tokenService.getAndValidateToken(requestContext, future);
UserToken serialized = future.get();
assertAuthentication(authentication, serialized.getAuthentication());
}
}

private void rotateKeys(TokenService tokenService) {
TokenMetaData tokenMetaData = tokenService.generateSpareKey();
tokenService.refreshMetaData(tokenMetaData);
tokenMetaData = tokenService.rotateToSpareKey();
tokenService.refreshMetaData(tokenMetaData);
}

public void testKeyExchange() throws Exception {
TokenService tokenService = new TokenService(tokenServiceEnabledSettings, systemUTC(), client, securityIndex, clusterService);
int numRotations = randomIntBetween(1, 5);
for (int i = 0; i < numRotations; i++) {
rotateKeys(tokenService);
}
TokenService otherTokenService = new TokenService(tokenServiceEnabledSettings, systemUTC(), client, securityIndex,
clusterService);
otherTokenService.refreshMetaData(tokenService.getTokenMetaData());
Authentication authentication = new Authentication(new User("joe", "admin"), new RealmRef("native_realm", "native", "node1"), null);
PlainActionFuture<Tuple<UserToken, String>> tokenFuture = new PlainActionFuture<>();
tokenService.createUserToken(authentication, authentication, tokenFuture, Collections.emptyMap(), true);
final UserToken token = tokenFuture.get().v1();
assertNotNull(token);
mockGetTokenFromId(token, false);
authentication = token.getAuthentication();

ThreadContext requestContext = new ThreadContext(Settings.EMPTY);
requestContext.putHeader("Authorization", "Bearer " + tokenService.getUserTokenString(token));
try (ThreadContext.StoredContext ignore = requestContext.newStoredContext(true)) {
PlainActionFuture<UserToken> future = new PlainActionFuture<>();
otherTokenService.getAndValidateToken(requestContext, future);
UserToken serialized = future.get();
assertAuthentication(authentication, serialized.getAuthentication());
}

rotateKeys(tokenService);

otherTokenService.refreshMetaData(tokenService.getTokenMetaData());

try (ThreadContext.StoredContext ignore = requestContext.newStoredContext(true)) {
PlainActionFuture<UserToken> future = new PlainActionFuture<>();
otherTokenService.getAndValidateToken(requestContext, future);
UserToken serialized = future.get();
assertEquals(authentication, serialized.getAuthentication());
}
}

public void testPruneKeys() throws Exception {
TokenService tokenService = new TokenService(tokenServiceEnabledSettings, systemUTC(), client, securityIndex, clusterService);
Authentication authentication = new Authentication(new User("joe", "admin"), new RealmRef("native_realm", "native", "node1"), null);
PlainActionFuture<Tuple<UserToken, String>> tokenFuture = new PlainActionFuture<>();
tokenService.createUserToken(authentication, authentication, tokenFuture, Collections.emptyMap(), true);
final UserToken token = tokenFuture.get().v1();
assertNotNull(token);
mockGetTokenFromId(token, false);
authentication = token.getAuthentication();

ThreadContext requestContext = new ThreadContext(Settings.EMPTY);
requestContext.putHeader("Authorization", "Bearer " + tokenService.getUserTokenString(token));

try (ThreadContext.StoredContext ignore = requestContext.newStoredContext(true)) {
PlainActionFuture<UserToken> future = new PlainActionFuture<>();
tokenService.getAndValidateToken(requestContext, future);
UserToken serialized = future.get();
assertAuthentication(authentication, serialized.getAuthentication());
}
TokenMetaData metaData = tokenService.pruneKeys(randomIntBetween(0, 100));
tokenService.refreshMetaData(metaData);

int numIterations = scaledRandomIntBetween(1, 5);
for (int i = 0; i < numIterations; i++) {
rotateKeys(tokenService);
}

try (ThreadContext.StoredContext ignore = requestContext.newStoredContext(true)) {
PlainActionFuture<UserToken> future = new PlainActionFuture<>();
tokenService.getAndValidateToken(requestContext, future);
UserToken serialized = future.get();
assertAuthentication(authentication, serialized.getAuthentication());
}

PlainActionFuture<Tuple<UserToken, String>> newTokenFuture = new PlainActionFuture<>();
tokenService.createUserToken(authentication, authentication, newTokenFuture, Collections.emptyMap(), true);
final UserToken newToken = newTokenFuture.get().v1();
assertNotNull(newToken);
assertNotEquals(tokenService.getUserTokenString(newToken), tokenService.getUserTokenString(token));

metaData = tokenService.pruneKeys(1);
tokenService.refreshMetaData(metaData);

try (ThreadContext.StoredContext ignore = requestContext.newStoredContext(true)) {
PlainActionFuture<UserToken> future = new PlainActionFuture<>();
tokenService.getAndValidateToken(requestContext, future);
UserToken serialized = future.get();
assertNull(serialized);
}

requestContext = new ThreadContext(Settings.EMPTY);
requestContext.putHeader("Authorization", "Bearer " + tokenService.getUserTokenString(newToken));
mockGetTokenFromId(newToken, false);
try (ThreadContext.StoredContext ignore = requestContext.newStoredContext(true)) {
PlainActionFuture<UserToken> future = new PlainActionFuture<>();
tokenService.getAndValidateToken(requestContext, future);
UserToken serialized = future.get();
assertAuthentication(authentication, serialized.getAuthentication());
}

}

public void testPassphraseWorks() throws Exception {
TokenService tokenService = new TokenService(tokenServiceEnabledSettings, systemUTC(), client, securityIndex, clusterService);
Authentication authentication = new Authentication(new User("joe", "admin"), new RealmRef("native_realm", "native", "node1"), null);
PlainActionFuture<Tuple<UserToken, String>> tokenFuture = new PlainActionFuture<>();
tokenService.createUserToken(authentication, authentication, tokenFuture, Collections.emptyMap(), true);
final UserToken token = tokenFuture.get().v1();
assertNotNull(token);
mockGetTokenFromId(token, false);
authentication = token.getAuthentication();

ThreadContext requestContext = new ThreadContext(Settings.EMPTY);
requestContext.putHeader("Authorization", "Bearer " + tokenService.getUserTokenString(token));

try (ThreadContext.StoredContext ignore = requestContext.newStoredContext(true)) {
PlainActionFuture<UserToken> future = new PlainActionFuture<>();
tokenService.getAndValidateToken(requestContext, future);
UserToken serialized = future.get();
assertAuthentication(authentication, serialized.getAuthentication());
}

try (ThreadContext.StoredContext ignore = requestContext.newStoredContext(true)) {
// verify a second separate token service with its own passphrase cannot verify
TokenService anotherService = new TokenService(Settings.EMPTY, systemUTC(), client, securityIndex,
clusterService);
PlainActionFuture<UserToken> future = new PlainActionFuture<>();
anotherService.getAndValidateToken(requestContext, future);
assertNull(future.get());
}
}

public void testGetTokenWhenKeyCacheHasExpired() throws Exception {
TokenService tokenService = new TokenService(tokenServiceEnabledSettings, systemUTC(), client, securityIndex, clusterService);
Authentication authentication = new Authentication(new User("joe", "admin"), new RealmRef("native_realm", "native", "node1"), null);
Expand Down
4 changes: 3 additions & 1 deletion .../qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/AbstractUpgradeTestCase.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,8 +86,10 @@ public void setupForTests() throws Exception {
boolean success = true;
for (String template : templatesToWaitFor()) {
try {
Request headRequest = new Request("HEAD", "_template/" + template);
headRequest.setOptions(allowTypeRemovalWarnings());
final boolean exists = adminClient()
.performRequest(new Request("HEAD", "_template/" + template))
.performRequest(headRequest)
.getStatusLine().getStatusCode() == 200;
success &= exists;
logger.debug("template [{}] exists [{}]", template, exists);
Expand Down
Loading