-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add google sheets oauth flow to server + fix auth rootObject type in protocol #7131
Merged
Merged
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
30c0659
adding google sheets oauth flow to server
Phlair ffa4cc8
Merge branch 'master' into george/google-oauth
Phlair 44f6f44
fix oauth type in protocol yaml
Phlair 3a5034a
bump sheets version in definitions
Phlair b3230b4
added GDrive scope
Phlair bb27766
Merge branch 'master' into george/google-oauth
Phlair ab09b8b
update sheets to master changes
Phlair 1fd4a2d
update protocol incl. cdk
Phlair 99a6ce3
protocol typing for oauth rootobject
Phlair a3c8b55
format
Phlair c418e42
Merge branch 'master' into george/google-oauth
Phlair File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4 changes: 2 additions & 2 deletions
4
airbyte-integrations/connectors/source-klaviyo/integration_tests/invalid_config.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,4 @@ | ||
{ | ||
"api_key": "api-key", | ||
"start_date": "2021-01-01T00:00:00Z" | ||
"api_key": "api-key", | ||
"start_date": "2021-01-01T00:00:00Z" | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -19,6 +19,7 @@ | |
Status, | ||
Type, | ||
) | ||
|
||
# helpers | ||
from airbyte_cdk.sources import Source | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
49 changes: 49 additions & 0 deletions
49
airbyte-oauth/src/main/java/io/airbyte/oauth/flows/google/GoogleSheetsOAuthFlow.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,49 @@ | ||
/* | ||
* Copyright (c) 2021 Airbyte, Inc., all rights reserved. | ||
*/ | ||
|
||
package io.airbyte.oauth.flows.google; | ||
|
||
import com.fasterxml.jackson.databind.JsonNode; | ||
import com.google.common.annotations.VisibleForTesting; | ||
import com.google.common.base.Preconditions; | ||
import io.airbyte.config.persistence.ConfigRepository; | ||
import java.net.http.HttpClient; | ||
import java.util.function.Supplier; | ||
|
||
public class GoogleSheetsOAuthFlow extends GoogleOAuthFlow { | ||
|
||
// space-delimited string for multiple scopes, see: | ||
// https://datatracker.ietf.org/doc/html/rfc6749#section-3.3 | ||
@VisibleForTesting | ||
static final String SCOPE_URL = "https://www.googleapis.com/auth/spreadsheets.readonly https://www.googleapis.com/auth/drive.readonly"; | ||
|
||
public GoogleSheetsOAuthFlow(final ConfigRepository configRepository) { | ||
super(configRepository); | ||
} | ||
|
||
@VisibleForTesting | ||
GoogleSheetsOAuthFlow(final ConfigRepository configRepository, final HttpClient httpClient, final Supplier<String> stateSupplier) { | ||
super(configRepository, httpClient, stateSupplier); | ||
} | ||
|
||
@Override | ||
protected String getScope() { | ||
return SCOPE_URL; | ||
} | ||
|
||
@Override | ||
protected String getClientIdUnsafe(final JsonNode config) { | ||
// the config object containing client ID and secret is nested inside the "credentials" object | ||
Preconditions.checkArgument(config.hasNonNull("credentials")); | ||
return super.getClientIdUnsafe(config.get("credentials")); | ||
} | ||
|
||
@Override | ||
protected String getClientSecretUnsafe(final JsonNode config) { | ||
// the config object containing client ID and secret is nested inside the "credentials" object | ||
Preconditions.checkArgument(config.hasNonNull("credentials")); | ||
return super.getClientSecretUnsafe(config.get("credentials")); | ||
} | ||
|
||
} |
168 changes: 168 additions & 0 deletions
168
...-integration/java/io/airbyte/oauth/flows/google/GoogleSheetsOAuthFlowIntegrationTest.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,168 @@ | ||
/* | ||
* Copyright (c) 2021 Airbyte, Inc., all rights reserved. | ||
*/ | ||
|
||
package io.airbyte.oauth.flows.google; | ||
|
||
import static org.junit.jupiter.api.Assertions.assertTrue; | ||
import static org.mockito.Mockito.mock; | ||
import static org.mockito.Mockito.when; | ||
|
||
import com.fasterxml.jackson.databind.JsonNode; | ||
import com.google.common.collect.ImmutableMap; | ||
import com.sun.net.httpserver.HttpExchange; | ||
import com.sun.net.httpserver.HttpHandler; | ||
import com.sun.net.httpserver.HttpServer; | ||
import io.airbyte.commons.json.Jsons; | ||
import io.airbyte.config.SourceOAuthParameter; | ||
import io.airbyte.config.persistence.ConfigNotFoundException; | ||
import io.airbyte.config.persistence.ConfigRepository; | ||
import io.airbyte.validation.json.JsonValidationException; | ||
import java.io.IOException; | ||
import java.io.OutputStream; | ||
import java.net.InetSocketAddress; | ||
import java.nio.file.Files; | ||
import java.nio.file.Path; | ||
import java.util.HashMap; | ||
import java.util.List; | ||
import java.util.Map; | ||
import java.util.UUID; | ||
import org.junit.jupiter.api.AfterEach; | ||
import org.junit.jupiter.api.BeforeEach; | ||
import org.junit.jupiter.api.Test; | ||
import org.slf4j.Logger; | ||
import org.slf4j.LoggerFactory; | ||
|
||
public class GoogleSheetsOAuthFlowIntegrationTest { | ||
|
||
private static final Logger LOGGER = LoggerFactory.getLogger(GoogleSheetsOAuthFlowIntegrationTest.class); | ||
private static final String REDIRECT_URL = "http://localhost/code"; | ||
private static final Path CREDENTIALS_PATH = Path.of("secrets/google_sheets.json"); | ||
|
||
private ConfigRepository configRepository; | ||
private GoogleSheetsOAuthFlow googleSheetsOAuthFlow; | ||
private HttpServer server; | ||
private ServerHandler serverHandler; | ||
|
||
@BeforeEach | ||
public void setup() throws IOException { | ||
if (!Files.exists(CREDENTIALS_PATH)) { | ||
throw new IllegalStateException( | ||
"Must provide path to a oauth credentials file."); | ||
} | ||
configRepository = mock(ConfigRepository.class); | ||
googleSheetsOAuthFlow = new GoogleSheetsOAuthFlow(configRepository); | ||
|
||
server = HttpServer.create(new InetSocketAddress(80), 0); | ||
server.setExecutor(null); // creates a default executor | ||
server.start(); | ||
serverHandler = new ServerHandler("code"); | ||
server.createContext("/code", serverHandler); | ||
} | ||
|
||
@AfterEach | ||
void tearDown() { | ||
server.stop(1); | ||
} | ||
|
||
@Test | ||
public void testFullGoogleOAuthFlow() throws InterruptedException, ConfigNotFoundException, IOException, JsonValidationException { | ||
int limit = 20; | ||
final UUID workspaceId = UUID.randomUUID(); | ||
final UUID definitionId = UUID.randomUUID(); | ||
final String fullConfigAsString = new String(Files.readAllBytes(CREDENTIALS_PATH)); | ||
final JsonNode credentialsJson = Jsons.deserialize(fullConfigAsString); | ||
when(configRepository.listSourceOAuthParam()).thenReturn(List.of(new SourceOAuthParameter() | ||
.withOauthParameterId(UUID.randomUUID()) | ||
.withSourceDefinitionId(definitionId) | ||
.withWorkspaceId(workspaceId) | ||
.withConfiguration(Jsons.jsonNode(Map.of("credentials", ImmutableMap.builder() | ||
.put("client_id", credentialsJson.get("credentials").get("client_id").asText()) | ||
.put("client_secret", credentialsJson.get("credentials").get("client_secret").asText()) | ||
.build()))))); | ||
final String url = googleSheetsOAuthFlow.getSourceConsentUrl(workspaceId, definitionId, REDIRECT_URL); | ||
LOGGER.info("Waiting for user consent at: {}", url); | ||
// TODO: To automate, start a selenium job to navigate to the Consent URL and click on allowing | ||
// access... | ||
while (!serverHandler.isSucceeded() && limit > 0) { | ||
Thread.sleep(1000); | ||
limit -= 1; | ||
} | ||
assertTrue(serverHandler.isSucceeded(), "Failed to get User consent on time"); | ||
final Map<String, Object> params = googleSheetsOAuthFlow.completeSourceOAuth(workspaceId, definitionId, | ||
Map.of("code", serverHandler.getParamValue()), REDIRECT_URL); | ||
LOGGER.info("Response from completing OAuth Flow is: {}", params.toString()); | ||
assertTrue(params.containsKey("credentials")); | ||
final Map<String, Object> credentials = (Map<String, Object>) params.get("credentials"); | ||
assertTrue(credentials.containsKey("refresh_token")); | ||
assertTrue(credentials.get("refresh_token").toString().length() > 0); | ||
assertTrue(credentials.containsKey("access_token")); | ||
assertTrue(credentials.get("access_token").toString().length() > 0); | ||
} | ||
|
||
static class ServerHandler implements HttpHandler { | ||
|
||
final private String expectedParam; | ||
private String paramValue; | ||
private boolean succeeded; | ||
|
||
public ServerHandler(final String expectedParam) { | ||
this.expectedParam = expectedParam; | ||
this.paramValue = ""; | ||
this.succeeded = false; | ||
} | ||
|
||
public boolean isSucceeded() { | ||
return succeeded; | ||
} | ||
|
||
public String getParamValue() { | ||
return paramValue; | ||
} | ||
|
||
@Override | ||
public void handle(final HttpExchange t) { | ||
final String query = t.getRequestURI().getQuery(); | ||
LOGGER.info("Received query: '{}'", query); | ||
final Map<String, String> data; | ||
try { | ||
data = deserialize(query); | ||
final String response; | ||
if (data != null && data.containsKey(expectedParam)) { | ||
paramValue = data.get(expectedParam); | ||
response = String.format("Successfully extracted %s:\n'%s'\nTest should be continuing the OAuth Flow to retrieve the refresh_token...", | ||
expectedParam, paramValue); | ||
LOGGER.info(response); | ||
t.sendResponseHeaders(200, response.length()); | ||
succeeded = true; | ||
} else { | ||
response = String.format("Unable to parse query params from redirected url: %s", query); | ||
t.sendResponseHeaders(500, response.length()); | ||
} | ||
final OutputStream os = t.getResponseBody(); | ||
os.write(response.getBytes()); | ||
os.close(); | ||
} catch (final RuntimeException | IOException e) { | ||
LOGGER.error("Failed to parse from body {}", query, e); | ||
} | ||
} | ||
|
||
private static Map<String, String> deserialize(final String query) { | ||
if (query == null) { | ||
return null; | ||
} | ||
final Map<String, String> result = new HashMap<>(); | ||
for (final String param : query.split("&")) { | ||
final String[] entry = param.split("="); | ||
if (entry.length > 1) { | ||
result.put(entry[0], entry[1]); | ||
} else { | ||
result.put(entry[0], ""); | ||
} | ||
} | ||
return result; | ||
} | ||
|
||
} | ||
|
||
} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
would it make sense to DRY this component? feels like we use it in every test class
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes all oauth backends are going to copy/paste this test structure, maybe worth DRYing it and making it easier for #4982
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed, I think there's a lot of DRY to be had on the testing but thought I'd avoid until #4982 was clear in case the effort would be made redundant. If these will likely exist as is regardless then I'll DRY as part of this PR.