Skip to content

Commit

Permalink
Merge branch 'master' into bump-version-s3-gcs-9th-April
Browse files Browse the repository at this point in the history
  • Loading branch information
subodh1810 committed Apr 9, 2022
2 parents 6018933 + 58b4fab commit abd6f94
Show file tree
Hide file tree
Showing 2 changed files with 14 additions and 9 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -46,10 +46,10 @@ public AirbyteConnectionStatus check(final JsonNode config) {
final AmazonS3 s3Client = destinationConfig.getS3Client();

// Test single upload (for small files) permissions
S3Destination.testSingleUpload(s3Client, destinationConfig.getBucketName());
S3Destination.testSingleUpload(s3Client, destinationConfig.getBucketName(), destinationConfig.getBucketPath());

// Test multipart upload with stream transfer manager
S3Destination.testMultipartUpload(s3Client, destinationConfig.getBucketName());
S3Destination.testMultipartUpload(s3Client, destinationConfig.getBucketName(), destinationConfig.getBucketPath());

return new AirbyteConnectionStatus().withStatus(Status.SUCCEEDED);
} catch (final Exception e) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,10 @@ public AirbyteConnectionStatus check(final JsonNode config) {
S3Destination.attemptS3WriteAndDelete(storageOperations, destinationConfig, destinationConfig.getBucketName());

// Test single upload (for small files) permissions
testSingleUpload(s3Client, destinationConfig.getBucketName());
testSingleUpload(s3Client, destinationConfig.getBucketName(), destinationConfig.getBucketPath());

// Test multipart upload with stream transfer manager
testMultipartUpload(s3Client, destinationConfig.getBucketName());
testMultipartUpload(s3Client, destinationConfig.getBucketName(), destinationConfig.getBucketPath());

return new AirbyteConnectionStatus().withStatus(Status.SUCCEEDED);
} catch (final Exception e) {
Expand All @@ -77,9 +77,12 @@ public AirbyteConnectionStatus check(final JsonNode config) {
}
}

public static void testSingleUpload(final AmazonS3 s3Client, final String bucketName) {
public static void testSingleUpload(final AmazonS3 s3Client, final String bucketName, String bucketPath) {
LOGGER.info("Started testing if all required credentials assigned to user for single file uploading");
final String testFile = "test_" + System.currentTimeMillis();
if (bucketPath.endsWith("/")) {
throw new RuntimeException("Bucket Path should not end with /");
}
final String testFile = bucketPath + "/" + "test_" + System.currentTimeMillis();
try {
s3Client.putObject(bucketName, testFile, "this is a test file");
} finally {
Expand All @@ -88,10 +91,12 @@ public static void testSingleUpload(final AmazonS3 s3Client, final String bucket
LOGGER.info("Finished checking for normal upload mode");
}

public static void testMultipartUpload(final AmazonS3 s3Client, final String bucketName) throws IOException {
public static void testMultipartUpload(final AmazonS3 s3Client, final String bucketName, String bucketPath) throws IOException {
LOGGER.info("Started testing if all required credentials assigned to user for multipart upload");

final String testFile = "test_" + System.currentTimeMillis();
if (bucketPath.endsWith("/")) {
throw new RuntimeException("Bucket Path should not end with /");
}
final String testFile = bucketPath + "/" + "test_" + System.currentTimeMillis();
final StreamTransferManager manager = StreamTransferManagerHelper.getDefault(
bucketName,
testFile,
Expand Down

2 comments on commit abd6f94

@github-actions
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SonarQube Report

SonarQube report for Airbyte Connectors Destination S3(#11864)

Measures

Name Value Name Value Name Value
Quality Gate Status OK Security Rating A Vulnerabilities 0
Duplicated Blocks 3 Duplicated Lines (%) 1.2 Lines to Cover 1338
Lines of Code 2974 Code Smells 49 Coverage 0.0
Bugs 1 Reliability Rating C Blocker Issues 0
Critical Issues 4 Major Issues 40 Minor Issues 6

Detected Issues

Rule File Description Message
java:S112 (MAJOR) s3/S3Destination.java:83 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S112 (MAJOR) s3/S3Destination.java:97 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S1155 (MINOR) s3/S3StorageOperations.java:111 Collection.isEmpty() should be used to test for emptiness Use isEmpty() to check whether the collection is empty or not.
java:S112 (MAJOR) s3/BlobStorageOperations.java:19 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S112 (MAJOR) s3/BlobStorageOperations.java:31 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S117 (MINOR) s3/BlobStorageOperations.java:33 Local variable and method parameter names should comply with a naming convention Rename this local variable to match the regular expression '^[a-z][a-zA-Z0-9]*$'.
java:S1192 (CRITICAL) s3/S3StorageOperations.java:229 String literals should not be duplicated Define a constant instead of duplicating this literal "[0-9]{2}" 5 times.
java:S119 (MINOR) credential/BlobStorageCredentialConfig.java:7 Type parameter names should comply with a naming convention Rename this generic name to match the regular expression '^[A-Z][0-9]?$'.
java:S1172 (MAJOR) s3/S3ConsumerFactory.java:73 Unused method parameters should be removed Remove this unused method parameter "namingResolver".
java:S1611 (MINOR) s3/S3ConsumerFactory.java:152 Parentheses should be removed from a single lambda input parameter when its type is inferred Remove the parentheses around the "hasFailed" parameter (sonar.java.source not set. Assuming 8 or greater.)
java:S1118 (MAJOR) s3/SerializedBufferFactory.java:25 Utility classes should not have public constructors Add a private constructor to hide the implicit public one.
java:S112 (MAJOR) s3/SerializedBufferFactory.java:72 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S112 (MAJOR) avro/AvroSerializedBuffer.java:34 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
common-java:DuplicatedBlocks (MAJOR) avro/S3AvroWriter.java Source files should not have any duplicated blocks 1 duplicated blocks of code must be removed.
common-java:DuplicatedBlocks (MAJOR) csv/S3CsvWriter.java Source files should not have any duplicated blocks 1 duplicated blocks of code must be removed.
java:S112 (MAJOR) jsonl/JsonLSerializedBuffer.java:33 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S1172 (MAJOR) jsonl/JsonLSerializedBuffer.java:63 Unused method parameters should be removed Remove this unused method parameter "config".
common-java:DuplicatedBlocks (MAJOR) jsonl/S3JsonlWriter.java Source files should not have any duplicated blocks 1 duplicated blocks of code must be removed.
java:S1118 (MAJOR) util/StreamTransferManagerHelper.java:12 Utility classes should not have public constructors Add a private constructor to hide the implicit public one.
java:S3358 (MAJOR) avro/JsonToAvroSchemaConverter.java:179 Ternary operators should not be nested Extract this nested ternary operation into an independent statement.
java:S112 (MAJOR) s3/BlobStorageOperations.java:26 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S112 (MAJOR) s3/S3StorageOperations.java:124 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S112 (MAJOR) s3/S3StorageOperations.java:148 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S112 (MAJOR) s3/S3StorageOperations.java:158 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S112 (MAJOR) csv/CsvSerializedBuffer.java:33 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S112 (MAJOR) csv/CsvSerializedBuffer.java:47 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S6213 (MAJOR) avro/S3AvroWriter.java:106 Restricted Identifiers should not be used as Identifiers Rename this variable to not match a restricted identifier.
java:S2259 (MAJOR) avro/JsonToAvroSchemaConverter.java:164 Null pointers should not be dereferenced A "NullPointerException" could be thrown; "properties" is nullable here.
java:S3776 (CRITICAL) avro/JsonToAvroSchemaConverter.java:338 Cognitive Complexity of methods should not be too high Refactor this method to reduce its Cognitive Complexity from 22 to the 15 allowed.
java:S3252 (CRITICAL) avro/JsonToAvroSchemaConverter.java:372 "static" base class members should not be accessed via derived types Use static access with "!Unknown!" for "Entry".
java:S1121 (MAJOR) avro/JsonToAvroSchemaConverter.java:261 Assignments should not be made from within sub-expressions Extract the assignment out of this expression.
java:S107 (MAJOR) s3/S3DestinationConfig.java:42 Methods should not have too many parameters Constructor has 9 parameters, which is greater than 7 authorized.
java:S107 (MAJOR) csv/S3CsvWriter.java:40 Methods should not have too many parameters Constructor has 9 parameters, which is greater than 7 authorized.
java:S3358 (MAJOR) s3/S3Destination.java:141 Ternary operators should not be nested Extract this nested ternary operation into an independent statement.
java:S5361 (CRITICAL) s3/S3Destination.java:142 "String#replace" should be preferred to "String#replaceAll" Replace this call to "replaceAll()" by a call to the "replace()" method.
java:S1121 (MAJOR) avro/JsonToAvroSchemaConverter.java:217 Assignments should not be made from within sub-expressions Extract the assignment out of this expression.
java:S1118 (MAJOR) avro/AvroConstants.java:10 Utility classes should not have public constructors Add a private constructor to hide the implicit public one.
java:S125 (MAJOR) avro/JsonToAvroSchemaConverter.java:209 Sections of code should not be commented out This block of commented-out lines of code should be removed.
java:S1068 (MAJOR) jsonl/S3JsonlWriter.java:37 Unused "private" fields should be removed Remove this unused "WRITER" private field.
java:S1118 (MAJOR) util/AvroRecordHelper.java:17 Utility classes should not have public constructors Add a private constructor to hide the implicit public one.
java:S1700 (MAJOR) avro/JsonSchemaType.java:23 A field should not duplicate the name of its containing class Rename field "jsonSchemaType"
java:S1118 (MAJOR) parquet/S3ParquetConstants.java:9 Utility classes should not have public constructors Add a private constructor to hide the implicit public one.
java:S1118 (MAJOR) util/S3OutputPathHelper.java:13 Utility classes should not have public constructors Add a private constructor to hide the implicit public one.
java:S112 (MAJOR) writer/ProductionWriterFactory.java:59 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S112 (MAJOR) writer/S3WriterFactory.java:21 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S1118 (MAJOR) s3/S3FormatConfigs.java:16 Utility classes should not have public constructors Add a private constructor to hide the implicit public one.
java:S112 (MAJOR) s3/S3FormatConfigs.java:39 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S1118 (MAJOR) csv/CsvSheetGenerator.java:25 Utility classes should not have public constructors Add a private constructor to hide the implicit public one.
java:S2094 (MINOR) csv/CsvSheetGenerators.java:7 Classes should not be empty Remove this empty class, write its code or make it an "interface".
java:S1116 (MINOR) csv/RootLevelFlatteningSheetGenerator.java:25 Empty statements should be removed Remove this empty statement.

Coverage (0.0%)

File Coverage File Coverage
src/main/java/io/airbyte/integrations/destination/s3/avro/AvroConstants.java 0.0 src/main/java/io/airbyte/integrations/destination/s3/avro/AvroNameTransformer.java 0.0
src/main/java/io/airbyte/integrations/destination/s3/avro/AvroRecordFactory.java 0.0 src/main/java/io/airbyte/integrations/destination/s3/avro/AvroSerializedBuffer.java 0.0
src/main/java/io/airbyte/integrations/destination/s3/avro/JsonFieldNameUpdater.java 0.0 src/main/java/io/airbyte/integrations/destination/s3/avro/JsonSchemaType.java 0.0
src/main/java/io/airbyte/integrations/destination/s3/avro/JsonToAvroSchemaConverter.java 0.0 src/main/java/io/airbyte/integrations/destination/s3/avro/S3AvroFormatConfig.java 0.0
src/main/java/io/airbyte/integrations/destination/s3/avro/S3AvroWriter.java 0.0 src/main/java/io/airbyte/integrations/destination/s3/credential/S3AccessKeyCredentialConfig.java 0.0
src/main/java/io/airbyte/integrations/destination/s3/credential/S3InstanceProfileCredentialConfig.java 0.0 src/main/java/io/airbyte/integrations/destination/s3/csv/BaseSheetGenerator.java 0.0
src/main/java/io/airbyte/integrations/destination/s3/csv/CsvSerializedBuffer.java 0.0 src/main/java/io/airbyte/integrations/destination/s3/csv/CsvSheetGenerator.java 0.0
src/main/java/io/airbyte/integrations/destination/s3/csv/NoFlatteningSheetGenerator.java 0.0 src/main/java/io/airbyte/integrations/destination/s3/csv/RootLevelFlatteningSheetGenerator.java 0.0
src/main/java/io/airbyte/integrations/destination/s3/csv/S3CsvFormatConfig.java 0.0 src/main/java/io/airbyte/integrations/destination/s3/csv/S3CsvWriter.java 0.0
src/main/java/io/airbyte/integrations/destination/s3/csv/StagingDatabaseCsvSheetGenerator.java 0.0 src/main/java/io/airbyte/integrations/destination/s3/jsonl/JsonLSerializedBuffer.java 0.0
src/main/java/io/airbyte/integrations/destination/s3/jsonl/S3JsonlFormatConfig.java 0.0 src/main/java/io/airbyte/integrations/destination/s3/jsonl/S3JsonlWriter.java 0.0
src/main/java/io/airbyte/integrations/destination/s3/parquet/ParquetSerializedBuffer.java 0.0 src/main/java/io/airbyte/integrations/destination/s3/parquet/S3ParquetConstants.java 0.0
src/main/java/io/airbyte/integrations/destination/s3/parquet/S3ParquetFormatConfig.java 0.0 src/main/java/io/airbyte/integrations/destination/s3/parquet/S3ParquetWriter.java 0.0
src/main/java/io/airbyte/integrations/destination/s3/S3ConsumerFactory.java 0.0 src/main/java/io/airbyte/integrations/destination/s3/S3Destination.java 0.0
src/main/java/io/airbyte/integrations/destination/s3/S3DestinationConfig.java 0.0 src/main/java/io/airbyte/integrations/destination/s3/S3DestinationConfigFactory.java 0.0
src/main/java/io/airbyte/integrations/destination/s3/S3DestinationConstants.java 0.0 src/main/java/io/airbyte/integrations/destination/s3/S3Format.java 0.0
src/main/java/io/airbyte/integrations/destination/s3/S3FormatConfig.java 0.0 src/main/java/io/airbyte/integrations/destination/s3/S3FormatConfigs.java 0.0
src/main/java/io/airbyte/integrations/destination/s3/S3StorageOperations.java 0.0 src/main/java/io/airbyte/integrations/destination/s3/SerializedBufferFactory.java 0.0
src/main/java/io/airbyte/integrations/destination/s3/util/AvroRecordHelper.java 0.0 src/main/java/io/airbyte/integrations/destination/s3/util/S3NameTransformer.java 0.0
src/main/java/io/airbyte/integrations/destination/s3/util/S3OutputPathHelper.java 0.0 src/main/java/io/airbyte/integrations/destination/s3/util/StreamTransferManagerHelper.java 0.0
src/main/java/io/airbyte/integrations/destination/s3/WriteConfig.java 0.0 src/main/java/io/airbyte/integrations/destination/s3/writer/BaseS3Writer.java 0.0
src/main/java/io/airbyte/integrations/destination/s3/writer/ProductionWriterFactory.java 0.0

@github-actions
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SonarQube Report

SonarQube report for Airbyte Connectors Destination Gcs(#11864)

Measures

Name Value Name Value Name Value
Bugs 0 Quality Gate Status OK Duplicated Blocks 0
Security Rating A Lines of Code 739 Lines to Cover 297
Code Smells 6 Reliability Rating A Duplicated Lines (%) 0.0
Vulnerabilities 0 Coverage 0.0 Blocker Issues 0
Critical Issues 0 Major Issues 5 Minor Issues 1

Detected Issues

Rule File Description Message
java:S1301 (MINOR) gcs/GcsDestinationConfig.java:60 "switch" statements should have at least 3 "case" clauses Replace this "switch" statement by "if" statements to increase readability.
java:S6213 (MAJOR) avro/GcsAvroWriter.java:97 Restricted Identifiers should not be used as Identifiers Rename this variable to not match a restricted identifier.
java:S1118 (MAJOR) util/GcsUtils.java:16 Utility classes should not have public constructors Add a private constructor to hide the implicit public one.
java:S1118 (MAJOR) credential/GcsCredentialConfigs.java:10 Utility classes should not have public constructors Add a private constructor to hide the implicit public one.
java:S112 (MAJOR) credential/GcsCredentialConfigs.java:19 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S1068 (MAJOR) parquet/GcsParquetWriter.java:39 Unused "private" fields should be removed Remove this unused "MAPPER" private field.

Coverage (0.0%)

File Coverage File Coverage
src/main/java/io/airbyte/integrations/destination/gcs/avro/GcsAvroWriter.java 0.0 src/main/java/io/airbyte/integrations/destination/gcs/credential/GcsCredentialConfigs.java 0.0
src/main/java/io/airbyte/integrations/destination/gcs/credential/GcsHmacKeyCredentialConfig.java 0.0 src/main/java/io/airbyte/integrations/destination/gcs/csv/GcsCsvWriter.java 0.0
src/main/java/io/airbyte/integrations/destination/gcs/GcsDestination.java 0.0 src/main/java/io/airbyte/integrations/destination/gcs/GcsDestinationConfig.java 0.0
src/main/java/io/airbyte/integrations/destination/gcs/GcsStorageOperations.java 0.0 src/main/java/io/airbyte/integrations/destination/gcs/jsonl/GcsJsonlWriter.java 0.0
src/main/java/io/airbyte/integrations/destination/gcs/parquet/GcsParquetWriter.java 0.0 src/main/java/io/airbyte/integrations/destination/gcs/util/GcsS3FileSystem.java 0.0
src/main/java/io/airbyte/integrations/destination/gcs/util/GcsUtils.java 0.0 src/main/java/io/airbyte/integrations/destination/gcs/writer/BaseGcsWriter.java 0.0

Please sign in to comment.