gcscopy e2e tests scenarios by priyabhatnagar25 · Pull Request #1283 · data-integrations/google-cloud

priyabhatnagar25 · 2023-08-28T07:51:07Z

No description provided.

GnsP · 2023-09-04T08:36:22Z

+public class GCSCopy {
+
+  @Then("Validate GCSCopy successfully copies object {string} to destination bucket")
+  public void validateGCSCopySuccessfullyCopiedObjectToDestinationBucket(String path) throws IOException {


In this step only the existence of the object in both source and target buckets is validated. Shouldn't it also verify if the contents of the objects in both buckets are identical or not ?

Hi ,
We will add the steps to verify the contents .

Hi, Validation step to check the contents of the objects in both buckets are identical or not is added. Please review it.

GnsP · 2023-09-04T08:44:14Z

+    Then Open and capture logs
+    Then Verify the pipeline status is "Succeeded"
+    Then Close the pipeline logs
+    Then Validate GCSCopy successfully copies object "gcsCsvFile" to destination bucket


this step does not validate whether the contents of the object was copied successfully.

Hi ,
We will add the steps to verify the contents .

Hi ,
We have added the steps to verify the contents inside the files .

GnsP · 2023-09-04T08:53:53Z

+    String gcsObject = PluginPropertyUtils.pluginProp(path);
+    boolean isPresentatsource = false;
+    for (Blob blob : StorageClient.listObjects(sourceGCSBucket).iterateAll()) {
+      if (blob.getName().contains(gcsObject)) {


why is this a contains check and not an equals check ?

Changed to equals check

It was not working with equals for the scenario when sub directories toggle button is set to true, so changed to contains check. As it has to check the subdirectories also.

I think in case of subdirectories, we need to check if all the files in the subdirectory (in the fixtures) are present in the source bucket.

In this context, using String.contains to check if the path of any of the objects in the bucket contains (as a substring) the path of the directory containing the test CSV files, seems to be fundamentally incorrect.

I think we need to restructure this check to work correctly in case of subdirectories, rather than trying to make this check pass with a substring (String.contains) check.

Hi, We have created two separate validation steps for subdirectories set to true and set to false.

The first validation step when subdirectory sets to true is checking in the source bucket that the subdirectory present in the source bucket along with its file is copied to the destination bucket along with its file with equals check and validating both the source and the destination bucket.

The second validation step when subdirectory set to false is checking that there is one subdirectory along with its file in the source bucket but when the subdirectory toggle button is marked as false its not copying subdirectory along with its files in the destination bucket. So , the validation is checked at the destination bucket with empty check that if the destination bucket is empty it has not copied the subdirectory along with it files to the destination bucket.

We have already covered the scenario for the files (not inside any folder), when the bucket is only having the files and not subdirectory and by default overwrite and copy subdirectory toggle button is marked as false.

GnsP · 2023-09-04T08:54:03Z

+    String targetGCSBucket = TestSetupHooks.gcsTargetBucketName;
+    try {
+      for (Blob blob : StorageClient.listObjects(targetGCSBucket).iterateAll()) {
+        if (blob.getName().contains(gcsObject)) {


why is this a contains check and not an equals check ?

Changed to equals check.

The comment regarding checking subdirectories in the source bucket also applies here.

GnsP · 2023-09-04T09:03:25Z

+    String sourceGCSBucket = TestSetupHooks.gcsSourceBucketName;
+    String gcsObject = PluginPropertyUtils.pluginProp(subdirectory);
+    for (Blob blob : StorageClient.listObjects(sourceGCSBucket).iterateAll()) {
+      if (blob.getName().contains(gcsObject)) {


why is this a contains check and not an equals check ?

Here it is the scenario to check that the sub-directories are not copied to the destination bucket.This is not working for equals check and the scenario is failing for it when equals check is applied. It shows the assertion error stating that subdirectory is copied from the source GCS bucket.Its working fine with contains check.

The comment regarding checking subdirectories in the source bucket (in the method validateGCSCopySuccessfullyCopiedObjectToDestinationBucket) also applies here.

GnsP · 2023-09-04T09:03:53Z

+    String sourceGCSBucket = TestSetupHooks.gcsSourceBucketName;
+    String gcsObject = PluginPropertyUtils.pluginProp(path);
+    for (Blob blob : StorageClient.listObjects(sourceGCSBucket).iterateAll()) {
+      if (blob.getName().contains(gcsObject)) {


why is this a contains check and not an equals check ?

Changed to equals check.

GnsP · 2023-09-04T09:05:48Z

+    }
+
+  @Then("Validate GCSCopy failed to copy object {string} to destination bucket")
+  public void validateGCSCopyFailedToCopyObjectToDestinationBucket(String path) throws IOException {


this only seems to check the existence of the object in the source bucket. Shouldn't it also validate the non-existence of the object in the destination bucket ?

So here this step is used to check that the pipeline is getting failed when we are selecting overwrite existing file as false. Here we already have the file it the destination bucket and we are selecting overwrite option as false and the pipeline is failing.So here we already have the file in the destination bucket.

GnsP · 2023-09-13T08:25:10Z

+    String gcsObject = PluginPropertyUtils.pluginProp(path);
+    boolean isPresentatsource = false;
+    for (Blob blob : StorageClient.listObjects(sourceGCSBucket).iterateAll()) {
+      if (blob.getName().contains(gcsObject)) {


I think in case of subdirectories, we need to check if all the files in the subdirectory (in the fixtures) are present in the source bucket.

In this context, using String.contains to check if the path of any of the objects in the bucket contains (as a substring) the path of the directory containing the test CSV files, seems to be fundamentally incorrect.

I think we need to restructure this check to work correctly in case of subdirectories, rather than trying to make this check pass with a substring (String.contains) check.

GnsP · 2023-09-13T08:26:02Z

+    String targetGCSBucket = TestSetupHooks.gcsTargetBucketName;
+    try {
+      for (Blob blob : StorageClient.listObjects(targetGCSBucket).iterateAll()) {
+        if (blob.getName().contains(gcsObject)) {


The comment regarding checking subdirectories in the source bucket also applies here.

GnsP · 2023-09-13T08:27:21Z

+    String sourceGCSBucket = TestSetupHooks.gcsSourceBucketName;
+    String gcsObject = PluginPropertyUtils.pluginProp(subdirectory);
+    for (Blob blob : StorageClient.listObjects(sourceGCSBucket).iterateAll()) {
+      if (blob.getName().contains(gcsObject)) {


The comment regarding checking subdirectories in the source bucket (in the method validateGCSCopySuccessfullyCopiedObjectToDestinationBucket) also applies here.

itsankit-google added the build Trigger unit test build label Aug 28, 2023

bharatgulati force-pushed the gcscopy_e2e_tests branch 2 times, most recently from ac57eba to 4c0e34d Compare August 31, 2023 16:34

priyabhatnagar25 force-pushed the gcscopy_e2e_tests branch 3 times, most recently from 1a9f1f8 to 1347cbd Compare September 1, 2023 11:40

sau42shri requested a review from GnsP September 1, 2023 12:40

bharatgulati force-pushed the gcscopy_e2e_tests branch 2 times, most recently from 1347cbd to 76e56a7 Compare September 4, 2023 08:41

GnsP requested changes Sep 4, 2023

View reviewed changes

priyabhatnagar25 force-pushed the gcscopy_e2e_tests branch 3 times, most recently from b0dc520 to fef2089 Compare September 11, 2023 12:51

GnsP requested changes Sep 13, 2023

View reviewed changes

priyabhatnagar25 force-pushed the gcscopy_e2e_tests branch 5 times, most recently from 17d18fc to 33a1f7b Compare September 21, 2023 11:11

bharatgulati added build Trigger unit test build and removed build Trigger unit test build labels Sep 21, 2023

priyabhatnagar25 force-pushed the gcscopy_e2e_tests branch from 33a1f7b to 20b0be7 Compare September 29, 2023 07:10

priyabhatnagar25 force-pushed the gcscopy_e2e_tests branch 2 times, most recently from d5001bc to ae8cd65 Compare October 10, 2023 06:10

gcscopy e2e scenarios review comments addressed

e121875

priyabhatnagar25 force-pushed the gcscopy_e2e_tests branch from ae8cd65 to e121875 Compare October 17, 2023 10:06

GnsP approved these changes Oct 19, 2023

View reviewed changes

GnsP merged commit e6fe8d8 into data-integrations:develop Oct 19, 2023

psainics deleted the gcscopy_e2e_tests branch February 13, 2025 05:48

Conversation

priyabhatnagar25 commented Aug 28, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

priyabhatnagar25 Sep 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

priyabhatnagar25 Sep 18, 2023 •

edited

Loading