MSF-21292: Add capability to remove old master workspaces to LCM brick by dangdinh · Pull Request #1802 · gooddata/gooddata-ruby

dangdinh · 2021-09-08T03:04:11Z

Add capability to remove old master workspaces to LCM Rollout brick.
Verified ADS and Netapp configuration on PI

ghost · 2021-09-08T03:27:01Z

Build succeeded (check pipeline).

https://checklist.intgdc.com/job/compliance/job/gooddata-ruby-oss-check/860/ : SUCCESS in 8s
https://jenkins-ii.intgdc.com/job/gooddata-ruby/job/gooddata-ruby-lcm-build-pipeline/759/ : SUCCESS in 22m 31s

ghost · 2021-09-08T04:13:43Z

Build in pipeline check aborted.

ghost · 2021-09-08T04:43:24Z

Build in pipeline check aborted.

ghost · 2021-09-08T04:44:34Z

Build in pipeline check aborted.

ghost · 2021-09-08T05:06:06Z

Build succeeded (check pipeline).

https://checklist.intgdc.com/job/compliance/job/gooddata-ruby-oss-check/864/ : SUCCESS in 6s
https://jenkins-ii.intgdc.com/job/gooddata-ruby/job/gooddata-ruby-lcm-build-pipeline/763/ : SUCCESS in 21m 21s

ghost · 2021-09-08T06:40:33Z

Build in pipeline check aborted.

ghost · 2021-09-08T07:03:04Z

Build succeeded (check pipeline).

https://checklist.intgdc.com/job/compliance/job/gooddata-ruby-oss-check/866/ : SUCCESS in 7s
https://jenkins-ii.intgdc.com/job/gooddata-ruby/job/gooddata-ruby-lcm-build-pipeline/765/ : SUCCESS in 22m 18s

ghost · 2021-09-10T05:14:58Z

Build in pipeline check aborted.

ghost · 2021-09-10T05:38:12Z

Build succeeded (check pipeline).

https://checklist.intgdc.com/job/compliance/job/gooddata-ruby-oss-check/879/ : SUCCESS in 6s
https://jenkins-ii.intgdc.com/job/gooddata-ruby/job/gooddata-ruby-lcm-build-pipeline/777/ : SUCCESS in 23m 03s

danh-ung · 2021-09-10T10:53:58Z

+                      project = client.projects(project_wrapper[:master_project_id])
+                      GoodData.logger.info "Segment #{segment.segment_id}: Deleting old master workspace, project: '#{project.title}', PID: (#{project.pid})."
+                      project.delete
+                      removal_master_project_ids << project_wrapper[:master_project_id]


Add catch exception for each project deletion.

ghost · 2021-09-14T02:59:00Z

Build in pipeline check aborted.

ghost · 2021-09-14T03:20:13Z

Build succeeded (check pipeline).

https://checklist.intgdc.com/job/compliance/job/gooddata-ruby-oss-check/887/ : SUCCESS in 8s
https://jenkins-ii.intgdc.com/job/gooddata-ruby/job/gooddata-ruby-lcm-build-pipeline/784/ : SUCCESS in 20m 59s

ghost · 2021-09-14T03:34:21Z

Build in pipeline check aborted.

ghost · 2021-09-14T03:56:09Z

Build succeeded (check pipeline).

https://checklist.intgdc.com/job/compliance/job/gooddata-ruby-oss-check/889/ : SUCCESS in 6s
https://jenkins-ii.intgdc.com/job/gooddata-ruby/job/gooddata-ruby-lcm-build-pipeline/786/ : SUCCESS in 21m 33s

ghost · 2021-09-16T08:46:54Z

Build in pipeline check aborted.

ghost · 2021-09-16T09:09:20Z

Build succeeded (check pipeline).

https://checklist.intgdc.com/job/compliance/job/gooddata-ruby-oss-check/891/ : SUCCESS in 7s
https://jenkins-ii.intgdc.com/job/gooddata-ruby/job/gooddata-ruby-lcm-build-pipeline/788/ : SUCCESS in 19m 08s

sangtm · 2021-09-16T18:19:09Z

+                begin
+                  removal_master_project_ids = remove_old_master_workspace(params, segment.segment_id, master_projects, number_of_deleted_projects)
+                  remove_release_table(params, domain_name, data_product.data_product_id, segment.segment_id, master_projects, removal_master_project_ids)
+                rescue => e


rescue StandardError => e , Or we can move rescue to remove_release_table function, we already have rescue for remove_old_master_workspace

sangtm · 2021-09-16T18:19:27Z

                   "Details: #{sync_result['links']['details']}")
            end

+            # rubocop:disable Style/RescueStandardError


Remove this

sangtm · 2021-09-16T18:19:38Z

+                end
+              end
+            end
+            # rubocop:enable Style/RescueStandardError


And remove this

sangtm · 2021-09-16T18:51:25Z

+        [
+         { master_project_id: 'foo', version: 2, segment_id: 'premium' },
+         { master_project_id: 'bar', version: 3, segment_id: 'premium' },
+         { master_project_id: 'baz', version: 1, segment_id: 'premium' }


please add test data with: 9, 12, 113

sangtm · 2021-09-16T19:02:54Z

+        expect(GoodData::LCM2::Helpers.get_master_project_list_from_nfs(domain_id, data_prod_id, 'first_segment')).to eq(result)
+      end
+    end
  end


Add test for update_master_project_from_nfs and remove_old_master_workspace

sangtm · 2021-09-16T19:03:47Z

+              GoodData.logger.info "Segment #{segment_id}: Deleting old master workspace, project: '#{project.title}', PID: (#{project.pid})."
+              project.delete if project && !project.deleted?
+              removal_master_project_ids << project_wrapper[:master_project_id]
+            rescue => ex


also rescue StandardError => ex

sangtm · 2021-09-16T19:09:07Z

+            if keep_only_previous_masters_count >= 0
+              number_of_deleted_projects = master_projects.count - (keep_only_previous_masters_count + 1)
+
+              if number_of_deleted_projects.positive?


why don't we just simply check something like: master_projects.count - keep_only_previous_masters_count > 1

discussed about it and we keep it as current because we need to remove latest master workspace from list

sangtm · 2021-09-16T19:20:51Z

+              project.delete if project && !project.deleted?
+              removal_master_project_ids << project_wrapper[:master_project_id]
+            rescue => ex
+              GoodData.logger.error "Unable to remove master workspace, reason: #{ex}"


"Unable to remove master workspace: '#{master_project_id}', Error: #{ex}"

ghost · 2021-09-20T02:33:37Z

Build in pipeline check aborted.

ghost · 2021-09-20T02:36:31Z

Build in pipeline check aborted.

ghost · 2021-09-20T02:36:55Z

Build in pipeline check aborted.

ghost · 2021-09-22T11:23:17Z

Build in pipeline check aborted.

ghost · 2021-09-22T11:30:33Z

Build in pipeline check aborted.

ghost · 2021-09-22T11:30:52Z

Build in pipeline check aborted.

ghost · 2021-09-22T11:34:19Z

Build in pipeline check aborted.

ghost · 2021-09-22T11:34:34Z

Build in pipeline check aborted.

ghost · 2021-09-22T12:30:37Z

Build succeeded (check pipeline).

https://checklist.intgdc.com/job/compliance/job/gooddata-ruby-oss-check/910/ : SUCCESS in 14s
https://jenkins-ii.intgdc.com/job/gooddata-ruby/job/gooddata-ruby-lcm-build-pipeline/802/ : SUCCESS in 26m 53s

ghost · 2021-09-24T11:02:48Z

Build in pipeline check aborted.

ghost · 2021-09-24T11:15:23Z

Build in pipeline check aborted.

ghost · 2021-09-24T11:15:44Z

Build in pipeline check aborted.

ghost · 2021-09-24T11:37:36Z

Build succeeded (check pipeline).

https://checklist.intgdc.com/job/compliance/job/gooddata-ruby-oss-check/914/ : SUCCESS in 7s
https://jenkins-ii.intgdc.com/job/gooddata-ruby/job/gooddata-ruby-lcm-build-pipeline/806/ : SUCCESS in 21m 40s

danh-ung · 2021-09-25T01:27:29Z

LGTM

danh-ung · 2021-09-25T01:36:14Z

+      let(:latest_data) do
+        [
+            { master_project_id: 'opi15', version: 3 }
+        ]


Here we test only WRITE "one master project", missing usecase WRITE "many master projects". It can miss the bug that our loop to write multiple projects doesn't work.

ghost · 2021-09-28T02:21:18Z

Build in pipeline check aborted.

ghost · 2021-09-28T02:43:15Z

Build succeeded (check pipeline).

https://checklist.intgdc.com/job/compliance/job/gooddata-ruby-oss-check/916/ : SUCCESS in 10s
https://jenkins-ii.intgdc.com/job/gooddata-ruby/job/gooddata-ruby-lcm-build-pipeline/808/ : SUCCESS in 21m 42s

ghost · 2021-09-28T04:17:23Z

Build in pipeline check aborted.

ghost · 2021-09-28T04:41:26Z

Build succeeded (check pipeline).

https://checklist.intgdc.com/job/compliance/job/gooddata-ruby-oss-check/918/ : SUCCESS in 6s
https://jenkins-ii.intgdc.com/job/gooddata-ruby/job/gooddata-ruby-lcm-build-pipeline/810/ : SUCCESS in 23m 52s

yenkins · 2021-10-04T11:20:40Z

Sonar scan result

More detail, see in https://sonarqube-gate.intgdc.com/dashboard?id=gooddata-ruby-gate-PR1802

To scan for vulnerabilities in dependencies and run unit tests (to get coverage report in sonar) please comment your PR with 'extended check sonar'.

ghost · 2021-10-04T11:49:14Z

Build succeeded (gate pipeline).

https://jenkins-ii.intgdc.com/job/gooddata-ruby/job/gooddata-ruby-lcm-promote-pipeline/152/ : SUCCESS in 30m 47s
https://checklist.intgdc.com/job/compliance/job/gooddata-ruby-merge-compliance-check/582/ : SUCCESS in 19s
https://checklist.intgdc.com/job/compliance/job/gooddata-ruby-sonarqube-scan-zuul-gate/40/ : SUCCESS in 1m 55s
https://checklist.intgdc.com/job/compliance/job/gooddata-ruby-fossa-scanning-zuul-el7/125/ : FAILURE in 2m 58s (non-voting)

dangdinh force-pushed the MSF-21292 branch from 57b695e to f503021 Compare September 8, 2021 04:13

dangdinh force-pushed the MSF-21292 branch from 9dbf8c8 to 292bd65 Compare September 8, 2021 04:44

dangdinh force-pushed the MSF-21292 branch from 1f53fbc to e180ce2 Compare September 8, 2021 06:40

dangdinh force-pushed the MSF-21292 branch from b965f04 to 6b3e64d Compare September 10, 2021 05:14

danh-ung reviewed Sep 10, 2021

View reviewed changes

dangdinh force-pushed the MSF-21292 branch from c815f38 to 4c9869a Compare September 14, 2021 02:58

dangdinh force-pushed the MSF-21292 branch from 260dfb5 to 649af21 Compare September 14, 2021 03:34

danh-ung added the do not merge Do not merge this yet label Sep 14, 2021

dangdinh force-pushed the MSF-21292 branch from d676d25 to c410e6b Compare September 16, 2021 08:46

sangtm reviewed Sep 16, 2021

View reviewed changes

dangdinh force-pushed the MSF-21292 branch from ec653c6 to e8d2e69 Compare September 20, 2021 02:33

dangdinh force-pushed the MSF-21292 branch from 927dd65 to 86b92a5 Compare September 20, 2021 02:36

dangdinh force-pushed the MSF-21292 branch from 0661b6f to fff6212 Compare September 22, 2021 11:23

dangdinh force-pushed the MSF-21292 branch from 4c2b8c2 to c481f7b Compare September 22, 2021 11:30

dangdinh force-pushed the MSF-21292 branch from 196e426 to eccc1d8 Compare September 22, 2021 11:34

dangdinh force-pushed the MSF-21292 branch from 6e08c75 to 3bd125e Compare September 24, 2021 11:02

dangdinh force-pushed the MSF-21292 branch from 53fb06a to 948fd98 Compare September 24, 2021 11:15

danh-ung removed the do not merge Do not merge this yet label Sep 25, 2021

danh-ung reviewed Sep 25, 2021

View reviewed changes

dangdinh force-pushed the MSF-21292 branch from ffaf8a0 to d58c002 Compare September 28, 2021 02:21

MSF-21292: Add capability to remove old master workspaces to LCM brick

c2c5846

dangdinh force-pushed the MSF-21292 branch from 6800570 to c2c5846 Compare September 28, 2021 04:17

sangtm added the merge label Oct 4, 2021

ghost removed the merge label Oct 4, 2021

ghost merged commit ac4cb7b into gooddata:master Oct 4, 2021

This pull request was closed.

Conversation

dangdinh commented Sep 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Sep 8, 2021

Uh oh!

ghost commented Sep 8, 2021

Uh oh!

ghost commented Sep 8, 2021

Uh oh!

ghost commented Sep 8, 2021

Uh oh!

ghost commented Sep 8, 2021

Uh oh!

ghost commented Sep 8, 2021

Uh oh!

ghost commented Sep 8, 2021

Uh oh!

ghost commented Sep 10, 2021

Uh oh!

ghost commented Sep 10, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ghost commented Sep 14, 2021

Uh oh!

ghost commented Sep 14, 2021

Uh oh!

ghost commented Sep 14, 2021

Uh oh!

ghost commented Sep 14, 2021

Uh oh!

ghost commented Sep 16, 2021

Uh oh!

ghost commented Sep 16, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ghost commented Sep 20, 2021

Uh oh!

ghost commented Sep 20, 2021

Uh oh!

ghost commented Sep 20, 2021

Uh oh!

ghost commented Sep 22, 2021

Uh oh!

ghost commented Sep 22, 2021

Uh oh!

dangdinh commented Sep 8, 2021 •

edited

Loading