Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[YW Platform] Enabling backup method in testupgrade throws exception for 2.0.9.0_13 and 2.2.7.0_5 configs #9270

Closed
nmantrav2 opened this issue Jul 13, 2021 · 5 comments
Assignees
Milestone

Comments

@nmantrav2
Copy link
Contributor

Repro steps:

  1. Enable backup method in testupgrade itest
  2. Test fails for

-- 2.0.9.0_13 config with following exception

Create backups : >>> Integration Test Failed <<<
51215 wait_for_task: Failed task with errors in 120.24636673927307s:
51216 Failed to execute task {"errorString":null,"nodeExporterUser":"prometheus","deviceInfo":null,"un iverseUUID":"b71658b1-5529-4a40-855b-acb7315453c2","expectedUniverseVersion":0,"cmkArn":null,"en cryptionAtRestConfig":{"encr..., hit error java.lang.RuntimeException: java.lang.RuntimeException: {"error": "Exception: Command '['ssh', '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHost sFile=/dev/null', '-i', '/opt/yugabyte/yugaware/data/keys/1fa4e623-8edb-4a77-8678-fb6a8f53c203/yb-itest-21289660d8-20210712-184619_1fa4e623-8edb-4a77-8678-fb6a8f53c203-key.pem', '-p', '54422', '-q', 'yugabyte@172.151.25.97', "cd / && bash -c '/home/yugabyte/master/bin/yb-admin --master _addresses 172.151.25.97:7100,172.151.46.184:7100,172.151.49.130:7100 ysql_catalog_version'"]' returned non-zero exit status 1"}.

-- 2.2.7.0_5 with following exception

Create backups : >>> Integration Test Failed <<<
51327 wait_for_task: Failed task with errors in 30.168999195098877s:
51328 Failed to execute task {"errorString":null,"nodeExporterUser":"prometheus","deviceInfo":null,"un iverseUUID":"cb7b70f3-2cf4-421b-a3ad-fe2e7e8ca5f8","expectedUniverseVersion":0,"cmkArn":null,"en cryptionAtRestConfig":{"encr..., hit error java.lang.RuntimeException: java.lang.RuntimeExceptio n: {"error": "Exception: 'namespace_name'"}.

@hsu880
Copy link

hsu880 commented Jul 13, 2021

Is this a consistent failure?
Which version of platform is this failure happening? 2.7, 2.6, etc?

@hsu880
Copy link

hsu880 commented Jul 13, 2021

More info from nmantrav2, these are consistent failures in 2.7.2 and 2.6 versions of platform. After fixing these, we should probably re-enable these tests.

@hsu880 hsu880 added this to Backlog in Platform Jul 14, 2021
@hsu880 hsu880 added this to the 2.7.x milestone Jul 14, 2021
@OlegLoginov
Copy link
Contributor

Fixed by the commit in the master branch: e8c72f3

@OlegLoginov OlegLoginov self-assigned this Sep 3, 2021
OlegLoginov added a commit that referenced this issue Sep 3, 2021
Summary:
`yb_backup` script must correctly parse output of `yb-admin list_snapshots show_details` command for older releases.

Old format:
```
{"type":"NAMESPACE","id":"e4c5591446db417f83a52c679de03118","data":{"name":"a",...}}
{"type":"TABLE","id":"d9603c2cab0b48ec807936496ac0e70e","data":{"name":"t2",...,"namespace_id"=...}}}}
{"type":"NAMESPACE","id":"e4c5591446db417f83a52c679de03118","data":{"name":"a",...}}
{"type":"TABLE","id":"28b5cebe9b0c4cdaa70ce9ceab31b1e5","data":{\
       "name":"t2idx","indexed_table_id":"d9603c2cab0b48ec807936496ac0e70e",...,"namespace_id"=...}}}}
```
Plus the old format (before Aug 2020) does not have 'namespace_name' field, which was added later.
New format:
```
{"type":"NAMESPACE","id":"e4c5591446db417f83a52c679de03118","data":{"name":"a",...}}
{"type":"TABLE","id":"d9603c2cab0b48ec807936496ac0e70e","data":{"name":"t2",...,"namespace_id"=...}}
{"type":"TABLE","id":"28b5cebe9b0c4cdaa70ce9ceab31b1e5","data":{\
       "name":"t2idx","indexed_table_id":"d9603c2cab0b48ec807936496ac0e70e",...,"namespace_id"=...}}}}
```
Original diff: https://phabricator.dev.yugabyte.com/D11420
Original commit: e8c72f3

Test Plan:
Jenkins: rebase: 2.6, hot

Existing backup tests:

ybd --cxx-test tools_yb-backup-test_ent

ybd --java-test org.yb.pgsql.TestYbBackup
ybd --java-test org.yb.cql.TestYbBackup --tp 1

Reviewers: bogdan, sergei

Reviewed By: sergei

Subscribers: jenkins-bot, yql

Differential Revision: https://phabricator.dev.yugabyte.com/D12873
@OlegLoginov OlegLoginov added this to To do in Backups via automation Sep 3, 2021
@Arpit-yb
Copy link
Contributor

Arpit-yb commented Nov 9, 2021

its still failing on master - https://jenkins.dev.yugabyte.com/view/Test%20Jobs/job/itest-developer/2182/
Backups on 2.0.9 are failing

@Arpit-yb
Copy link
Contributor

CC: @nmantrav2 @VijiYB Can we prioritize this issue, since its been failing for quite some time now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Backups
  
To do
Platform
  
Backlog
Development

No branches or pull requests

6 participants