Add check to not overwrite an existing key when migrating after downgrade

[fasmat]

Motivation

Downgrading to v1.3.x after upgrading to v1.4.x will generate a new key that when the node is updated again to v1.4.x will overwrite the existing key.

This adds a check that prevents the node from overwriting a possibly already existing key.

Description

When migrating the key the code now checks if there is already a local.key before copying the old key there.

Test Plan

• added test for this specific occurance

TODO

• Explain motivation or link existing issue(s)
• Test changes and document test plan
• Update documentation as needed
• Update changelog as needed

[codecov]

Codecov Report

Attention: Patch coverage is 71.42857% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 79.8%. Comparing base (d8fa47c) to head (251c7be).

Files	Patch %	Lines
node/node_identities.go	71.4%	4 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           develop   #5709   +/-   ##
=======================================
  Coverage     79.7%   79.8%           
=======================================
  Files          279     279           
  Lines        28527   28541   +14     
=======================================
+ Hits         22755   22782   +27     
+ Misses        4202    4185   -17     
- Partials      1570    1574    +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[fasmat]

bors merge

[spacemesh-bors]

Canceled.

[fasmat]

bors merge

[spacemesh-bors]

Canceled.

[fasmat]

bors merge

[spacemesh-bors]

Build failed (retrying...):

• systest-status

[spacemesh-bors]

Build failed:

• systest-status

[dshulyak]

test failed because one of the nodes didn't publish atx in epoch 3
https://grafana.spacemesh.dev/explore?panes=%7B%22OKQ%22:%7B%22datasource%22:%22loki-gke-dev%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22test-hmjx%5C%22%7D%20%7C%3D%20%5C%22atx%20published%5C%22%20%7C%20json%20%7C%20epoch%3D3%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki-gke-dev%22%7D,%22editorMode%22:%22code%22%7D%5D,%22range%22:%7B%22from%22:%221710520648590%22,%22to%22:%221710521810475%22%7D%7D%7D&schemaVersion=1&orgId=1

[fasmat]

@dshulyak looks unrelated to the change of this PR and the other PR that was included in the bors batch

[fasmat]

bors merge

[spacemesh-bors]

Build failed:

• systest-status

[fasmat]

rpc error: code = DeadlineExceeded

bors merge

[dshulyak]

i agree with you that it is not related to your change. retrying just doesn't make any sense, just open develop and push changes. either somebody will have to pay attention to such failures, or tests can be simply disabled

in the last case test exited before transaction was added to proposals and executed
https://grafana.spacemesh.dev/explore?panes=%7B%22PIN%22:%7B%22datasource%22:%22loki-gke-dev%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22test-rjed%5C%22,%20pod%3D%5C%22boot-0-7ffd7d9555-rcmlz%5C%22%7D%20%7C%20json%20%7C%20stream%3D%5C%22stdout%5C%22%20%7C~%20%5C%22executor%7Cnew%20proposal%7Ccleanup%5C%22%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki-gke-dev%22%7D,%22editorMode%22:%22code%22%7D%5D,%22range%22:%7B%22from%22:%221710529257956%22,%22to%22:%221710530619021%22%7D%7D%7D&schemaVersion=1&orgId=1

[dshulyak]

last job failure. all changes from march 8 were merged without single systest retry, i think there is either env fault or something else that makes it unusable

[dshulyak]

based on merge history repeated retries started from 14th, i would check what in 12-13 changes could cause them, otherwise look at env

[fasmat]

I would not say that they started march 14, we had failing merges every day for the last week or so. Most of them failed due to flaky tests which should now have been fixed 🤷

[fasmat]

I don't immediately see any PR that was merged recently that could have caused system tests to be more flaky.

In the last 3 days we had merges of:

• No code changes in the node: [Merged by Bors] - release v1.4.0 #5700, [Merged by Bors] - Add node merge tool #5685
• Extra config options (that are not changed in system tests): [Merged by Bors] - p2p: add gossip-peer-outbound-queue-size setting #5696
• Add extra payload to events: [Merged by Bors] - Add NodeID to events #5698
• Removal of GRPC timeout: [Merged by Bors] - Remove max connection age for GRPC #5693 - now no timeout; before 2 hour timeout - don't think this makes tests flakier
• Swap ed25519 lib: [Merged by Bors] - Swap stdlib for oasis for ed25519 signing and verification #5318 - don't see this as a possibility but maybe
• Changing logging levels: [Merged by Bors] - Add ChangeLogLevel to DebugService #5678 - not used in tests
• Different latency calcucation [Merged by Bors] - fetch/peers: make latency based selection biased towards last requests #5688 - I don't think this could be the cause?
• Reduce logging [Merged by Bors] - tune down logs that don't provide additional information #5689
• Update of dependencies: [Merged by Bors] - build(deps): Bump github.com/spacemeshos/post from 0.12.2 to 0.12.3 #5699, [Merged by Bors] - Bump post to v0.12.2 and post-rs to v0.7.2 #5691, [Merged by Bors] - build(deps): Bump cloud.google.com/go/storage from 1.39.0 to 1.39.1 #5687, [Merged by Bors] - build(deps): Bump azure/setup-kubectl from 3 to 4 #5686 - maybe but I'm not sure if they made tests flakier
• Fixes to migrations (unused in tests): [Merged by Bors] - Discard stale state instead of failing migration #5679
• Fixes to flaky tests: [Merged by Bors] - lower interval so that test doesn't stuck waiting 4h #5683, [Merged by Bors] - Flaky test TestRewardService_List/coinbase #5680

On March 11th we had a long string of failing merges because of flaky tests that should be fixed now. I am not sure a recent merge caused the issues we see 🤔

[pigmej]

Env is the same, no changes on env side at all.

[spacemesh-bors]

Build failed:

• systest-status

[dshulyak]

last test failed because 1 node didn't publish atx. this test failed the same way twice in this change. even if it is not related it doesn't make sense to retry. someone needs to find the root cause of that or just disable that test.

https://grafana.spacemesh.dev/explore?panes=%7B%22_J_%22:%7B%22datasource%22:%22loki-gke-dev%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22test-dvrv%5C%22%7D%20%7C%3D%20%5C%22atx%20published%5C%22%20%7C%20json%20%7C%20epoch%3D7%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki-gke-dev%22%7D,%22editorMode%22:%22code%22%7D,%7B%22refId%22:%22B%22,%22expr%22:%22%7Bnamespace%3D%5C%22test-dvrv%5C%22%7D%20%7C~%20%5C%22App%20version%7Csync%20state%5C%22%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki-gke-dev%22%7D,%22editorMode%22:%22code%22%7D%5D,%22range%22:%7B%22from%22:%221710536219108%22,%22to%22:%221710538854064%22%7D%7D%7D&schemaVersion=1&orgId=1

regarding grpc failures i will take a look on monday what can cause that

[fasmat]

Manually merged.

[fasmat]

@dshulyak The node probably didn't publish an ATX because of this error which happened in epoch 5 and persisted for the rest of the run:

{"L":"WARN","T":"2024-03-15T21:22:15.583Z","N":"node.sync","M":"background atx sync failed","sessionId":"b4540da2","epoch_id":5,"errmsg":"failed to persist state for epoch 5: insert synced atx id 5/f34cac63b5 failed: prepare insert into atx_sync_state \n\t\t(epoch, id, requests) values (?1, ?2, ?3)\n\t\ton conflict(epoch, id) do update set requests = ?3;: sqlite.Stmt.ClearBindings: SQLITE_INTERRUPT (insert into atx_sync_state \n\t\t(epoch, id, requests) values (?1, ?2, ?3)\n\t\ton conflict(epoch, id) do update set requests = ?3;)","name":"sync"}

{"L":"WARN","T":"2024-03-15T21:23:15.606Z","N":"node.sync","M":"background atx sync failed","sessionId":"0d233121","epoch_id":6,"errmsg":"failed to persist state for epoch 6: begin: sqlite.Stmt.Step: SQLITE_BUSY: database is locked (BEGIN IMMEDIATE;)","name":"sync"}  
{"L":"WARN","T":"2024-03-15T21:24:15.621Z","N":"node.sync","M":"background atx sync failed","sessionId":"a2b64219","epoch_id":7,"errmsg":"failed to persist state for epoch 7: begin: sqlite.Stmt.Step: SQLITE_BUSY: database is locked (BEGIN IMMEDIATE;)","name":"sync"}
{"L":"WARN","T":"2024-03-15T21:25:15.640Z","N":"node.sync","M":"background atx sync failed","sessionId":"e9039bf3","epoch_id":8,"errmsg":"failed to persist state for epoch 8: begin: sqlite.Stmt.Step: SQLITE_BUSY: database is locked (BEGIN IMMEDIATE;)","name":"sync"}

https://grafana.spacemesh.dev/explore?panes=%7B%22_J_%22:%7B%22datasource%22:%22loki-gke-dev%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22test-dvrv%5C%22,%20pod%3D%5C%22smesher-4-7469ccd99d-8rzpr%5C%22%7D%20%7C~%20%5C%22ERROR%7CWARN%5C%22%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki-gke-dev%22%7D,%22editorMode%22:%22code%22%7D%5D,%22range%22:%7B%22from%22:%221710536160000%22,%22to%22:%221710538860000%22%7D%7D%7D&schemaVersion=1&orgId=1