Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: remove external dependency from the datanode migrate-ipfs subcommand #10700

Merged
merged 4 commits into from
Feb 19, 2024

Conversation

daniel1302
Copy link
Contributor

@daniel1302 daniel1302 commented Feb 18, 2024

Why?

The data node was downloading the required binaries from IPFS. During the validators-testnet, We observed 504 errors for hardcoded IPFS gateways. We cannot rely on it.

Change

At the moment data-node needs to download external binaries to run the migration for the ipfs repository. However, binaries are implemented in the golang, and it is a better idea to compile it into the vega binary.

Testing

vega datanode migrate-ipfs

# go run main.go cmd/vega/main.go datanode migrate-ipfs --home /home/daniel/vega_home2

applying 13-to-14 repo migration
locking repo at "/home/daniel/vega_home2/state/data-node/networkhistory/store/ipfs"
  - verifying version is '13'
> Upgrading config to new format
updated version file
Migration 13 to 14 succeeded
applying 14-to-15 repo migration
locking repo at "/home/daniel/vega_home2/state/data-node/networkhistory/store/ipfs"
  - verifying version is '14'
> Upgrading config to new format
updated version file
Migration 14 to 15 succeeded

#  go run main.go cmd/vega/main.go datanode migrate-ipfs --home /home/daniel/vega_home2
2024-02-18T22:34:48.936+0100	INFO	root	ipfs/migration.go:55	The IPFS for the network-history is up to date. Migration not needed

vega datanode start

# go run main.go cmd/vega/main.go datanode start --home /home/daniel/vega_home2

2024-02-18T22:54:16.874+0100	INFO	datanode.cfgwatcher	config/watcher.go:94	config watcher started successfully	{"config": "/home/daniel/vega_home2/config/data-node/config.toml"}
2024-02-18T22:54:16.875+0100	INFO	datanode.start.persistentPre	start/node_pre.go:70	Starting Vega Datanode	{"version": "v0.74.2", "version-hash": ""}
2024-02-18T22:54:16.875+0100	INFO	datanode.start.persistentPre	start/node_pre.go:111	Migrating the IPFS storage to the latest version
applying 13-to-14 repo migration
locking repo at "/home/daniel/vega_home2/state/data-node/networkhistory/store/ipfs"
  - verifying version is '13'
> Upgrading config to new format
updated version file
Migration 13 to 14 succeeded
applying 14-to-15 repo migration
locking repo at "/home/daniel/vega_home2/state/data-node/networkhistory/store/ipfs"
  - verifying version is '14'
> Upgrading config to new format
updated version file
Migration 14 to 15 succeeded
...

# go run main.go cmd/vega/main.go datanode start --home /home/daniel/vega_home2
2024-02-18T22:56:13.931+0100	INFO	datanode.cfgwatcher	config/watcher.go:94	config watcher started successfully	{"config": "/home/daniel/vega_home2/config/data-node/config.toml"}
2024-02-18T22:56:13.932+0100	INFO	datanode.start.persistentPre	start/node_pre.go:70	Starting Vega Datanode	{"version": "v0.74.2", "version-hash": ""}
2024-02-18T22:56:13.932+0100	INFO	datanode.start.persistentPre	start/node_pre.go:111	Migrating the IPFS storage to the latest version
2024-02-18T22:56:15.776+0100	INFO	datanode.start.persistentPre	ipfs/migration.go:50	The IPFS for the network-history is up to date. Migration not needed

...
...

@daniel1302 daniel1302 merged commit f2d3bc6 into develop Feb 19, 2024
14 checks passed
@daniel1302 daniel1302 deleted the improve-ipfs-sync branch February 19, 2024 11:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants