Node issue post upgrade from v1.16.0 to saffron v1.17.0

[Palak3196]

Hi Team,

We have been running an archival provenance node with docker and have been updating it accordingly over the past year.

Our node stopped syncing at block 13736000, using image: provenanceio/provenance:v1.16.0
At this block, we went for upgrade to Saffrom v.1.17.0:

• Stopped the existing node container.
• Started it back again with command: docker run -d --rm -it --name node-prov -p 1317:1317 -p 26657:26657 -p 9090:9090 -v /DataDisk/provenance-arch-node:/home/provenance provenanceio/provenance:v1.17.0 provenanced start

But the node logs are stuck at this for the past 48 hours:

Please let us know if any changes are needed to the docker command we're using.

Thank you for supporting us for more than a year now!

Palak

[nullpointer0x00]

Hi @Palak3196 ,

Do you happen to know what database your node is using. Is it the cleveldb or goleveldb?

[Palak3196]

@nullpointer0x00 Nope, Is there a way to find this out?

[Palak3196]

Hi Team, Still waiting for help here. Our node is still down

[Taztingo]

@Palak3196 We have a dbmigrate tool in the Provenance repo. I would recommend backing up your $PIO_HOME/data and trying that out.

You can build and install dbmigrate with the following commands
make build-dbmigrate
make install-dbmigrate

You can then run the dbmigrate command with goleveldb set as your target.
Make sure $PIO_HOME is the home directory you wish to migrate otherwise include the --home flag.
dbmigrate goleveldb

This should hopefully update your databases and allow you to continue running.

[Palak3196]

@Taztingo How do i do it for a node that we run using docker?

[Taztingo]

It looks like you have configured your volume to be at /DataDisk/provenance-arch-node:/home/provenance. You can verify if this is your home directory by checking if the config and data directories exist at /DataDisk/provenance-arch-node. You should be able to just run dbmigrate goleveldb --home /DataDisk/provenance-arch-node. This will migrate all your databases for your container.

[Palak3196]

sounds perfect, let me try. Thank you for the quick support

[Palak3196]

Hi @Taztingo, do I need to have provenanced installed for the dbmigrate command to run? Because it doesn't run if I simply docker exec -it inside the container.

And yes the config and data directories exist at /DataDisk/provenance-arch-node

[Taztingo]

The docker images were updated to no longer use CLEVELDB in #1709 so make sure your docker containers are up to date. The dbmigrate binary is separate and is not installed with the provenanced binary. You will either need to clone the provenanced repo on your host or inside the docker container. The commands I listed above will help you install and use dbmigrate.

[Palak3196]

Okay, I can understand whats to be done, phew!

I'm not sure if I should open a new issue for this but I tried to install provenanced using the latest git repo code (with go installed) on my host machine.

This is what I get once I run 'make install'

Post this 'provenanced version --long' doesn't recognise provananced as a command

[Taztingo]

You may want to run whereis provenanced to find the exact location of the binary and verify that is part of your PATH.

[Palak3196]

Hi, unfortunately, whereis provenanced also gives no response

I have tried the installation on Ubuntu 18.04 and 20.04.

[iramiller]

This means that you do not have your GOBIN in your path ... this is the folder where go will install binaries built with make build

%%> echo $GOBIN
/Users/[...]/Projects/go/bin

and

 %%> echo $PATH
/opt/homebrew/opt/libpq/bin /opt/homebrew/bin /opt/homebrew/sbin /opt/homebrew/opt/fzf/bin /Users/[...]/Projects/go/bin [...]

This is why I am able to resolve my installed provenanced binary like so

%%> which provenanced
/Users/[...]/Projects/go/bin/provenanced

[Palak3196]

Thanks @iramiller I added GOBIN in the path as suggested:

And then ran make build in the provenance binaries' directory (taken from v 1.17.0), but it ran into errors (attached in the note)
Prov error.txt

Also, I saw the release notes for Release v1.17.1 and tried the below steps too to goleveldb from cleveldb:
Stop your node.
Back up your data and config directories.
Update your config.toml to have db_backend = "goleveldb".
Update your app.toml to have either app-db-backend = "" or app-db-backend = "goleveldb".
Restart your node.

For this, my docker container runs into the below error:

Sorry that's a lot of information! Let me know if there's anything wrong here

[Taztingo]

@Palak3196 It looks like you are having an issue with ziphash when attempting to build. This may be an issue with your go modcache.

As a follow up, what binary are you attempting to use? Is this the binary you are attempting to build, or are you using the prebuilt 1.17.1?