Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Globaldb migration: new tables and move assets #3400

Merged
merged 11 commits into from Nov 2, 2021

Conversation

yabirgb
Copy link
Member

@yabirgb yabirgb commented Sep 1, 2021

Danger

This branch will modify your globaldb and break it. Make a copy of it before executing ANY code of it.

What has been done

  • Create new tables
  • Scripts to move assets from old database to new database
  • Modify globaldb handler to manage new ids for evm compatible assets

Copy link
Member

@LefterisJP LefterisJP left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some initial comments.

@@ -622,7 +622,7 @@ def __post_init__(
object.__setattr__(self, 'decimals', data.decimals)
object.__setattr__(self, 'protocol', data.protocol)

underlying_tokens = GlobalDBHandler().fetch_underlying_tokens(data.ethereum_address)
underlying_tokens = GlobalDBHandler().fetch_underlying_tokens(data.identifier)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to rename some things. Will do in the next PR

rotkehlchen/constants/resolver.py Outdated Show resolved Hide resolved
rotkehlchen/constants/resolver.py Outdated Show resolved Hide resolved
rotkehlchen/constants/resolver.py Outdated Show resolved Hide resolved
rotkehlchen/globaldb/handler.py Outdated Show resolved Hide resolved
rotkehlchen/globaldb/upgrades/v2_v3.py Outdated Show resolved Hide resolved
rotkehlchen/globaldb/upgrades/v2_v3.py Outdated Show resolved Hide resolved
return {k.value: addr for k in CHAINID}


def query_coingecko_for_addresses(identifier: str) -> Dict[str, str]: # noqa: E501 pylint: disable=unused-argument
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed. This CAN NOT be part of the actual DB migration. This can only happen after the migration is complete, the user has logged in and a long-running task can executed.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I agree. Won't use it right here but needed a function to query ids. Will move this function to a different place since I don't use it


for entry in query:
try:
ids = query_coingecko_for_addresses(entry[6])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Querying any external service, especially in a for loop should not be part of a DB upgrade, ever. A DB upgrade is one of the most sensitive things we can do and if it goes wrong the user's installation may be fucked. So we need to keep it as simple as possible.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah. This logic has to move but it will be in another PR. They ids and information would come from the packaged globaldb but at this point I needed something to simulate this and test that the migration works. Also will be useful to move it into the script we will use to create the new globaldb

rotkehlchen/globaldb/upgrades/v2_v3.py Outdated Show resolved Hide resolved
Copy link
Member

@LefterisJP LefterisJP left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed in standup, do the 1 change I told you either here or in the next PR. And we can merge this so you can continue work in the branch.

@@ -148,6 +160,49 @@
);
"""

DB_V3_CREATE_COMMON_ASSETS_DETAILS = """
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And this won't stay right? We willl only have 1 version of creation command per table.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking in someone comming from an old version but I believe it doesn't make sense? We can go ahead and remove the versions in the names

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it doesn't make sense. We never keep old version creation schemas around. That's what DB upgrades are for.

Copy link
Member

@LefterisJP LefterisJP left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed yesterday ,will just merge this and later review the long lasting branch

@LefterisJP LefterisJP merged commit 0b79017 into rotki:multichain_globaldb_hell Nov 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants