Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
feat: Voting power snapshot automation script (#294)
In this PR: - Started using Poetry for dep management - Added import snapshot command but still missing some parts of the workflow: - Couldn't test it with the working snapshot_tool (field names will probably change) - Couldn't get dreps from GVC API I'm opening this as it is to be able to work on other stuff while I figure out the missing parts. PS: I'll rename the tool in a future PR (it's not only ideascale importer anymore). --------- Co-authored-by: Sasha Prokhorenko <djminikin@gmail.com>
- Loading branch information
1 parent
d1123bb
commit 1bb027c
Showing
35 changed files
with
1,993 additions
and
390 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,4 @@ | ||
venv | ||
.venv | ||
__pycache__ | ||
.mypy_cache | ||
snapshot-importer-config.dev.json |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
3.11 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
# Importing IdeaScale Data | ||
|
||
The easiest way is to run: | ||
|
||
```sh | ||
PYTHONPATH=$(pwd) poetry run python ideascale_importer \ | ||
--api-token IDEASCALE_API_TOKEN \ | ||
--database-url POSTGRES_URL | ||
``` | ||
|
||
And go through the interactive steps. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
# Importing Snapshot Data | ||
|
||
## Configuration | ||
|
||
See the [example config](../snapshot-importer-example-config.json) for the available configuration fields. | ||
|
||
## Command | ||
|
||
In order to import snapshot data you'll need: | ||
|
||
1. a database with the latest `event-db` migrations applied with an event row inserted | ||
3. another database populated with dbsync | ||
4. the snapshot_tool binary | ||
5. the catalyst-toolbox binary | ||
|
||
*The script uses the event information in the database to define voting power threshold and max voting power percentage.* | ||
|
||
With that you can run: | ||
|
||
```sh | ||
PYTHONPATH=(pwd) poetry run python ideascale_importer snapshot import --config-path PATH_TO_CONFIG_FILE --event-id EVENT_ROW_ID --database-url VITSS_DB_URL --output-dir OUTDIR_PATH | ||
``` | ||
|
||
If everything went as expected you should have snapshot, voters and contributions data inserted to the database. |
File renamed without changes.
File renamed without changes.
12 changes: 12 additions & 0 deletions
12
utilities/ideascale-importer/ideascale_importer/__main__.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
import typer | ||
import ideascale_importer.cli.db | ||
import ideascale_importer.cli.ideascale | ||
import ideascale_importer.cli.snapshot | ||
|
||
|
||
app = typer.Typer(add_completion=False) | ||
app.add_typer(ideascale_importer.cli.db.app, name="db", help="Postgres DB commands (e.g. seeding)") | ||
app.add_typer(ideascale_importer.cli.ideascale.app, name="ideascale", help="IdeaScale commands (e.g. importing data)") | ||
app.add_typer(ideascale_importer.cli.snapshot.app, name="snapshot", help="Snapshot commands (e.g. importing data)") | ||
|
||
app() |
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
43 changes: 43 additions & 0 deletions
43
utilities/ideascale-importer/ideascale_importer/cli/snapshot.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
import asyncio | ||
import typer | ||
|
||
from ideascale_importer.snapshot_importer.importer import Importer | ||
|
||
app = typer.Typer(add_completion=False) | ||
|
||
|
||
@app.command(name="import") | ||
def import_snapshot( | ||
config_path: str = typer.Option(..., help="Path to the configuration file"), | ||
event_id: int = typer.Option(..., help="Database event id to link all snapshot data to"), | ||
database_url: str = typer.Option(..., help="URL of the Postgres database in which to import the data to"), | ||
output_dir: str = typer.Option(..., help="Output directory for generated files"), | ||
raw_snapshot_file: str = typer.Option( | ||
None, | ||
help=( | ||
"Raw snapshot file generated by Catalyst snapshot_tool." | ||
"If this is set, running snapshot_tool will be skipped and the contents of this file will be used" | ||
) | ||
), | ||
dreps_file: str = typer.Option( | ||
None, | ||
help=( | ||
"Should be a file containing the list of dreps as returned by the GVC API." | ||
"If this is set, calling GVC dreps API will be skipped and the contents of this file will be used" | ||
) | ||
) | ||
): | ||
""" | ||
Import snapshot data into the database | ||
""" | ||
|
||
async def inner(): | ||
importer = Importer(config_path=config_path, | ||
database_url=database_url, | ||
event_id=event_id, | ||
output_dir=output_dir, | ||
raw_snapshot_file=raw_snapshot_file, | ||
dreps_file=dreps_file) | ||
await importer.import_all() | ||
|
||
asyncio.run(inner()) |
Oops, something went wrong.