Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
feat: Voting power snapshot automation script (#294)
In this PR:

- Started using Poetry for dep management
- Added import snapshot command but still missing some parts of the
workflow:
- Couldn't test it with the working snapshot_tool (field names will
probably change)
   - Couldn't get dreps from GVC API

I'm opening this as it is to be able to work on other stuff while I
figure out the missing parts.

PS: I'll rename the tool in a future PR (it's not only ideascale
importer anymore).

---------

Co-authored-by: Sasha Prokhorenko <djminikin@gmail.com>
  • Loading branch information
FelipeRosa and minikin committed Mar 23, 2023
1 parent d1123bb commit 1bb027c
Show file tree
Hide file tree
Showing 35 changed files with 1,993 additions and 390 deletions.
3 changes: 2 additions & 1 deletion utilities/ideascale-importer/.gitignore
@@ -1,3 +1,4 @@
venv
.venv
__pycache__
.mypy_cache
snapshot-importer-config.dev.json
1 change: 1 addition & 0 deletions utilities/ideascale-importer/.python-version
@@ -0,0 +1 @@
3.11
30 changes: 13 additions & 17 deletions utilities/ideascale-importer/README.md
Expand Up @@ -3,46 +3,42 @@ IdeaScale Importer

## Getting Started

First install [pyenv](https://github.com/pyenv/pyenv#installation) AND [pyenv virtualenv](https://github.com/pyenv/pyenv-virtualenv).
We recommend installing [pyenv](https://github.com/pyenv/pyenv#installation) to manage Python versions.

Then create a virtual environment and install dependencies:
Install Python 3.11:

```sh
pyenv install 3.11.1
pyenv virtualenv 3.11.1 ideascale-importer-venv-3.11.1
pyenv activate ideascale-importer-venv-3.11.1
pip install -r requirements.txt
pyenv install 3.11
```

To see the available commands:
Install [Poetry](https://python-poetry.org/docs/#installation). Then install dependencies:

```sh
python src/main.py --help
poetry env use python
poetry install
```

## Importing IdeaScale Data

The easiest way is to run:
To see the available commands:

```sh
python src/main.py \
--api-token IDEASCALE_API_TOKEN \
--database-url POSTGRES_URL
PYTHONPATH=$(pwd) poetry run python ideascale_importer --help
```

And go through the interactive steps.
## Documentation

For documentation about the available commands see the [docs](docs) folder.

## Development

### Linting

```sh
# If you haven't already:
python -m flake8 src
poetry run python -m flake8 ideascale_importer
```

### Type checking

```sh
python -m mypy src --check-untyped-defs
poetry run python -m mypy ideascale_importer --check-untyped-defs
```
11 changes: 11 additions & 0 deletions utilities/ideascale-importer/docs/ideascale.md
@@ -0,0 +1,11 @@
# Importing IdeaScale Data

The easiest way is to run:

```sh
PYTHONPATH=$(pwd) poetry run python ideascale_importer \
--api-token IDEASCALE_API_TOKEN \
--database-url POSTGRES_URL
```

And go through the interactive steps.
24 changes: 24 additions & 0 deletions utilities/ideascale-importer/docs/snapshot.md
@@ -0,0 +1,24 @@
# Importing Snapshot Data

## Configuration

See the [example config](../snapshot-importer-example-config.json) for the available configuration fields.

## Command

In order to import snapshot data you'll need:

1. a database with the latest `event-db` migrations applied with an event row inserted
3. another database populated with dbsync
4. the snapshot_tool binary
5. the catalyst-toolbox binary

*The script uses the event information in the database to define voting power threshold and max voting power percentage.*

With that you can run:

```sh
PYTHONPATH=(pwd) poetry run python ideascale_importer snapshot import --config-path PATH_TO_CONFIG_FILE --event-id EVENT_ROW_ID --database-url VITSS_DB_URL --output-dir OUTDIR_PATH
```

If everything went as expected you should have snapshot, voters and contributions data inserted to the database.
File renamed without changes.
12 changes: 12 additions & 0 deletions utilities/ideascale-importer/ideascale_importer/__main__.py
@@ -0,0 +1,12 @@
import typer
import ideascale_importer.cli.db
import ideascale_importer.cli.ideascale
import ideascale_importer.cli.snapshot


app = typer.Typer(add_completion=False)
app.add_typer(ideascale_importer.cli.db.app, name="db", help="Postgres DB commands (e.g. seeding)")
app.add_typer(ideascale_importer.cli.ideascale.app, name="ideascale", help="IdeaScale commands (e.g. importing data)")
app.add_typer(ideascale_importer.cli.snapshot.app, name="snapshot", help="Snapshot commands (e.g. importing data)")

app()
@@ -1,11 +1,13 @@
import asyncio
from datetime import datetime
import db
from db import models
import random
import rich
import typer

import ideascale_importer.db
from ideascale_importer.db import models


app = typer.Typer(add_completion=False)


Expand All @@ -18,11 +20,11 @@ def seed_compatible(database_url: str = typer.Option(..., help="Postgres databas
async def inner(database_url: str):
console = rich.console.Console()

conn = await db.connect(database_url)
console.log("Connected to database")
conn = await ideascale_importer.db.connect(database_url)
console.print("Connected to database")

async with conn.transaction():
election = models.Election(
event = models.Event(
name="Fund 10",
description="Fund 10 event",
registration_snapshot_time=datetime.now(),
Expand All @@ -42,35 +44,35 @@ async def inner(database_url: str):
tallying_end=datetime.now(),
extra={
"url": {
"results": "https://election.com/results/10",
"survey": "https://election.com/survey/10",
"results": "https://event.com/results/10",
"survey": "https://event.com/survey/10",
}
})
election_id = await conn.insert(election, returning="row_id")
console.log(f"Inserted election row_id={election_id}")
event_id = await ideascale_importer.db.insert(conn, event, returning="row_id")
console.print(f"Inserted event row_id={event_id}")

voting_group = models.VotingGroup(group_id="group-id-1", election_id=election_id, token_id="token-id-1")
voting_group_row_id = await conn.insert(voting_group, returning="row_id")
console.log(f"Inserted voting_group row_id={voting_group_row_id}")
voting_group = models.VotingGroup(group_id="group-id-1", event_id=event_id, token_id="token-id-1")
voting_group_row_id = await ideascale_importer.db.insert(conn, voting_group, returning="row_id")
console.print(f"Inserted voting_group row_id={voting_group_row_id}")

voteplan = models.Voteplan(election_id=election_id, id="voteplan-1", category="public",
voteplan = models.Voteplan(event_id=event_id, id="voteplan-1", category="public",
encryption_key="encryption-key-1", group_id=voting_group_row_id)
voteplan_row_id = await conn.insert(voteplan, returning="row_id")
console.log(f"Inserted voteplan row_id={voteplan_row_id}")
voteplan_row_id = await ideascale_importer.db.insert(conn, voteplan, returning="row_id")
console.print(f"Inserted voteplan row_id={voteplan_row_id}")

for i in range(2):
challenge_id = random.randint(1, 1000)

challenge = models.Challenge(
id=challenge_id,
election=election_id,
event=event_id,
category="simple",
title=f"Challenge {i}",
description=f"Random challenge {i}",
rewards_currency="ADA",
rewards_total=100000,
proposers_rewards=10000,
vote_options=await conn.get_vote_options_id("yes,no"),
vote_options=await ideascale_importer.db.get_vote_options_id(conn, "yes,no"),
extra={
"url": {
"challenge": f"https://challenge.com/{i}"
Expand All @@ -80,8 +82,8 @@ async def inner(database_url: str):
},
})

challenge_row_id = await conn.insert(challenge, returning="row_id")
console.log(f"Inserted challenge row_id={challenge_row_id}")
challenge_row_id = await ideascale_importer.db.insert(conn, challenge, returning="row_id")
console.print(f"Inserted challenge row_id={challenge_row_id}")

for j in range(2):
proposal_id = random.randint(1, 1000)
Expand All @@ -106,16 +108,16 @@ async def inner(database_url: str):
bb_vote_options="yes,no"
)

proposal_row_id = await conn.insert(proposal, returning="row_id")
console.log(f"Inserted proposal row_id={proposal_row_id}")
proposal_row_id = await ideascale_importer.db.insert(conn, proposal, returning="row_id")
console.print(f"Inserted proposal row_id={proposal_row_id}")

proposal_voteplan = models.ProposalVoteplan(
proposal_id=proposal_row_id, voteplan_id=voteplan_row_id, bb_proposal_index=(i+1)*(j+1))
await conn.insert(proposal_voteplan)
await ideascale_importer.db.insert(conn, proposal_voteplan)

for i in range(1):
goal = models.Goal(election_id=election_id, idx=i, name=f"Goal {i}")
goal_id = await conn.insert(goal, returning="id")
console.log(f"Inserted goal id={goal_id}")
goal = models.Goal(event_id=event_id, idx=i, name=f"Goal {i}")
goal_id = await ideascale_importer.db.insert(conn, goal, returning="id")
console.print(f"Inserted goal id={goal_id}")

asyncio.run(inner(database_url))
Expand Up @@ -2,7 +2,7 @@
from typing import Optional
import typer

from importer import Importer
from ideascale_importer.ideascale.importer import Importer

app = typer.Typer(add_completion=False)

Expand All @@ -11,13 +11,13 @@
def import_all(
api_token: str = typer.Option(..., help="IdeaScale API token"),
database_url: str = typer.Option(..., help="Postgres database URL"),
election_id: Optional[int] = typer.Option(
event_id: Optional[int] = typer.Option(
None,
help="Database row id of the election which data will be imported",
help="Database row id of the event which data will be imported",
),
campaign_group_id: Optional[int] = typer.Option(
None,
help="IdeaScale campaign group id for the election which data will be imported",
help="IdeaScale campaign group id for the event which data will be imported",
),
stage_id: Optional[int] = typer.Option(
None,
Expand All @@ -29,11 +29,11 @@ def import_all(
),
):
"""
Import all election data from IdeaScale for a given election
Import all event data from IdeaScale for a given event
"""

async def inner(
election_id: Optional[int],
event_id: Optional[int],
campaign_group_id: Optional[int],
stage_id: Optional[int],
proposals_scores_csv_path: Optional[str]
Expand All @@ -42,7 +42,7 @@ async def inner(
api_token,
database_url,
None,
election_id,
event_id,
campaign_group_id,
stage_id,
proposals_scores_csv_path,
Expand All @@ -52,4 +52,4 @@ async def inner(
await importer.import_all()
await importer.close()

asyncio.run(inner(election_id, campaign_group_id, stage_id, proposals_scores_csv))
asyncio.run(inner(event_id, campaign_group_id, stage_id, proposals_scores_csv))
43 changes: 43 additions & 0 deletions utilities/ideascale-importer/ideascale_importer/cli/snapshot.py
@@ -0,0 +1,43 @@
import asyncio
import typer

from ideascale_importer.snapshot_importer.importer import Importer

app = typer.Typer(add_completion=False)


@app.command(name="import")
def import_snapshot(
config_path: str = typer.Option(..., help="Path to the configuration file"),
event_id: int = typer.Option(..., help="Database event id to link all snapshot data to"),
database_url: str = typer.Option(..., help="URL of the Postgres database in which to import the data to"),
output_dir: str = typer.Option(..., help="Output directory for generated files"),
raw_snapshot_file: str = typer.Option(
None,
help=(
"Raw snapshot file generated by Catalyst snapshot_tool."
"If this is set, running snapshot_tool will be skipped and the contents of this file will be used"
)
),
dreps_file: str = typer.Option(
None,
help=(
"Should be a file containing the list of dreps as returned by the GVC API."
"If this is set, calling GVC dreps API will be skipped and the contents of this file will be used"
)
)
):
"""
Import snapshot data into the database
"""

async def inner():
importer = Importer(config_path=config_path,
database_url=database_url,
event_id=event_id,
output_dir=output_dir,
raw_snapshot_file=raw_snapshot_file,
dreps_file=dreps_file)
await importer.import_all()

asyncio.run(inner())

0 comments on commit 1bb027c

Please sign in to comment.