Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Package the pyspec #1584

Merged
merged 9 commits into from Feb 9, 2020
Merged

Package the pyspec #1584

merged 9 commits into from Feb 9, 2020

Conversation

protolambda
Copy link
Collaborator

@protolambda protolambda commented Jan 22, 2020

Branched of my remerkleable PR, with the phase1 PR merged in.

Changes:

  • Integration things to make remerkleable work for test generation etc.
  • Organize config loading code better
  • Do away with the spec build code, move in the core logic to setup.py, as a dist-utils command, and integrated into default build + dev command.
  • Updated build/makefile things, read the readme pyspec doc in the tests/core/pyspec dir for usage instructions.

With this PR, it becomes super easy to install the spec as dependency, and write python tools to toy around with the spec.

Example set up:

#!/bin/bash

mkdir spec-experiment
cd spec-experiment

python3 -m venv venv
. venv/bin/activate

# Or use your branch of choice
pip install "git+https://github.com/ethereum/eth2.0-specs@pkg-the-pyspec"

# Or editable mode (make sure to run pyspecdev to get editable spec files, instead of relying on re-building the spec):
# pip install -e /path/to/spec
# cd /path/to/spec && python setup.py pyspecdev

echo "
from eth2spec.phase0 import spec
print(spec.BeaconState)
" >> experiment_code.py

python3 experiment_code.py

Or in steps:

  • create your venv (as with any project)
  • one command to install spec from git (whatever branch you need, or locally) with pip
  • from eth2spec.phase0 import spec (phase 1 works too)
  • write some fun python!

It's that easy now 馃帀

@protolambda
Copy link
Collaborator Author

protolambda commented Jan 24, 2020

Rebased and squashed the commits to do the packaging, branched off from the remerkleable PR. Changing base when that is merged.

@protolambda protolambda force-pushed the pkg-the-pyspec branch 4 times, most recently from ee5789f to a97f8f6 Compare Jan 25, 2020
See tests/core/pyspec/README.md for usage description.

This commit:
- refactors config loading to be part of the pyspec package
- updates requirements and main files to use new config loading
- cleans up the build script
- converts the build script to a distutil command
- runs pyspec build as part of build package command
- provides pyspecdev command to get editable spec python files
@protolambda protolambda marked this pull request as ready for review Jan 25, 2020
@protolambda protolambda requested a review from djrtwo Jan 25, 2020
hwwhww
hwwhww approved these changes Jan 25, 2020
Copy link
Contributor

@hwwhww hwwhww left a comment

I like this direction of removing requirements-testing.txt!
Mostly nitpicks. Great work!

setup.py Outdated Show resolved Hide resolved
setup.py Outdated Show resolved Hide resolved
setup.py Outdated Show resolved Hide resolved
tests/generators/epoch_processing/main.py Show resolved Hide resolved
tests/core/pyspec/README.md Outdated Show resolved Hide resolved
setup.py Outdated Show resolved Hide resolved
setup.py Outdated Show resolved Hide resolved
setup.py Outdated Show resolved Hide resolved
setup.py Outdated Show resolved Hide resolved
setup.py Outdated Show resolved Hide resolved
@protolambda
Copy link
Collaborator Author

protolambda commented Jan 27, 2020

Although this feature is mostly done, I think it's good to hold off this PR to include the next version upgrade of remerkleable; much more readable repr (even of mainnet beacon state), more extensibility of the tree (lazily loaded subtrees are possible now), and extra's such as leaf-iteration of trees. It basically makes the use of the spec for tooling even better :)

@djrtwo
Copy link
Contributor

djrtwo commented Jan 29, 2020

@protolambda Is the recent commit (e118045) the version bump you were referencing?

@protolambda
Copy link
Collaborator Author

protolambda commented Jan 29, 2020

@djrtwo Yes, see remerkleable changelog.

And now #1596, and a minor build_spec merge conflict, is all that is left.

djrtwo
djrtwo approved these changes Jan 30, 2020
Copy link
Contributor

@djrtwo djrtwo left a comment

awesome! one minor comment.

Var name tripped me up until I saw how it was used.

setup.py Outdated Show resolved Hide resolved
@protolambda
Copy link
Collaborator Author

protolambda commented Jan 31, 2020

Merged in dev to resolve merge conflict quickly.

Last open question is: do we include the spec version, and if so, where?

I'm ok with just putting 0.0.1 in there for now, and properly version it in some way before we make a release out of dev. What do you think?

@hwwhww
Copy link
Contributor

hwwhww commented Jan 31, 2020

IMHO it would be useful if the pyspec version is equal to the master release version. However practically, it requires more love when releasing the spec, could be done with a script that bumps master release + pyspec version altogether.

(edited)
Putting 0.0.1 for the initial stub seems fine. 馃憤

@djrtwo
Copy link
Contributor

djrtwo commented Feb 1, 2020

Yeah, agreed. stub fine.
Let's figure out a flow to integrate version before next release

@djrtwo
Copy link
Contributor

djrtwo commented Feb 9, 2020

excellent job!

@djrtwo djrtwo merged commit c2456a2 into dev Feb 9, 2020
10 checks passed
@djrtwo djrtwo deleted the pkg-the-pyspec branch Feb 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants