diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b721d1a9..ee661748 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -40,17 +40,9 @@ Once the virtualization-sdk repository is forked, clone the forked repository in `git clone git@github.com:your-username/virtualization-sdk.git` -### Build the virtualization-sdk Repository +### Development -The virtualization-sdk repository has a Gradle task that creates a virtualenv and builds the code locally. It can be triggered from the root directory of the source tree using the command: - -`./gradlew build` - -### Run Unit Tests - -Unit tests for the virtualization-sdk repository can be triggered from the root directory of the source tree using Gradle with the command: - -`./gradlew test` +For development instructions, refer to [README-dev.md](https://github.com/delphix/virtualization-sdk/blob/develop/README-dev.md). ## Asking for Help Please raise a GitHub issue to ask for help with appropriate GitHub tag . @@ -74,7 +66,7 @@ Some guidelines for Pull Requests: Once the pull request has required approvals from code owners of the repository, the code owner will merge the pull request into the actual virtualization-sdk repository. ### Code Owners -Code owners defined by the codeowners file in the repository are the gatekeepers of the repository. For a pull request to be merged, it requires approval from at least 2 codeowners. +Code owners defined by the codeowners file in the repository are the gatekeepers of the repository. For a pull request to be merged, it requires approval from at least one codeowner. ## Testing and CI/CD CI/CD for this repository is managed through GitHub Actions. All the checks need to succeed for the pull request to be merged. @@ -90,5 +82,6 @@ Commit messages for new changes must meet the following guidelines: `Fixes #123 Format of error is incorrect` `Optional Description of the issue` +* If the commit doesn't address a specific issue, it should include a description of changes. diff --git a/README-dev.md b/README-dev.md index f2ddab75..d3593628 100644 --- a/README-dev.md +++ b/README-dev.md @@ -21,49 +21,56 @@ This is done automatically by `dvp build`. This is what causes the slightly different workflows in development. Changes to `tools` are completely isolated from the Delphix Engine and wrappers changes only impact the plugin build. -## Development process +## Development + +### Development process At a very high level, our development process usually looks like this: 1. Create a fork of the delphix/virtualization-sdk repository. 2. Clone the forked repository. -3. Make changes to SDK code. Test these changes manually and with unit tests. Iterate on this until you have everything working. -4. Commit your changes and build all Python package distributions. Make sure the version number of the packages is updated appropriately. -5. Publish Python distributions to artifactory. -7. Run blackbox against the newly uploaded SDK version. -7. Publish a pull request to the delphix/virtualization-sdk once your code is ready for review. -8. Once the pull request is approved, it will merged into delphix/virtualization-sdk repository. +3. Make changes to SDK code. Test these changes manually and with [unit tests](#unit-testing). Iterate on this until you have everything working. +4. Bump major/minor/patch/build version depending on the scope of the change. Refer to [versioning section](#versioning) for instructions. +5. Commit your changes. Refer to the [contribution guideline](https://github.com/delphix/virtualization-sdk/blob/develop/CONTRIBUTING.md#commit-message-format) + for commit message format. +6. Make sure the version number of the packages is updated appropriately in your commit. That includes all files mentioned +in `.bumpversion.cfg` and the string in `test_get_version()` inside `test_package_util.py`. +7. Push your changes to a branch in the forked repository. +8. Run [blackbox tests](#functional-blackbox-testing) against that branch. +9. Publish a pull request to the delphix/virtualization-sdk once your code is ready for review. +10. Once the pull request is approved, merge the pull request into delphix/virtualization-sdk repository. These steps are described in more detail below. -## Local SDK Development +### Development environment +Development should be done in a personal virtualenv. To setup the virtual environment: -To setup local development, refer to README-dev.md in the `tools` directory. This walks through the setup of a local virtualenv for development. This should be done for _all_ SDK changes. +1. `virtualenv /path/to/env/root`. This should be a Python 2.7 virtualenv. +2. `source ~/path/to/env/root/bin/activate`. -### Configure pip index +### Installing the SDK from source +To install the SDK, follow these steps: -`dvp build` executes `pip` to install the wrappers. By default `pip` looks at pypi.org for packages to install. Internal builds of the SDK are published to artifactory, not pypi. In order to configure pip to look at artifactory, create a file at `/pip.conf` that contains: +1. Create a file at + `/pip.conf` that contains: + + ``` + [install] + index-url=https://pypi.org/simple/ + extra-index-url=https://test.pypi.org/simple/ + ``` + + One of the SDK dependencies - dvp-api - is currently hosted on [TestPyPi](https://test.pypi.org/project/dvp-api/). + By default `pip` looks at pypi.org for packages to install. In order to successfully install the SDK, you have to + configure pip to search an additional package repository - test.pypi.org. +2. Go into one of the package directories (common, dvp, libs, platform, tools) and run the commands below. +3. Install the package's development dependencies: `pip install -r requirements.txt`. +4. Install the package itself (use `-e` flag if you want to install the package in editable mode): `pip install .`. -``` -[install] -trusted-host=artifactory.delphix.com -index-url=http://artifactory.delphix.com/artifactory/api/pypi/dvp-virtual-pypi/simple/ -``` ### CLI changes -To better understand how to develop and test `tools` changes, see README-dev.md in the `tools` directory. - -### Wrappers changes - -Run `dvp build --dev` to build your plugin and then upload it to a Delphix Engine to test. - -The wrappers are built with the plugin. `dvp build` has a hidden `--dev` flag. This builds `common`, `libs`, and `platform` locally and bundles them with the plugin. A special configuration entry is needed in your dvp config file which is located at `~/.dvp/config`: - -``` -[dev] -vsdk_root = /path/to/vsdk_repo_root -``` +To better understand how to develop and test `tools` changes, see [tools/README-dev.md](https://github.com/delphix/virtualization-sdk/blob/develop/tools/README-dev.md). ## Versioning @@ -71,8 +78,8 @@ The SDK is shipped as five Python packages that are currently versioned and ship dvp-platform, and dvp-tools. The first thing to do is to change the version number of all the packages. Our versioning scheme follows the rules of -semantic versioning in order to help developers manage their "dependency hell". We use bump2version -(https://github.com/c4urself/bump2version) to make the version management of all five packages easier. Semantic versioning rules are the following: +semantic versioning in order to help developers manage their "dependency hell". We use [bump2version](https://github.com/c4urself/bump2version) +to make the version management of all five packages easier. Semantic versioning rules are the following: ``` Given a version number MAJOR.MINOR.PATCH, increment the: @@ -90,7 +97,7 @@ If you want to bump the build number from `1.1.0-internal-7` to `1.1.0-internal- If you want to bump the major/minor/patch version, run `bumpversion [major|minor|patch]`. -If you want to get rid of the pre-release label (bump from `1.1.1-internal-7` to `1.1.0`), run `bumpversion release`. +If you want to get rid of the pre-release label (bump from `1.1.0-internal-7` to `1.1.0`), run `bumpversion release`. ## Testing @@ -98,47 +105,37 @@ Currently, there are three types of SDK testing: unit, manual, and functional (b ### Unit Testing -Go into one of the package directories (common, dvp, libs, platform, tools) and run the following commands (if you haven't done it already): -1. Install the package's development dependencies: `pip install -r requirements.txt`. -2. Install the package itself in editable mode: `pip install -e .`. -3. Run unit tests: `python -m pytest src/main/python`. +Go into one of the package directories (common, dvp, libs, platform, tools) and follow these steps: -There's no way to locally run unit tests in all packages with one command. However, they will be run automatically through GitHub Actions when you open a pull request. +1. Install the package's development dependencies and package itself by following the directions in [the SDK installation section](#installing-the-sdk-from-source). +2. Run unit tests: `python -m pytest src/test/python`. -### Testing sdk-gate changes with app-gate code +There's no way to locally run unit tests in all packages with one command. However, they will be run automatically +through GitHub Actions when you open a pull request. You can always open a draft pull request -#### Manual testing +### Manual testing -Run `dvp build --dev` to build your plugin and then upload it to a Delphix Engine to test. +#### Wrappers: dvp, common, platform, libs +The only way to manually test the new wrappers code is to build a plugin, upload it to a Delphix Engine and run through +all the standard workflows. The same workflows will be exercised by functional (blackbox) tests. -The wrappers are built with the plugin. `dvp build` has a hidden `--dev` flag. This builds `common`, `libs`, and `platform` locally and bundles them with the plugin. A special configuration entry is needed in your dvp config file which is located at `~/.dvp/config`: - -``` -[dev] -vsdk_root = /path/to/vsdk_repo_root -``` - -##### Functional (blackbox) testing -(Now) Let's assume you're working on the SDK version `1.1.0-internal-10`. -To run blackbox tests, follow these steps: -1. Navigate to each package directory (common, dvp, libs, platform, tools) and run `python setup.py sdist bdist_wheel`. This will build Python package distributions. -2. Run `./bin/upload.sh` to upload Python distributions to artifactory. -2. Navigate to the app-gate directory and run -`git blackbox -s appdata_samples --extra-params="-p sdk-version=1.0.0-internal-10"`. - - -(Soon) We will be able to move to this process once blackbox runner can build Python distributions without using Gradle. +### Functional (blackbox) testing To run blackbox tests, follow these steps: -1. Push your code to a branch in the forked repository on Github. Let's say the branch is called `feature1` in repository called `username/virtualization-sdk`. -2. Navigate to the app-gate directory and run -`git blackbox -s appdata_samples --extra-params="-p virt-sdk-repo=https://github.com/username/virtualization-sdk.git -p virt-sdk-branch=feature1"`. - -#### Setup - -1. There are two environment variables that need to be set in order to publish: `ARTIFACTORY_PYPI_USER` and `ARTIFACTORY_PYPI_PASS`. - - `ARTIFACTORY_PYPI_USER` and `ARTIFACTORY_PYPI_PASS` are one set of credentials used to upload the Python distributions to our internal PyPI repositories. The credentials are the same for both internal PyPI repositories mentioned above. - - - `ARTIFACTORY_PYPI_USER` and `ARTIFACTORY_PYPI_PASS` is the username/password combo given to you by whoever setup your Artifactory pypi account. This is an account separate from your Artifactory account. If you do not have one, please reach out to the `#artifactory` channel and request a `dvp-uploaders-python` account. See for directions on how to add the account. These are used to upload the Python distributions to our internal PyPI repositories. The credentials are the same for both internal PyPI repositories mentioned above. - -2. `twine` needs to be installed. This is a Python package that is used to upload Python distributions. If it's not installed, install it by running `pip install twine`. +1. Push your code to a branch in the forked repository on Github. Let's say the branch is called `my-feature` in repository called `/virtualization-sdk`. +2. Navigate to the app-gate directory and start tests using `git blackbox`. For the guide on which test suite to use, +see the next sections. + +At a minimum, each pull request should pass `appdata_python_samples` and `appdata_sanity` tests with a direct or staged plugin. +See the section below for the description of each test suite. + +#### Blackbox tests targeting wrappers (mostly Delphix Engine workflows) +* appdata_python_samples (sample plugins from the app-gate): +`git blackbox -s appdata_python_samples --extra-params="-p virt-sdk-repo=https://github.com//virtualization-sdk.git -p virt-sdk-branch=my-feature"`, +* appdata_sanity with a direct Python plugin on CentOS 7.3: `git blackbox -s appdata_sanity -c APPDATA_PYTHON_DIRECT_CENTOS73 -a --extra-params="-p virt-sdk-repo=https://github.com//virtualization-sdk.git -p virt-sdk-branch=my-feature"`, +* appdata_sanity with a staged Python plugin on CentOS 7.3: `git blackbox -s appdata_sanity -c APPDATA_PYTHON_STAGED_CENTOS73 -a --extra-params="-p virt-sdk-repo=https://github.com//virtualization-sdk.git -p virt-sdk-branch=my-feature"`. + +#### Blackbox tests targeting the CLI (~80% CLI tests) +* virtualization_sdk (installs and tests a direct Python plugin on Ubuntu 18): +`git blackbox -s virtualization_sdk -c APPDATA_SDK_UBUNTU18_DIRECT_CENTOS73 --extra-params="-p virt-sdk-repo=https://github.com//virtualization-sdk.git -p virt-sdk-branch=my-feature"`, +* virtualization_sdk (installs and tests a staged Python plugin on Ubuntu 18): +`git blackbox -s virtualization_sdk -c APPDATA_SDK_UBUNTU18_STAGED_CENTOS73 --extra-params="-p virt-sdk-repo=https://github.com//virtualization-sdk.git -p virt-sdk-branch=my-feature"`. \ No newline at end of file diff --git a/tools/README-dev.md b/tools/README-dev.md index 0e8e7170..740a1e1d 100644 --- a/tools/README-dev.md +++ b/tools/README-dev.md @@ -1,58 +1,74 @@ # Delphix Virtualization SDK Tools ## Purpose -These tools will be one piece of the Delphix Virtualization SDK. Plugin developers will install these tools locally and use them to develop, build, and ship virtualization plugins. - -## NOTE -This guide assumes that the current working directory is the root of the tools directory where this README is. +The `tools` package represents a CLI of the Delphix Virtualization SDK. Plugin developers will install the CLI to build +and upload virtualization plugins. ## Getting Started ### Development Environment -Development should be done in a personal virtualenv. To setup the virtual environment: - -1. `virtualenv /path/to/env/root`. This should be a Python 2.7 virtualenv. -2. `source ~/path/ot/env/root/bin/activate` -3. `pip install -r requirements.txt`. This installs the required development packages. -4. `pip install -e .` +To setup the development environment, follow the instructions in [README-dev.md](https://github.com/delphix/virtualization-sdk/blob/develop/README-dev.md) -NOTE: If build reports any issues with 'virtualenv' not found and/or errors finding virtualenv in the PATH, check and make sure virtualenv from Python 2.7 is being used and not from 3.x. Few places to check, for e.g. on Mac - /usr/local/bin/virtualenv, ~/Library/Python/2.7/virtualenv. +For quick iterations, install the `tools` package in editable mode (`pip install -e .`). This means that changes to the +code will automatically be reflected in your environment. You will not need to reinstall the tools module each time +a change is made. -You’re ready to start developing. +Changes to the code can be tested using `dvp `. `dvp` is the CLI tool that is built from the source here that +helps build and upload a plugin to DE. -The development environment will already have all development and production dependencies installed in it. The tools module will be installed in ‘editable’ mode. This means that changes to the code will automatically be reflected in your environment. You will not need to reinstall the tools module each time a change is made. +### Adding a command +The CLI uses [Click](https://click.palletsprojects.com/en/7.x/). This is a decorator driven library that builds a full +CLI. -Changes to the code can be tested using `dvp `. `dvp` is the CLI tool that is built from the source here that helps build and upload a plugin to DE. +The starting point for the CLI is `virtualization._internal.cli:delphix_sdk`. This is annotated with the `@click.group` +annotation indicating there will be subcommands. -NOTE: If you run into: `ERROR: The Python zlib extension was not compiled. Missing the zlib?` this is because you're missing a piece of XCode. Install the requirements listed [here](https://github.com/pyenv/pyenv/wiki/Common-build-problems). +All that is needed to add a command is to define a new method in `virtualization._internal.cli`. **The method's name +will be the name of the command.** Annotate the method with `@delphix_sdk.command()`. It will automatically be added +as a subcommand. Look at the [Click documentation](https://click.palletsprojects.com/en/7.x/) to see other annotations +available to configure the command. -## Adding a command -The CLI uses [Click](https://click.palletsprojects.com/en/7.x/). This is a decorator driven library builds a full CLI. +Click should be contained to `virtualization._internal.cli`. The implementation of the method should call immediately +into a method in another module. `virtualization._internal.cli` is the single source of truth for our CLI and should not +contain any business logic. -The starting point for the CLI is `virtualization._internal.cli:delphix_sdk`. This is annotated with the `@click.group` annotation indicating there will be subcommands. +### Testing -All that is needed to add a command is to define a new method in `virtualization._internal.cli`. **The method's name will be the name of the command.** Annotate the method with `@delphix_sdk.command()`. It will automatically be added as a subcommand. Look at the [Click documentation](https://click.palletsprojects.com/en/7.x/) to see other annotations available to configure the command. +#### Manual testing +It's easy to manually test the CLI as you can invoke different SDK CLI commands from command line. -Click should be contained to `virtualization._internal.cli`. The implementation of the method should call immediately into a method in another module. `virtualization._internal.cli` is the single source of truth for our CLI and should not contain any business logic. +To test the build command, run `dvp build --dev`. -## Committing Code -`../gradlew check` and `../gradlew sdist` must pass for code to be committed. `check` does three things at a high level: lint, format, and test. This will actually format the whitespace in both the src and test directories and organize imports. All tests will be executed and coverage will be printed. +When you run `dvp build`, the wrappers are built with the plugin. This builds `common`, `libs`, and `platform` +locally and bundles them with the plugin. If you don't pass `--dev` flag, `dvp build` will search for wrappers on PyPI. +To enable building wrappers from source instead, a special configuration entry is needed in your dvp config file which +is located at `~/.dvp/config`: -The output of `../gradlew check` command should be copied and pasted into reviews. +``` +[dev] +vsdk_root = /path/to/vsdk_repo_root +``` -## Distribution -For distribution instructions, see the README in the root of this repository. +#### Unit and functional (blackbox) testing +Refer to [../README-dev.md](https://github.com/delphix/virtualization-sdk/blob/develop/README-dev.md). ## Decisions ### CLI -We chose [Click](https://click.palletsprojects.com/en/7.x/) to build our CLI. It is simple, powerful, and dramatically reduces the amount of code we would have to write if we used argparse. Click is more limiting in ways. It is opinionated and not completely flexible. This is a limitation we're okay with. The benefits of having a feature full library out way the risks. Our CLI is also relatively simple which reduces the impact of using an opinionated library. +We chose [Click](https://click.palletsprojects.com/en/7.x/) to build our CLI. It is simple, powerful, and dramatically +reduces the amount of code we would have to write if we used argparse. Click is more limiting in ways. It is opinionated +and not completely flexible. This is a limitation we're okay with. The benefits of having a feature full library out way +the risks. Our CLI is also relatively simple which reduces the impact of using an opinionated library. -We looked at cement. This was heavier weight and only supported Python 3. We also looked at argparse, but chose Click due to the reasons above. +We looked at cement. This was heavier weight and only supported Python 3. We also looked at argparse, but chose Click +due to the reasons above. ### Testing -We chose [PyTest](https://docs.pytest.org/en/latest/) in combination with [coverage](https://pytest-cov.readthedocs.io/en/latest/). The builtin unittest is the other popular option. PyTest seems to be more popular and flexible. It was also recommended internally by people at Delphix. This is a decision we can revisit if we see that PyTest is insufficient. +We chose [PyTest](https://docs.pytest.org/en/latest/) in combination with [coverage](https://pytest-cov.readthedocs.io/en/latest/). +The builtin unittest is the other popular option. PyTest seems to be more popular and flexible. It was also recommended +internally by people at Delphix. This is a decision we can revisit if we see that PyTest is insufficient. ### Formatting -We chose [yapf](https://github.com/google/yapf) as our formatter. It's easy and does what we want right out of the box. We would like to use [black](https://github.com/ambv/black) as this is what the QA gate uses but it only supports Python 3. +We chose [yapf](https://github.com/google/yapf) as our formatter. It's easy and does what we want right out of the box. +We would like to use [black](https://github.com/ambv/black) as this is what the QA gate uses but it only supports Python 3.