From 4335e6b72fcdbc5ae940957a3290050b1967201d Mon Sep 17 00:00:00 2001 From: Filip Drozdowski Date: Mon, 9 Mar 2020 17:29:25 -0700 Subject: [PATCH 1/8] Update the README --- README-dev.md | 112 ++++++++++++++++++++------------------------ tools/README-dev.md | 74 +++++++++++++++++------------ 2 files changed, 95 insertions(+), 91 deletions(-) diff --git a/README-dev.md b/README-dev.md index f2ddab75..7906cf55 100644 --- a/README-dev.md +++ b/README-dev.md @@ -21,49 +21,49 @@ This is done automatically by `dvp build`. This is what causes the slightly different workflows in development. Changes to `tools` are completely isolated from the Delphix Engine and wrappers changes only impact the plugin build. -## Development process +### Development + +### Development process At a very high level, our development process usually looks like this: 1. Create a fork of the delphix/virtualization-sdk repository. 2. Clone the forked repository. 3. Make changes to SDK code. Test these changes manually and with unit tests. Iterate on this until you have everything working. -4. Commit your changes and build all Python package distributions. Make sure the version number of the packages is updated appropriately. -5. Publish Python distributions to artifactory. -7. Run blackbox against the newly uploaded SDK version. -7. Publish a pull request to the delphix/virtualization-sdk once your code is ready for review. -8. Once the pull request is approved, it will merged into delphix/virtualization-sdk repository. +4. Bump major/minor/patch/build version depending on the scope of the change. +5. Commit your changes. Make sure the version number of the packages is updated appropriately. +6. Push your changes to a branch in the forked repository. +7. Run blackbox tests against that branch. +8. Publish a pull request to the delphix/virtualization-sdk once your code is ready for review. +9. Once the pull request is approved, merge the pull request into delphix/virtualization-sdk repository. These steps are described in more detail below. -## Local SDK Development +### Development environment +Development should be done in a personal virtualenv. To setup the virtual environment: -To setup local development, refer to README-dev.md in the `tools` directory. This walks through the setup of a local virtualenv for development. This should be done for _all_ SDK changes. +1. `virtualenv /path/to/env/root`. This should be a Python 2.7 virtualenv. +2. `source ~/path/to/env/root/bin/activate`. -### Configure pip index +### Installing the SDK from source +Go into one of the package directories (common, dvp, libs, platform, tools) and run the following commands: +1. Install the package's development dependencies: `pip install -r requirements.txt`. +2. Install the package itself (use `-e` flag if you want to install the package in editable mode): `pip install .`. -`dvp build` executes `pip` to install the wrappers. By default `pip` looks at pypi.org for packages to install. Internal builds of the SDK are published to artifactory, not pypi. In order to configure pip to look at artifactory, create a file at `/pip.conf` that contains: +One of the SDK dependencies - dvp-api - is currently hosted on [test PyPi](https://test.pypi.org/project/dvp-api/). +By default `pip` looks at pypi.org for packages to install. In order to successfully install the SDK, you have to +configure pip to search an additional package repository - test.pypi.org. To do that, create a file at +`/pip.conf` that contains: ``` [install] -trusted-host=artifactory.delphix.com -index-url=http://artifactory.delphix.com/artifactory/api/pypi/dvp-virtual-pypi/simple/ +index-url=https://pypi.org/simple/ +extra-index-url=https://test.pypi.org/simple/ ``` ### CLI changes -To better understand how to develop and test `tools` changes, see README-dev.md in the `tools` directory. - -### Wrappers changes - -Run `dvp build --dev` to build your plugin and then upload it to a Delphix Engine to test. - -The wrappers are built with the plugin. `dvp build` has a hidden `--dev` flag. This builds `common`, `libs`, and `platform` locally and bundles them with the plugin. A special configuration entry is needed in your dvp config file which is located at `~/.dvp/config`: - -``` -[dev] -vsdk_root = /path/to/vsdk_repo_root -``` +To better understand how to develop and test `tools` changes, see [README-dev.md in the tools directory](https://github.com/delphix/virtualization-sdk/blob/develop/tools/README-dev.md). ## Versioning @@ -71,8 +71,8 @@ The SDK is shipped as five Python packages that are currently versioned and ship dvp-platform, and dvp-tools. The first thing to do is to change the version number of all the packages. Our versioning scheme follows the rules of -semantic versioning in order to help developers manage their "dependency hell". We use bump2version -(https://github.com/c4urself/bump2version) to make the version management of all five packages easier. Semantic versioning rules are the following: +semantic versioning in order to help developers manage their "dependency hell". We use [bump2version](https://github.com/c4urself/bump2version) +to make the version management of all five packages easier. Semantic versioning rules are the following: ``` Given a version number MAJOR.MINOR.PATCH, increment the: @@ -90,7 +90,7 @@ If you want to bump the build number from `1.1.0-internal-7` to `1.1.0-internal- If you want to bump the major/minor/patch version, run `bumpversion [major|minor|patch]`. -If you want to get rid of the pre-release label (bump from `1.1.1-internal-7` to `1.1.0`), run `bumpversion release`. +If you want to get rid of the pre-release label (bump from `1.1.0-internal-7` to `1.1.0`), run `bumpversion release`. ## Testing @@ -99,46 +99,34 @@ Currently, there are three types of SDK testing: unit, manual, and functional (b ### Unit Testing Go into one of the package directories (common, dvp, libs, platform, tools) and run the following commands (if you haven't done it already): + 1. Install the package's development dependencies: `pip install -r requirements.txt`. 2. Install the package itself in editable mode: `pip install -e .`. 3. Run unit tests: `python -m pytest src/main/python`. -There's no way to locally run unit tests in all packages with one command. However, they will be run automatically through GitHub Actions when you open a pull request. - -### Testing sdk-gate changes with app-gate code - -#### Manual testing +There's no way to locally run unit tests in all packages with one command. However, they will be run automatically +through GitHub Actions when you open a pull request. You can always open a draft pull request -Run `dvp build --dev` to build your plugin and then upload it to a Delphix Engine to test. +### Manual testing -The wrappers are built with the plugin. `dvp build` has a hidden `--dev` flag. This builds `common`, `libs`, and `platform` locally and bundles them with the plugin. A special configuration entry is needed in your dvp config file which is located at `~/.dvp/config`: +#### Wrappers: dvp, common, platform, libs +The only way to manually test the new wrappers code is to build a plugin, upload it to a Delphix Engine and run through +all the standard workflows. The same workflows will be exercised by functional (blackbox) tests. -``` -[dev] -vsdk_root = /path/to/vsdk_repo_root -``` - -##### Functional (blackbox) testing -(Now) Let's assume you're working on the SDK version `1.1.0-internal-10`. -To run blackbox tests, follow these steps: -1. Navigate to each package directory (common, dvp, libs, platform, tools) and run `python setup.py sdist bdist_wheel`. This will build Python package distributions. -2. Run `./bin/upload.sh` to upload Python distributions to artifactory. -2. Navigate to the app-gate directory and run -`git blackbox -s appdata_samples --extra-params="-p sdk-version=1.0.0-internal-10"`. - - -(Soon) We will be able to move to this process once blackbox runner can build Python distributions without using Gradle. +### Functional (blackbox) testing To run blackbox tests, follow these steps: -1. Push your code to a branch in the forked repository on Github. Let's say the branch is called `feature1` in repository called `username/virtualization-sdk`. -2. Navigate to the app-gate directory and run -`git blackbox -s appdata_samples --extra-params="-p virt-sdk-repo=https://github.com/username/virtualization-sdk.git -p virt-sdk-branch=feature1"`. - -#### Setup - -1. There are two environment variables that need to be set in order to publish: `ARTIFACTORY_PYPI_USER` and `ARTIFACTORY_PYPI_PASS`. - - `ARTIFACTORY_PYPI_USER` and `ARTIFACTORY_PYPI_PASS` are one set of credentials used to upload the Python distributions to our internal PyPI repositories. The credentials are the same for both internal PyPI repositories mentioned above. - - - `ARTIFACTORY_PYPI_USER` and `ARTIFACTORY_PYPI_PASS` is the username/password combo given to you by whoever setup your Artifactory pypi account. This is an account separate from your Artifactory account. If you do not have one, please reach out to the `#artifactory` channel and request a `dvp-uploaders-python` account. See for directions on how to add the account. These are used to upload the Python distributions to our internal PyPI repositories. The credentials are the same for both internal PyPI repositories mentioned above. - -2. `twine` needs to be installed. This is a Python package that is used to upload Python distributions. If it's not installed, install it by running `pip install twine`. +1. Push your code to a branch in the forked repository on Github. Let's say the branch is called `my-feature` in repository called `username/virtualization-sdk`. +2. Navigate to the app-gate directory and start tests using `git blackbox`. For the guide on which test suite to use, +see the next sections. + +#### Blackbox tests targeting wrappers (mostly Delphix Engine workflows) +* appdata_python_samples (sample plugins from the app-gate): +`git blackbox -s appdata_python_samples --extra-params="-p virt-sdk-repo=https://github.com/username/virtualization-sdk.git -p virt-sdk-branch=my-feature"`, +* appdata_sanity with a direct Python plugin on CentOS 7.3: `git blackbox -s appdata_sanity -c APPDATA_PYTHON_DIRECT_CENTOS73 -a --extra-params="-p virt-sdk-repo=https://github.com/username/virtualization-sdk.git -p virt-sdk-branch=my-feature"`, +* appdata_sanity with a staged Python plugin on CentOS 7.3: `git blackbox -s appdata_sanity -c APPDATA_PYTHON_STAGED_CENTOS73 -a --extra-params="-p virt-sdk-repo=https://github.com/username/virtualization-sdk.git -p virt-sdk-branch=my-feature"`. + +#### Blackbox tests targeting the CLI (~80% CLI tests) +* virtualization_sdk (installs and tests a direct Python plugin on Ubuntu 18): +`git blackbox -s virtualization_sdk -c APPDATA_SDK_UBUNTU18_DIRECT_CENTOS73 --extra-params="-p virt-sdk-repo=https://github.com/username/virtualization-sdk.git -p virt-sdk-branch=my-feature"`, +* virtualization_sdk (installs and tests a staged Python plugin on Ubuntu 18): +`git blackbox -s virtualization_sdk -c APPDATA_SDK_UBUNTU18_STAGED_CENTOS73 --extra-params="-p virt-sdk-repo=https://github.com/username/virtualization-sdk.git -p virt-sdk-branch=my-feature"`. \ No newline at end of file diff --git a/tools/README-dev.md b/tools/README-dev.md index 0e8e7170..cffc3f01 100644 --- a/tools/README-dev.md +++ b/tools/README-dev.md @@ -1,58 +1,74 @@ # Delphix Virtualization SDK Tools ## Purpose -These tools will be one piece of the Delphix Virtualization SDK. Plugin developers will install these tools locally and use them to develop, build, and ship virtualization plugins. - -## NOTE -This guide assumes that the current working directory is the root of the tools directory where this README is. +The `tools` package represents a CLI of the Delphix Virtualization SDK. Plugin developers will install the CLI to build +and upload virtualization plugins. ## Getting Started ### Development Environment -Development should be done in a personal virtualenv. To setup the virtual environment: - -1. `virtualenv /path/to/env/root`. This should be a Python 2.7 virtualenv. -2. `source ~/path/ot/env/root/bin/activate` -3. `pip install -r requirements.txt`. This installs the required development packages. -4. `pip install -e .` +To setup the development environment, follow the instructions in [README-dev.md](https://github.com/delphix/virtualization-sdk/blob/develop/README-dev.md) -NOTE: If build reports any issues with 'virtualenv' not found and/or errors finding virtualenv in the PATH, check and make sure virtualenv from Python 2.7 is being used and not from 3.x. Few places to check, for e.g. on Mac - /usr/local/bin/virtualenv, ~/Library/Python/2.7/virtualenv. +For quick iterations, install the `tools` package in editable mode (`pip install -e .`). This means that changes to the +code will automatically be reflected in your environment. You will not need to reinstall the tools module each time +a change is made. -You’re ready to start developing. +Changes to the code can be tested using `dvp `. `dvp` is the CLI tool that is built from the source here that +helps build and upload a plugin to DE. -The development environment will already have all development and production dependencies installed in it. The tools module will be installed in ‘editable’ mode. This means that changes to the code will automatically be reflected in your environment. You will not need to reinstall the tools module each time a change is made. +### Adding a command +The CLI uses [Click](https://click.palletsprojects.com/en/7.x/). This is a decorator driven library that builds a full +CLI. -Changes to the code can be tested using `dvp `. `dvp` is the CLI tool that is built from the source here that helps build and upload a plugin to DE. +The starting point for the CLI is `virtualization._internal.cli:delphix_sdk`. This is annotated with the `@click.group` +annotation indicating there will be subcommands. -NOTE: If you run into: `ERROR: The Python zlib extension was not compiled. Missing the zlib?` this is because you're missing a piece of XCode. Install the requirements listed [here](https://github.com/pyenv/pyenv/wiki/Common-build-problems). +All that is needed to add a command is to define a new method in `virtualization._internal.cli`. **The method's name +will be the name of the command.** Annotate the method with `@delphix_sdk.command()`. It will automatically be added +as a subcommand. Look at the [Click documentation](https://click.palletsprojects.com/en/7.x/) to see other annotations +available to configure the command. -## Adding a command -The CLI uses [Click](https://click.palletsprojects.com/en/7.x/). This is a decorator driven library builds a full CLI. +Click should be contained to `virtualization._internal.cli`. The implementation of the method should call immediately +into a method in another module. `virtualization._internal.cli` is the single source of truth for our CLI and should not +contain any business logic. -The starting point for the CLI is `virtualization._internal.cli:delphix_sdk`. This is annotated with the `@click.group` annotation indicating there will be subcommands. +### Testing -All that is needed to add a command is to define a new method in `virtualization._internal.cli`. **The method's name will be the name of the command.** Annotate the method with `@delphix_sdk.command()`. It will automatically be added as a subcommand. Look at the [Click documentation](https://click.palletsprojects.com/en/7.x/) to see other annotations available to configure the command. +#### Manual testing +It's easy to manually test the CLI as you can invoke different SDK CLI commands from command line. -Click should be contained to `virtualization._internal.cli`. The implementation of the method should call immediately into a method in another module. `virtualization._internal.cli` is the single source of truth for our CLI and should not contain any business logic. +To test the build command, run `dvp build --dev`. -## Committing Code -`../gradlew check` and `../gradlew sdist` must pass for code to be committed. `check` does three things at a high level: lint, format, and test. This will actually format the whitespace in both the src and test directories and organize imports. All tests will be executed and coverage will be printed. +When you run `dvp build`, the wrappers are built with the plugin. This builds `common`, `libs`, and `platform` +locally and bundles them with the plugin. If you don't pass `--dev` flag, `dvp build` will search for wrappers on PyPI. +To enable building wrappers from source instead, a special configuration entry is needed in your dvp config file which +is located at `~/.dvp/config`: -The output of `../gradlew check` command should be copied and pasted into reviews. +``` +[dev] +vsdk_root = /path/to/vsdk_repo_root +``` -## Distribution -For distribution instructions, see the README in the root of this repository. +#### Unit and functional (blackbox) testing +Refer to [README-dev.md](https://github.com/delphix/virtualization-sdk/blob/develop/README-dev.md). ## Decisions ### CLI -We chose [Click](https://click.palletsprojects.com/en/7.x/) to build our CLI. It is simple, powerful, and dramatically reduces the amount of code we would have to write if we used argparse. Click is more limiting in ways. It is opinionated and not completely flexible. This is a limitation we're okay with. The benefits of having a feature full library out way the risks. Our CLI is also relatively simple which reduces the impact of using an opinionated library. +We chose [Click](https://click.palletsprojects.com/en/7.x/) to build our CLI. It is simple, powerful, and dramatically +reduces the amount of code we would have to write if we used argparse. Click is more limiting in ways. It is opinionated +and not completely flexible. This is a limitation we're okay with. The benefits of having a feature full library out way +the risks. Our CLI is also relatively simple which reduces the impact of using an opinionated library. -We looked at cement. This was heavier weight and only supported Python 3. We also looked at argparse, but chose Click due to the reasons above. +We looked at cement. This was heavier weight and only supported Python 3. We also looked at argparse, but chose Click +due to the reasons above. ### Testing -We chose [PyTest](https://docs.pytest.org/en/latest/) in combination with [coverage](https://pytest-cov.readthedocs.io/en/latest/). The builtin unittest is the other popular option. PyTest seems to be more popular and flexible. It was also recommended internally by people at Delphix. This is a decision we can revisit if we see that PyTest is insufficient. +We chose [PyTest](https://docs.pytest.org/en/latest/) in combination with [coverage](https://pytest-cov.readthedocs.io/en/latest/). +The builtin unittest is the other popular option. PyTest seems to be more popular and flexible. It was also recommended +internally by people at Delphix. This is a decision we can revisit if we see that PyTest is insufficient. ### Formatting -We chose [yapf](https://github.com/google/yapf) as our formatter. It's easy and does what we want right out of the box. We would like to use [black](https://github.com/ambv/black) as this is what the QA gate uses but it only supports Python 3. +We chose [yapf](https://github.com/google/yapf) as our formatter. It's easy and does what we want right out of the box. +We would like to use [black](https://github.com/ambv/black) as this is what the QA gate uses but it only supports Python 3. From cf0bc97dbfddec7258e3ab62ce6a0f2d9e3625c1 Mon Sep 17 00:00:00 2001 From: Filip Drozdowski Date: Mon, 9 Mar 2020 17:43:45 -0700 Subject: [PATCH 2/8] Fix Development section --- README-dev.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README-dev.md b/README-dev.md index 7906cf55..a29c3528 100644 --- a/README-dev.md +++ b/README-dev.md @@ -21,7 +21,7 @@ This is done automatically by `dvp build`. This is what causes the slightly different workflows in development. Changes to `tools` are completely isolated from the Delphix Engine and wrappers changes only impact the plugin build. -### Development +## Development ### Development process From 7d6865eb3379c452ee80530a6498a9b0ffe08165 Mon Sep 17 00:00:00 2001 From: Filip Drozdowski Date: Tue, 10 Mar 2020 14:09:50 -0700 Subject: [PATCH 3/8] Fix username placeholders, development process order --- README-dev.md | 42 +++++++++++++++++++++++------------------- 1 file changed, 23 insertions(+), 19 deletions(-) diff --git a/README-dev.md b/README-dev.md index a29c3528..566547ca 100644 --- a/README-dev.md +++ b/README-dev.md @@ -46,20 +46,24 @@ Development should be done in a personal virtualenv. To setup the virtual enviro 2. `source ~/path/to/env/root/bin/activate`. ### Installing the SDK from source -Go into one of the package directories (common, dvp, libs, platform, tools) and run the following commands: -1. Install the package's development dependencies: `pip install -r requirements.txt`. -2. Install the package itself (use `-e` flag if you want to install the package in editable mode): `pip install .`. - -One of the SDK dependencies - dvp-api - is currently hosted on [test PyPi](https://test.pypi.org/project/dvp-api/). -By default `pip` looks at pypi.org for packages to install. In order to successfully install the SDK, you have to -configure pip to search an additional package repository - test.pypi.org. To do that, create a file at -`/pip.conf` that contains: +To install the SDK, follow these steps: + +1. Create a file at + `/pip.conf` that contains: + + ``` + [install] + index-url=https://pypi.org/simple/ + extra-index-url=https://test.pypi.org/simple/ + ``` + + One of the SDK dependencies - dvp-api - is currently hosted on [test PyPi](https://test.pypi.org/project/dvp-api/). + By default `pip` looks at pypi.org for packages to install. In order to successfully install the SDK, you have to + configure pip to search an additional package repository - test.pypi.org. +2. Go into one of the package directories (common, dvp, libs, platform, tools) and run the commands below. +3. Install the package's development dependencies: `pip install -r requirements.txt`. +4. Install the package itself (use `-e` flag if you want to install the package in editable mode): `pip install .`. -``` -[install] -index-url=https://pypi.org/simple/ -extra-index-url=https://test.pypi.org/simple/ -``` ### CLI changes @@ -115,18 +119,18 @@ all the standard workflows. The same workflows will be exercised by functional ( ### Functional (blackbox) testing To run blackbox tests, follow these steps: -1. Push your code to a branch in the forked repository on Github. Let's say the branch is called `my-feature` in repository called `username/virtualization-sdk`. +1. Push your code to a branch in the forked repository on Github. Let's say the branch is called `my-feature` in repository called `/virtualization-sdk`. 2. Navigate to the app-gate directory and start tests using `git blackbox`. For the guide on which test suite to use, see the next sections. #### Blackbox tests targeting wrappers (mostly Delphix Engine workflows) * appdata_python_samples (sample plugins from the app-gate): -`git blackbox -s appdata_python_samples --extra-params="-p virt-sdk-repo=https://github.com/username/virtualization-sdk.git -p virt-sdk-branch=my-feature"`, -* appdata_sanity with a direct Python plugin on CentOS 7.3: `git blackbox -s appdata_sanity -c APPDATA_PYTHON_DIRECT_CENTOS73 -a --extra-params="-p virt-sdk-repo=https://github.com/username/virtualization-sdk.git -p virt-sdk-branch=my-feature"`, -* appdata_sanity with a staged Python plugin on CentOS 7.3: `git blackbox -s appdata_sanity -c APPDATA_PYTHON_STAGED_CENTOS73 -a --extra-params="-p virt-sdk-repo=https://github.com/username/virtualization-sdk.git -p virt-sdk-branch=my-feature"`. +`git blackbox -s appdata_python_samples --extra-params="-p virt-sdk-repo=https://github.com//virtualization-sdk.git -p virt-sdk-branch=my-feature"`, +* appdata_sanity with a direct Python plugin on CentOS 7.3: `git blackbox -s appdata_sanity -c APPDATA_PYTHON_DIRECT_CENTOS73 -a --extra-params="-p virt-sdk-repo=https://github.com//virtualization-sdk.git -p virt-sdk-branch=my-feature"`, +* appdata_sanity with a staged Python plugin on CentOS 7.3: `git blackbox -s appdata_sanity -c APPDATA_PYTHON_STAGED_CENTOS73 -a --extra-params="-p virt-sdk-repo=https://github.com//virtualization-sdk.git -p virt-sdk-branch=my-feature"`. #### Blackbox tests targeting the CLI (~80% CLI tests) * virtualization_sdk (installs and tests a direct Python plugin on Ubuntu 18): -`git blackbox -s virtualization_sdk -c APPDATA_SDK_UBUNTU18_DIRECT_CENTOS73 --extra-params="-p virt-sdk-repo=https://github.com/username/virtualization-sdk.git -p virt-sdk-branch=my-feature"`, +`git blackbox -s virtualization_sdk -c APPDATA_SDK_UBUNTU18_DIRECT_CENTOS73 --extra-params="-p virt-sdk-repo=https://github.com//virtualization-sdk.git -p virt-sdk-branch=my-feature"`, * virtualization_sdk (installs and tests a staged Python plugin on Ubuntu 18): -`git blackbox -s virtualization_sdk -c APPDATA_SDK_UBUNTU18_STAGED_CENTOS73 --extra-params="-p virt-sdk-repo=https://github.com/username/virtualization-sdk.git -p virt-sdk-branch=my-feature"`. \ No newline at end of file +`git blackbox -s virtualization_sdk -c APPDATA_SDK_UBUNTU18_STAGED_CENTOS73 --extra-params="-p virt-sdk-repo=https://github.com//virtualization-sdk.git -p virt-sdk-branch=my-feature"`. \ No newline at end of file From ee08526d59ef9f9f39abd77c2596a6088210a004 Mon Sep 17 00:00:00 2001 From: Filip Drozdowski Date: Tue, 10 Mar 2020 15:26:09 -0700 Subject: [PATCH 4/8] Fix contribution guideline --- CONTRIBUTING.md | 15 ++++----------- README-dev.md | 12 +++++++----- 2 files changed, 11 insertions(+), 16 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b721d1a9..ee661748 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -40,17 +40,9 @@ Once the virtualization-sdk repository is forked, clone the forked repository in `git clone git@github.com:your-username/virtualization-sdk.git` -### Build the virtualization-sdk Repository +### Development -The virtualization-sdk repository has a Gradle task that creates a virtualenv and builds the code locally. It can be triggered from the root directory of the source tree using the command: - -`./gradlew build` - -### Run Unit Tests - -Unit tests for the virtualization-sdk repository can be triggered from the root directory of the source tree using Gradle with the command: - -`./gradlew test` +For development instructions, refer to [README-dev.md](https://github.com/delphix/virtualization-sdk/blob/develop/README-dev.md). ## Asking for Help Please raise a GitHub issue to ask for help with appropriate GitHub tag . @@ -74,7 +66,7 @@ Some guidelines for Pull Requests: Once the pull request has required approvals from code owners of the repository, the code owner will merge the pull request into the actual virtualization-sdk repository. ### Code Owners -Code owners defined by the codeowners file in the repository are the gatekeepers of the repository. For a pull request to be merged, it requires approval from at least 2 codeowners. +Code owners defined by the codeowners file in the repository are the gatekeepers of the repository. For a pull request to be merged, it requires approval from at least one codeowner. ## Testing and CI/CD CI/CD for this repository is managed through GitHub Actions. All the checks need to succeed for the pull request to be merged. @@ -90,5 +82,6 @@ Commit messages for new changes must meet the following guidelines: `Fixes #123 Format of error is incorrect` `Optional Description of the issue` +* If the commit doesn't address a specific issue, it should include a description of changes. diff --git a/README-dev.md b/README-dev.md index 566547ca..9808c449 100644 --- a/README-dev.md +++ b/README-dev.md @@ -31,11 +31,13 @@ At a very high level, our development process usually looks like this: 2. Clone the forked repository. 3. Make changes to SDK code. Test these changes manually and with unit tests. Iterate on this until you have everything working. 4. Bump major/minor/patch/build version depending on the scope of the change. -5. Commit your changes. Make sure the version number of the packages is updated appropriately. -6. Push your changes to a branch in the forked repository. -7. Run blackbox tests against that branch. -8. Publish a pull request to the delphix/virtualization-sdk once your code is ready for review. -9. Once the pull request is approved, merge the pull request into delphix/virtualization-sdk repository. +5. Commit your changes. Refer to the [contribution guideline](https://github.com/delphix/virtualization-sdk/blob/develop/CONTRIBUTING.md#commit-message-format) + for commit message format. +6. Make sure the version number of the packages is updated appropriately. +7. Push your changes to a branch in the forked repository. +8. Run blackbox tests against that branch. +9. Publish a pull request to the delphix/virtualization-sdk once your code is ready for review. +10. Once the pull request is approved, merge the pull request into delphix/virtualization-sdk repository. These steps are described in more detail below. From 11a6ef7aa58eadf35b7baa3caad0309e635bd99a Mon Sep 17 00:00:00 2001 From: Filip Drozdowski Date: Thu, 12 Mar 2020 11:09:46 -0700 Subject: [PATCH 5/8] Fix unit testing section --- README-dev.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/README-dev.md b/README-dev.md index 9808c449..ce0f4e9f 100644 --- a/README-dev.md +++ b/README-dev.md @@ -47,7 +47,7 @@ Development should be done in a personal virtualenv. To setup the virtual enviro 1. `virtualenv /path/to/env/root`. This should be a Python 2.7 virtualenv. 2. `source ~/path/to/env/root/bin/activate`. -### Installing the SDK from source +###Installing the SDK from source To install the SDK, follow these steps: 1. Create a file at @@ -104,11 +104,10 @@ Currently, there are three types of SDK testing: unit, manual, and functional (b ### Unit Testing -Go into one of the package directories (common, dvp, libs, platform, tools) and run the following commands (if you haven't done it already): +Go into one of the package directories (common, dvp, libs, platform, tools) and follow these steps: -1. Install the package's development dependencies: `pip install -r requirements.txt`. -2. Install the package itself in editable mode: `pip install -e .`. -3. Run unit tests: `python -m pytest src/main/python`. +1. Install the package's development dependencies and package itself by following the directions in [the SDK installation section](#installing-the-sdk-from-source). +2. Run unit tests: `python -m pytest src/test/python`. There's no way to locally run unit tests in all packages with one command. However, they will be run automatically through GitHub Actions when you open a pull request. You can always open a draft pull request From 896525fced424bbbcb2c5bc99e652093c10dd2be Mon Sep 17 00:00:00 2001 From: Filip Drozdowski Date: Thu, 12 Mar 2020 11:19:06 -0700 Subject: [PATCH 6/8] Fix heading --- README-dev.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README-dev.md b/README-dev.md index ce0f4e9f..0744666e 100644 --- a/README-dev.md +++ b/README-dev.md @@ -47,7 +47,7 @@ Development should be done in a personal virtualenv. To setup the virtual enviro 1. `virtualenv /path/to/env/root`. This should be a Python 2.7 virtualenv. 2. `source ~/path/to/env/root/bin/activate`. -###Installing the SDK from source +### Installing the SDK from source To install the SDK, follow these steps: 1. Create a file at From 8167ee75efe5fa4d6daf6b4ffd3457e4147b8087 Mon Sep 17 00:00:00 2001 From: Filip Drozdowski Date: Thu, 12 Mar 2020 14:21:06 -0700 Subject: [PATCH 7/8] Reminder to change version string in test_package_util.py --- README-dev.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/README-dev.md b/README-dev.md index 0744666e..81bdba77 100644 --- a/README-dev.md +++ b/README-dev.md @@ -29,13 +29,14 @@ At a very high level, our development process usually looks like this: 1. Create a fork of the delphix/virtualization-sdk repository. 2. Clone the forked repository. -3. Make changes to SDK code. Test these changes manually and with unit tests. Iterate on this until you have everything working. -4. Bump major/minor/patch/build version depending on the scope of the change. +3. Make changes to SDK code. Test these changes manually and with [unit tests](#unit-testing). Iterate on this until you have everything working. +4. Bump major/minor/patch/build version depending on the scope of the change. Refer to [versioning section](#versioning) for instructions. 5. Commit your changes. Refer to the [contribution guideline](https://github.com/delphix/virtualization-sdk/blob/develop/CONTRIBUTING.md#commit-message-format) for commit message format. -6. Make sure the version number of the packages is updated appropriately. +6. Make sure the version number of the packages is updated appropriately in your commit. That includes all files mentioned +in `.bumpversion.cfg` and the string in `test_get_version()` inside `test_package_util.py`. 7. Push your changes to a branch in the forked repository. -8. Run blackbox tests against that branch. +8. Run [blackbox tests](#functional-blackbox-testing) against that branch. 9. Publish a pull request to the delphix/virtualization-sdk once your code is ready for review. 10. Once the pull request is approved, merge the pull request into delphix/virtualization-sdk repository. From 6706e03681e9a61bbf7ca7ac04daa8fa7f0a5f8a Mon Sep 17 00:00:00 2001 From: Filip Drozdowski Date: Fri, 13 Mar 2020 14:43:39 -0700 Subject: [PATCH 8/8] Address Ankur's feedback --- README-dev.md | 7 +++++-- tools/README-dev.md | 2 +- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/README-dev.md b/README-dev.md index 81bdba77..d3593628 100644 --- a/README-dev.md +++ b/README-dev.md @@ -60,7 +60,7 @@ To install the SDK, follow these steps: extra-index-url=https://test.pypi.org/simple/ ``` - One of the SDK dependencies - dvp-api - is currently hosted on [test PyPi](https://test.pypi.org/project/dvp-api/). + One of the SDK dependencies - dvp-api - is currently hosted on [TestPyPi](https://test.pypi.org/project/dvp-api/). By default `pip` looks at pypi.org for packages to install. In order to successfully install the SDK, you have to configure pip to search an additional package repository - test.pypi.org. 2. Go into one of the package directories (common, dvp, libs, platform, tools) and run the commands below. @@ -70,7 +70,7 @@ To install the SDK, follow these steps: ### CLI changes -To better understand how to develop and test `tools` changes, see [README-dev.md in the tools directory](https://github.com/delphix/virtualization-sdk/blob/develop/tools/README-dev.md). +To better understand how to develop and test `tools` changes, see [tools/README-dev.md](https://github.com/delphix/virtualization-sdk/blob/develop/tools/README-dev.md). ## Versioning @@ -125,6 +125,9 @@ To run blackbox tests, follow these steps: 2. Navigate to the app-gate directory and start tests using `git blackbox`. For the guide on which test suite to use, see the next sections. +At a minimum, each pull request should pass `appdata_python_samples` and `appdata_sanity` tests with a direct or staged plugin. +See the section below for the description of each test suite. + #### Blackbox tests targeting wrappers (mostly Delphix Engine workflows) * appdata_python_samples (sample plugins from the app-gate): `git blackbox -s appdata_python_samples --extra-params="-p virt-sdk-repo=https://github.com//virtualization-sdk.git -p virt-sdk-branch=my-feature"`, diff --git a/tools/README-dev.md b/tools/README-dev.md index cffc3f01..740a1e1d 100644 --- a/tools/README-dev.md +++ b/tools/README-dev.md @@ -50,7 +50,7 @@ vsdk_root = /path/to/vsdk_repo_root ``` #### Unit and functional (blackbox) testing -Refer to [README-dev.md](https://github.com/delphix/virtualization-sdk/blob/develop/README-dev.md). +Refer to [../README-dev.md](https://github.com/delphix/virtualization-sdk/blob/develop/README-dev.md). ## Decisions