Release plan for 2018.4.0

[brettcannon]

Helpful links & info:

• Milestone

Schedule

Monday, Apr 02

• Review the state of the current milestone
• Go through telemetry for GDPR
• Go through all merged pull requests and add the validate fix label as appropriate as well as checking for news entries

Planning

• Evaluate if TypeScript usage needs updating to sync with VS Code's usage
• Evaluate projects & meta issues
• Go through needs PR issues to see if there's anything we want to add to this milestone
• Finalize the initial set of issues for the milestone
• Make sure all issues for this milestone are assigned
• Close issues that have needed more info for over a month

Monday, Apr 09

• Review the state of the current milestone
• Go through telemetry for GDPR
• Go through all merged pull requests and add the validate fix label as appropriate as well as checking for news entries

Planning

• Check if there have been no performance regressions
• Read through VS Code's iteration plan (it may still be a draft)

Monday, Apr 16

• Review the state of the current milestone
• Go through telemetry for GDPR
• Go through all merged pull requests and add the validate fix label as appropriate as well as checking for news entries

Monday, Apr 23

• Review the state of the current milestone
• Go through telemetry for GDPR
• Go through all merged pull requests and add the validate fix label as appropriate as well as checking for news entries

Legal

• Announce the lock-down of dependencies for this release
• Notify CELA of all changes to the repository and distribution dependencies

Release a beta version for testing

• Update the version to be a beta & update the changelog
• Announce the beta development build (along with how to help validate fixes)
• Open appropriate documentation issues

Monday, Apr 30

• Review the state of the current milestone
• Go through telemetry for GDPR
• Merge any last-minute pull requests
• Go through all merged pull requests and add the validate fix label as appropriate as well as checking for news entries

Prep for the release candidate

• Announce feature freeze
• Make sure the repo and distribution TPNs have been updated appropriately

Test the release candidate code

• Update the version to be an rc & update the changelog
• Announce the release candidate development build
• Open appropriate documentation issues
• Begin drafting a blog post

Prep the release

• Ensure all new feature usages are tracked via telemetry
• Make sure no extraneous files are being included in the .vsix file (make sure to check for hidden files)
• Make sure the appropriate pull requests for the documentation -- including the WOW page -- are ready

Wednesday, May 02 (hopefully 😉)

Release

• Update the changelog (including the names of external contributors & projects)
• Update the version number to be final
• Make sure CI is passing
• Create the release- branch
• Generate final .vsix file from the release- branch
• Upload the final .vsix file to the marketplace
• Publish documentation changes
• Publish the blog post
• Create a release on GitHub (which creates an appropriate git tag)

Prep for the next release

• Bump the version number to the next alpha
• Make sure the next two milestones exist
• Lift the feature freeze
• Create a new release plan

Clean up after this release

• Clean up any straggling fixed issues needing validation
• Close the (now) old milestone
• Delete the previous releases' branch

[brettcannon]

How to participate

Please test against the latest development build of the extension.

Paste the release plan template into a new comment on this issue, fill in your system's details, go through the test plan, and then have the appropriate testing targets above checked off for your system.

Once you are finished, let the testing lead know you are completed so they can move your test setup from the "in progress" section to "finished" (an @ mention is fine).

Testing targets

OS

• Windows 10
• macOS
• Linux

Python

Distribution

• CPython
• Anaconda (miniconda is acceptable)

Version

• 2.7
• 3.6
• 3.7 beta

VS Code

• Stable
• Insiders

Tested

Finished

1. Windows 10, CPython 3.7.0b3, Insiders
2. macOS Sierra, CPython 3.6, Stable
3. Ubuntu 16.04, miniconda 3.6.4, Stable

In progress

1. Ubuntu 16.04, CPython 3.5, Stable
2. Windows 10, CPython 2.7, Stable

[DonJayamanne]

I'm assuming we'd assign one section (e.g. Linting) to one person.

[brettcannon]

Test plan

Environment

• OS: Windows 10
• Python
  • Distribution: CPython
  • Version: 3.7.0b3

Tests

Scenarios

ALWAYS check the Output window under Python for logged errors!

Environment

Interpreters

• Interpreter is shown in the status bar
• An interpreter can be manually specified using the Select Interpreter command
• Detected system-installed interpreters
• Detected an Anaconda installation
• "python.pythonPath" triggers an update in the status bar
• Run Python File in Terminal
• Run Selection/Line in Python Terminal

Virtual environments

ALWAYS create environments with a space in their name.*

• Detected a single virtual environment at the top-level of the workspace folder
  • Appropriate suffix label specified in status bar
  • "python.terminal.activateEnvironments": false deactivates detection
  • Create Terminal works
• Detect multiple virtual environments in a directory specified by "python.venvPath"
• Detected all conda environments created with an interpreter
  • Appropriate suffix label specified in status bar
  • Prompt to install Pylint installs into the conda environment
  • "python.terminal.activateEnvironments": false deactivates detection
  • Create Terminal works

Environment files

Sample files:

# example.py
import os
print('Hello,', os.environ.get('WHO'), '!')

# .env
WHO=world

• Environment variables in a .env file are exposed when running under the debugger
• "python.envFile" allows for specifying an environment file manually

Debugging

• pythonPath setting in your launch.json overrides your python.pythonPath default setting

Linting

ALWAYS check under the Problems tab to see e.g. if a linter is raising errors!

Pylint/default linting

[Prompting to install Pylint is covered under Environments above]

For testing the disablement of the default linting rules for Pylint:

# pylintrc
[MESSAGES CONTROL]
enable=bad-names

# example.py
foo = 42  # Marked as a blacklisted name.

• Installation via the prompt installs Pylint as appropriate

Other linters

• flake8 works
• pydocstyle works
• pep8 works
• 3 or more linters work simultaneously
  • Run Linting runs all linters
  • The Select Linter command lists all the above linters and prompts to install a linter when missing
  • "python.linting.enabled" disables all linters
  • The Enable Linting command changes "python.linting.enabled"
  • "python.linting.lintOnSave works

Editing

IntelliSense

Please also test for general accuracy on the most "interesting" code you can find.

• "python.autoComplete.extraPaths" works
• "python.autoComplete.preloadModules" works
• "python.autocomplete.addBrackets": true causes auto-completion of functions to append ()

Formatting

• autopep8 works
• yapf works
• "editor.formatOnType": true works and has expected results

Refactoring

• Extract Variable works
• Extract method works
• Sort Imports works

Debugging

Test both old and new debugger (and notice if the new debugger seems at least as fast as the old debugger).

• Configurations work
  • Current File
  • Module
  • Attach
  • Terminal (integrated)
  • Terminal (external)
  • Django
  • Flask
  • Pyramid
  • Watson
  • Scrapy
  • PySpark
  • All debug Options with appropriate values changed
• Breakpoints
  • Set
  • Hit
  • Watch
• Stepping
  • Over
  • Into
  • Out
• Can inspect variables
  • Through hovering over variable in code
  • Variables section of debugger sidebar
• Remote debugging works
  • ... over SSH
• App Engine

Unit testing

unittest

import unittest

class PassingTests(unittest.TestCase):

    def test_passing(self):
        self.assertEqual(42, 42)

    def test_passing_still(self):
        self.assertEqual("silly walk", "silly walk")


class FailingTests(unittest.TestCase):

    def test_failure(self):
        self.assertEqual(42, -13)

    def test_failure_still(self):
        self.assertEqual("I'm right!", "no, I am!")

• Run All Unit Tests triggers the prompt to configure the test runner
• Tests are discovered (as shown by code lenses on each test)

pytest

def test_passing():
    assert 42 == 42

def test_failure():
    assert 42 == -13

• Run All Unit Tests triggers the prompt to configure the test runner
  • Pytest gets installed
• Tests are discovered (as shown by code lenses on each test)

nose

def test_passing():
    assert 42 == 42

def test_failure():
    assert 42 == -13

• Run All Unit Tests triggers the prompt to configure the test runner
  • Nose gets installed
• Tests are discovered (as shown by code lenses on each test)

General

• Code lenses appears
  • Run Test lens works (and status bar updates as appropriate)
  • Debug Test lens works
  • Appropriate ✔/❌ shown for each test
• Status bar is functioning
  • Appropriate test results displayed
  • Run All Unit Tests works
  • Discover Unit Tests works (resets tests result display in status bar)
  • Run Unit Test Method ... works
  • View Unit Test Output works
  • After having at least one failure, Run Failed Tests works

[DonJayamanne]

Test plan

Environment

• OS: Mac
• Python
  • Distribution: CPython
  • Version: 3.6.5 (cpython)

Tests

ALWAYS check the Output window under Python for logged errors!

Environment

Interpreters

Scenarios

• Interpreter is shown in the status bar
• An interpreter can be manually specified using the Select Interpreter command
• Detected system-installed interpreters
• Detected an Anaconda installation
• (Linux/macOS) Detected all interpreters installed w/ pyenv detected
• "python.pythonPath" triggers an update in the status bar
• Run Python File in Terminal
• Run Selection/Line in Python Terminal

Virtual environments

ALWAYS create environments with a space in their name.*

• Detected a single virtual environment at the top-level of the workspace folder
  • Appropriate suffix label specified in status bar
  • Prompt to install Pylint uses --user
  • "python.terminal.activateEnvironments": false deactivates detection
  • Create Terminal works
• Detect multiple virtual environments in a directory specified by "python.venvPath"
• Detected all conda environments created with an interpreter
  • Appropriate suffix label specified in status bar
  • Prompt to install Pylint installs into the conda environment
  • "python.terminal.activateEnvironments": false deactivates detection
  • Create Terminal works
• (Linux/macOS until -m is supported) Detected the virtual environment created by pipenv
  • Appropriate suffix label specified in status bar
  • Prompt to install Pylint uses pipenv install --dev
  • "python.terminal.activateEnvironments": false deactivates detection
  • Create Terminal works
• (Linux/macOS) Detected virtual environments created under {workspaceFolder}/.direnv/python-{python_version} for direnv and its layout python3 support
  • Appropriate suffix label specified in status bar
  • "python.terminal.activateEnvironments": false deactivates detection

Environment files

Sample files:

# example.py
import os
print('Hello,', os.environ.get('WHO'), '!')

# .env
WHO=world

• Environment variables in a .env file are exposed when running under the debugger
• "python.envFile" allows for specifying an environment file manually

Debugging

• pythonPath setting in your launch.json overrides your python.pythonPath default setting

Linting

ALWAYS check under the Problems tab to see e.g. if a linter is raising errors!

Pylint/default linting

[Prompting to install Pylint is covered under Environments above]

For testing the disablement of the default linting rules for Pylint:

# pylintrc
[MESSAGES CONTROL]
enable=bad-names

# example.py
foo = 42  # Marked as a blacklisted name.

• Installation via the prompt installs Pylint as appropriate
• Pylint works
• "python.linting.pylintUseMinimalCheckers": false turns off the default rules w/ no pylintrc file present
• The existense of a pylintrc file turns off the default rules

Other linters

• flake8 works
• mypy works
• pydocstyle works
• pep8 works
• prospector works
• pylama works
• 3 or more linters work simultaneously
  • Run Linting runs all linters
  • The Select Linter command lists all the above linters and prompts to install a linter when missing
  • "python.linting.enabled" disables all linters
  • The Enable Linting command changes "python.linting.enabled"
  • "python.linting.lintOnSave works

Editing

IntelliSense

Please also test for general accuracy on the most "interesting" code you can find.

• "python.autoComplete.extraPaths" works
• "python.autoComplete.preloadModules" works
• "python.autocomplete.addBrackets": true causes auto-completion of functions to append ()

Formatting

• autopep8 works
• yapf works
• "editor.formatOnType": true works and has expected results

Refactoring

• Extract Variable works
• Extract method works
• Sort Imports works

Debugging

Test both old and new debugger (and notice if the new debugger seems at least as fast as the old debugger).

• Configurations work
  • Current File
  • Module
  • Attach
  • Terminal (integrated)
  • Terminal (external)
  • Django
  • Flask
  • Pyramid
  • Watson
  • Scrapy
  • PySpark
  • All debug Options with appropriate values changed
• Breakpoints
  • Set
  • Hit
  • Watch
• Stepping
  • Over
  • Into
  • Out
• Can inspect variables
  • Through hovering over variable in code
  • Variables section of debugger sidebar
• Remote debugging works
  • ... over SSH
• App Engine

Unit testing

unittest

import unittest

class PassingTests(unittest.TestCase):

    def test_passing(self):
        self.assertEqual(42, 42)

    def test_passing_still(self):
        self.assertEqual("silly walk", "silly walk")


class FailingTests(unittest.TestCase):

    def test_failure(self):
        self.assertEqual(42, -13)

    def test_failure_still(self):
        self.assertEqual("I'm right!", "no, I am!")

• Run All Unit Tests triggers the prompt to configure the test runner
• Tests are discovered (as shown by code lenses on each test)

pytest

def test_passing():
    assert 42 == 42

def test_failure():
    assert 42 == -13

• Run All Unit Tests triggers the prompt to configure the test runner
  • Pytest gets installed
• Tests are discovered (as shown by code lenses on each test)

nose

def test_passing():
    assert 42 == 42

def test_failure():
    assert 42 == -13

• Run All Unit Tests triggers the prompt to configure the test runner
  • Nose gets installed
• Tests are discovered (as shown by code lenses on each test)

General

• Code lenses appears
  • Run Test lens works (and status bar updates as appropriate)
  • Debug Test lens works
  • Appropriate ✔/❌ shown for each test
• Status bar is functioning
  • Appropriate test results displayed
  • Run All Unit Tests works
  • Discover Unit Tests works (resets tests result display in status bar)
  • Run Unit Test Method ... works
  • View Unit Test Output works
  • After having at least one failure, Run Failed Tests works

[DonJayamanne]

Test plan

Environment

• OS: Linux
• Python
  • Distribution: Miniconda
  • Version: 3.6.4

Tests

ALWAYS check the Output window under Python for logged errors!

Scenarios

Environment

Interpreters

• Interpreter is shown in the status bar
• An interpreter can be manually specified using the Select Interpreter command
• Detected system-installed interpreters
• Detected an Anaconda installation
• (Linux/macOS) Detected all interpreters installed w/ pyenv detected
• "python.pythonPath" triggers an update in the status bar
• Run Python File in Terminal
• Run Selection/Line in Python Terminal
  • Right-click
  • Command
  • Ctrl-Enter

Virtual environments

ALWAYS create environments with a space in their name.*

• Detected a single virtual environment at the top-level of the workspace folder
  • Appropriate suffix label specified in status bar
  • Create Terminal works
    • Steals focus
    • "python.terminal.activateEnvironment": false turns off automatic activation of the environment
• Detect multiple virtual environments in a directory specified by "python.venvPath"
• Detected all conda environments created with an interpreter
  • Appropriate suffix label specified in status bar
  • Prompted to install Pylint
    • Asked whether to install using conda or pip
    • Installs into environment
  • "python.terminal.activateEnvironments": false deactivates detection
  • Create Terminal works
• (Linux/macOS until -m is supported) Detected the virtual environment created by pipenv
  • Appropriate suffix label specified in status bar
  • Prompt to install Pylint uses pipenv install --dev
  • "python.terminal.activateEnvironments": false deactivates detection
  • Create Terminal works
• (Linux/macOS) Virtual environments created under {workspaceFolder}/.direnv/python-{python_version} are detected (for direnv and its layout python3 support)
  • Appropriate suffix label specified in status bar
  • "python.terminal.activateEnvironments": false deactivates detection

Environment files

Sample files:

# example.py
import os
print('Hello,', os.environ.get('WHO'), '!')

# .env
WHO=world
PYTHONPATH=some/path/somewhere

Make sure to use Reload Window between tests to reset your environment!

• Environment variables in a .env file are exposed when running under the debugger
• "python.envFile" allows for specifying an environment file manually (e.g. Jedi picks up PYTHONPATH changes)
• envFile in a launch.json configuration works

Debugging

• pythonPath setting in your launch.json overrides your python.pythonPath default setting

Linting

ALWAYS check under the Problems tab to see e.g. if a linter is raising errors!

Pylint/default linting

[Prompting to install Pylint is covered under Environments above]

For testing the disablement of the default linting rules for Pylint:

# pylintrc
[MESSAGES CONTROL]
enable=bad-names

# example.py
foo = 42  # Marked as a blacklisted name.

• Installation via the prompt installs Pylint as appropriate
  • Uses --user for system-install of Python
  • Installs into a virtual environment environment directly
• Pylint works
• "python.linting.pylintUseMinimalCheckers": false turns off the default rules w/ no pylintrc file present
• The existence of a pylintrc file turns off the default rules

Other linters

You can always use the Run Linting command to immediately trigger a newly installed linter.

• flake8 works
• mypy works
• pep8 works
• prospector works
• pydocstyle works
• pylama works
• 3 or more linters work simultaneously
  • Run Linting runs all linters
  • The Select Linter command lists all the above linters and prompts to install a linter when missing
  • "python.linting.enabled": false disables all linters
  • The Enable Linting command changes "python.linting.enabled"
  • "python.linting.lintOnSave works

Editing

IntelliSense

Please also test for general accuracy on the most "interesting" code you can find.

• "python.autoComplete.extraPaths" works
• "python.autoComplete.preloadModules" works
• "python.autocomplete.addBrackets": true causes auto-completion of functions to append ()

Formatting

Sample file:

# There should be _some_ change after running `Format Document`.
def foo():pass

• Prompted to install a formatter if none installed and Format Document is run
  • Installing autopep8 works
  • Installing yapf works
• autopep8 works
• yapf works
• "editor.formatOnType": true works and has expected results

Refactoring

• Extract Variable works
  • You are prompted to install rope if it is not already available
• Extract method works
  • You are prompted to install rope if it is not already available
• Sort Imports works

Debugging

Test both old and new debugger (and notice if the new debugger seems at least as fast as the old debugger).

• Configurations work
  • Current File
  • Module
  • Attach
  • Terminal (integrated)
  • Terminal (external)
  • Django
  • Flask
  • Pyramid
  • Watson
  • Scrapy
  • PySpark
  • All debug Options with appropriate values edited to make values valid
• Breakpoints
  • Set
  • Hit
  • Conditional
    • Expression
    • Hit count
  • Log points (experimental debugger only)
• Stepping
  • Over
  • Into
  • Out
• Can inspect variables
  • Through hovering over variable in code
  • Variables section of debugger sidebar
• Remote debugging works
  • ... over SSH
• App Engine

Unit testing

unittest

import unittest

class PassingTests(unittest.TestCase):

    def test_passing(self):
        self.assertEqual(42, 42)

    def test_passing_still(self):
        self.assertEqual("silly walk", "silly walk")


class FailingTests(unittest.TestCase):

    def test_failure(self):
        self.assertEqual(42, -13)

    def test_failure_still(self):
        self.assertEqual("I'm right!", "no, I am!")

• Run All Unit Tests triggers the prompt to configure the test runner
• Tests are discovered (as shown by code lenses on each test)

pytest

def test_passing():
    assert 42 == 42

def test_failure():
    assert 42 == -13

• Run All Unit Tests triggers the prompt to configure the test runner
  • pytest gets installed
• Tests are discovered (as shown by code lenses on each test)

nose

def test_passing():
    assert 42 == 42

def test_failure():
    assert 42 == -13

• Run All Unit Tests triggers the prompt to configure the test runner
  • Nose gets installed
• Tests are discovered (as shown by code lenses on each test)

General

• Code lenses appears
  • Run Test lens works (and status bar updates as appropriate)
  • Debug Test lens works
  • Appropriate ✔/❌ shown for each test
• Status bar is functioning
  • Appropriate test results displayed
  • Run All Unit Tests works
  • Discover Unit Tests works (resets tests result display in status bar)
  • Run Unit Test Method ... works
  • View Unit Test Output works
  • After having at least one failure, Run Failed Tests works

[ericsnowcurrently]

Test plan

Environment

• OS: Linux (Ubuntu 16.04)
• Python
  • Distribution: CPython
  • Version: 3.5

Tests

Tests

ALWAYS check the Output window under Python for logged errors!

Environment

Interpreters

• Interpreter is shown in the status bar
• An interpreter can be manually specified using the Select Interpreter command
• Detected system-installed interpreters
• Detected an Anaconda installation
• (Linux/macOS) Detected all interpreters installed w/ pyenv detected
• "python.pythonPath" triggers an update in the status bar
• Run Python File in Terminal
• Run Selection/Line in Python Terminal
  • Right-click
  • Command
  • Ctrl-Enter

Virtual environments

ALWAYS create environments with a space in their name.*

• Detected a single virtual environment at the top-level of the workspace folder
  • Appropriate suffix label specified in status bar
  • Create Terminal works
    • Steals focus
    • "python.terminal.activateEnvironment": false turns off automatic activation of the environment
• Detect multiple virtual environments in a directory specified by "python.venvPath"
• Detected all conda environments created with an interpreter
  • Appropriate suffix label specified in status bar
  • Prompted to install Pylint
    • Asked whether to install using conda or pip
    • Installs into environment
  • "python.terminal.activateEnvironments": false deactivates detection
  • Create Terminal works
• (Linux/macOS until -m is supported) Detected the virtual environment created by pipenv
  • Appropriate suffix label specified in status bar
  • Prompt to install Pylint uses pipenv install --dev
  • "python.terminal.activateEnvironments": false deactivates detection
  • Create Terminal works
• (Linux/macOS) Virtual environments created under {workspaceFolder}/.direnv/python-{python_version} are detected (for direnv and its layout python3 support)
  • Appropriate suffix label specified in status bar
  • "python.terminal.activateEnvironments": false deactivates detection

Environment files

Sample files:

# example.py
import os
print('Hello,', os.environ.get('WHO'), '!')

# .env
WHO=world
PYTHONPATH=some/path/somewhere

Make sure to use Reload Window between tests to reset your environment!

• Environment variables in a .env file are exposed when running under the debugger
• "python.envFile" allows for specifying an environment file manually (e.g. Jedi picks up PYTHONPATH changes)
• envFile in a launch.json configuration works

Debugging

• pythonPath setting in your launch.json overrides your python.pythonPath default setting

Linting

ALWAYS check under the Problems tab to see e.g. if a linter is raising errors!

Pylint/default linting

[Prompting to install Pylint is covered under Environments above]

For testing the disablement of the default linting rules for Pylint:

# pylintrc
[MESSAGES CONTROL]
enable=bad-names

# example.py
foo = 42  # Marked as a blacklisted name.

• Installation via the prompt installs Pylint as appropriate
  • Uses --user for system-install of Python
  • Installs into a virtual environment environment directly
• Pylint works
• "python.linting.pylintUseMinimalCheckers": false turns off the default rules w/ no pylintrc file present
• The existence of a pylintrc file turns off the default rules

Other linters

You can always use the Run Linting command to immediately trigger a newly installed linter.

• flake8 works
• mypy works
• pep8 works
• prospector works
• pydocstyle works
• pylama works
• 3 or more linters work simultaneously
  • Run Linting runs all linters
  • The Select Linter command lists all the above linters and prompts to install a linter when missing
  • "python.linting.enabled": false disables all linters
  • The Enable Linting command changes "python.linting.enabled"
  • "python.linting.lintOnSave works

Editing

IntelliSense

Please also test for general accuracy on the most "interesting" code you can find.

• "python.autoComplete.extraPaths" works
• "python.autoComplete.preloadModules" works
• "python.autocomplete.addBrackets": true causes auto-completion of functions to append ()

Formatting

Sample file:

# There should be _some_ change after running `Format Document`.
def foo():pass

• Prompted to install a formatter if none installed and Format Document is run
  • Installing autopep8 works
  • Installing yapf works
• autopep8 works
• yapf works
• "editor.formatOnType": true works and has expected results

Refactoring

• Extract Variable works
  • You are prompted to install rope if it is not already available
• Extract method works
  • You are prompted to install rope if it is not already available
• Sort Imports works

Debugging

Test both old and new debugger (and notice if the new debugger seems at least as fast as the old debugger).

• Configurations work
  • Current File
  • Module
  • Attach
  • Terminal (integrated)
  • Terminal (external)
  • Django
  • Flask
  • Pyramid
  • Watson
  • Scrapy
  • PySpark
  • All debug Options with appropriate values edited to make values valid
• Breakpoints
  • Set
  • Hit
  • Conditional
    • Expression
    • Hit count
  • Log points (experimental debugger only)
• Stepping
  • Over
  • Into
  • Out
• Can inspect variables
  • Through hovering over variable in code
  • Variables section of debugger sidebar
• Remote debugging works
  • ... over SSH
• App Engine

Unit testing

unittest

import unittest

class PassingTests(unittest.TestCase):

    def test_passing(self):
        self.assertEqual(42, 42)

    def test_passing_still(self):
        self.assertEqual("silly walk", "silly walk")


class FailingTests(unittest.TestCase):

    def test_failure(self):
        self.assertEqual(42, -13)

    def test_failure_still(self):
        self.assertEqual("I'm right!", "no, I am!")

• Run All Unit Tests triggers the prompt to configure the test runner
• Tests are discovered (as shown by code lenses on each test)

pytest

def test_passing():
    assert 42 == 42

def test_failure():
    assert 42 == -13

• Run All Unit Tests triggers the prompt to configure the test runner
  • pytest gets installed
• Tests are discovered (as shown by code lenses on each test)

nose

def test_passing():
    assert 42 == 42

def test_failure():
    assert 42 == -13

• Run All Unit Tests triggers the prompt to configure the test runner
  • Nose gets installed
• Tests are discovered (as shown by code lenses on each test)

General

• Code lenses appears
  • Run Test lens works (and status bar updates as appropriate)
  • Debug Test lens works
  • Appropriate ✔/❌ shown for each test
• Status bar is functioning
  • Appropriate test results displayed
  • Run All Unit Tests works
  • Discover Unit Tests works (resets tests result display in status bar)
  • Run Unit Test Method ... works
  • View Unit Test Output works
  • After having at least one failure, Run Failed Tests works

[int19h]

Test plan

Environment

• OS: Win10 16299
• Python
  • Distribution: CPython
  • Version: 2.7-64

Tests

ALWAYS check the Output window under Python for logged errors!

Environment

Interpreters

• Interpreter is shown in the status bar
• An interpreter can be manually specified using the Select Interpreter command
• Detected system-installed interpreters
• Detected an Anaconda installation
• (Linux/macOS) Detected all interpreters installed w/ pyenv detected
• "python.pythonPath" triggers an update in the status bar
• Run Python File in Terminal
• Run Selection/Line in Python Terminal
  • Right-click
  • Command
  • Ctrl-Enter

Virtual environments

ALWAYS create environments with a space in their name.*

• Detected a single virtual environment at the top-level of the workspace folder
  • Appropriate suffix label specified in status bar
  • Create Terminal works
    • Steals focus
    • "python.terminal.activateEnvironment": false turns off automatic activation of the environment
• Detect multiple virtual environments in a directory specified by "python.venvPath"
• Detected all conda environments created with an interpreter
  • Appropriate suffix label specified in status bar
  • Prompted to install Pylint
    • Asked whether to install using conda or pip
    • Installs into environment
  • "python.terminal.activateEnvironments": false deactivates detection
  • Create Terminal works
• (Linux/macOS until -m is supported) Detected the virtual environment created by pipenv
  • Appropriate suffix label specified in status bar
  • Prompt to install Pylint uses pipenv install --dev
  • "python.terminal.activateEnvironments": false deactivates detection
  • Create Terminal works
• (Linux/macOS) Virtual environments created under {workspaceFolder}/.direnv/python-{python_version} are detected (for direnv and its layout python3 support)
  • Appropriate suffix label specified in status bar
  • "python.terminal.activateEnvironments": false deactivates detection

Environment files

Sample files:

# example.py
import os
print('Hello,', os.environ.get('WHO'), '!')

# .env
WHO=world
PYTHONPATH=some/path/somewhere

Make sure to use Reload Window between tests to reset your environment!

• Environment variables in a .env file are exposed when running under the debugger
• "python.envFile" allows for specifying an environment file manually (e.g. Jedi picks up PYTHONPATH changes)
• envFile in a launch.json configuration works

Debugging

• pythonPath setting in your launch.json overrides your python.pythonPath default setting

Linting

ALWAYS check under the Problems tab to see e.g. if a linter is raising errors!

Pylint/default linting

[Prompting to install Pylint is covered under Environments above]

For testing the disablement of the default linting rules for Pylint:

# pylintrc
[MESSAGES CONTROL]
enable=bad-names

# example.py
foo = 42  # Marked as a blacklisted name.

• Installation via the prompt installs Pylint as appropriate
  • Uses --user for system-install of Python
  • Installs into a virtual environment environment directly
• Pylint works
• "python.linting.pylintUseMinimalCheckers": false turns off the default rules w/ no pylintrc file present
• The existence of a pylintrc file turns off the default rules

Other linters

You can always use the Run Linting command to immediately trigger a newly installed linter.

• flake8 works
• mypy works
• pep8 works
• prospector works
• pydocstyle works
• pylama works
• 3 or more linters work simultaneously
  • Run Linting runs all linters
  • The Select Linter command lists all the above linters and prompts to install a linter when missing
  • "python.linting.enabled": false disables all linters
  • The Enable Linting command changes "python.linting.enabled"
  • "python.linting.lintOnSave works

Editing

IntelliSense

Please also test for general accuracy on the most "interesting" code you can find.

• "python.autoComplete.extraPaths" works
• "python.autoComplete.preloadModules" works
• "python.autocomplete.addBrackets": true causes auto-completion of functions to append ()

Formatting

Sample file:

# There should be _some_ change after running `Format Document`.
def foo():pass

• Prompted to install a formatter if none installed and Format Document is run
  • Installing autopep8 works
  • Installing yapf works
• autopep8 works
• yapf works
• "editor.formatOnType": true works and has expected results

Refactoring

• Extract Variable works
  • You are prompted to install rope if it is not already available
• Extract method works
  • You are prompted to install rope if it is not already available
• Sort Imports works

Debugging

Test both old and new debugger (and notice if the new debugger seems at least as fast as the old debugger).

• Configurations work
  • Current File
  • Module
  • Attach
  • Terminal (integrated)
  • Terminal (external)
  • Django
  • Flask
  • Pyramid
  • Watson
  • Scrapy
  • PySpark
  • All debug Options with appropriate values edited to make values valid
• Breakpoints
  • Set
  • Hit
  • Conditional
    • Expression
    • Hit count
  • Log points (experimental debugger only)
• Stepping
  • Over
  • Into
  • Out
• Can inspect variables
  • Through hovering over variable in code
  • Variables section of debugger sidebar
• Remote debugging works
  • ... over SSH
• App Engine

Unit testing

unittest

import unittest

class PassingTests(unittest.TestCase):

    def test_passing(self):
        self.assertEqual(42, 42)

    def test_passing_still(self):
        self.assertEqual("silly walk", "silly walk")


class FailingTests(unittest.TestCase):

    def test_failure(self):
        self.assertEqual(42, -13)

    def test_failure_still(self):
        self.assertEqual("I'm right!", "no, I am!")

• Run All Unit Tests triggers the prompt to configure the test runner
• Tests are discovered (as shown by code lenses on each test)

pytest

def test_passing():
    assert 42 == 42

def test_failure():
    assert 42 == -13

• Run All Unit Tests triggers the prompt to configure the test runner
  • pytest gets installed
• Tests are discovered (as shown by code lenses on each test)

nose

def test_passing():
    assert 42 == 42

def test_failure():
    assert 42 == -13

• Run All Unit Tests triggers the prompt to configure the test runner
  • Nose gets installed
• Tests are discovered (as shown by code lenses on each test)

General

• Code lenses appears
  • Run Test lens works (and status bar updates as appropriate)
  • Debug Test lens works
  • Appropriate ✔/❌ shown for each test
• Status bar is functioning
  • Appropriate test results displayed
  • Run All Unit Tests works
  • Discover Unit Tests works (resets tests result display in status bar)
  • Run Unit Test Method ... works
  • View Unit Test Output works
  • After having at least one failure, Run Failed Tests works