Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add test discovery and runnables for Python files #12080

Open
2 of 3 tasks
JosephTLyons opened this issue May 21, 2024 · 8 comments · May be fixed by #18824
Open
2 of 3 tasks

Add test discovery and runnables for Python files #12080

JosephTLyons opened this issue May 21, 2024 · 8 comments · May be fixed by #18824
Labels
enhancement [core label] language An umbrella label for all programming languages syntax behaviors python Python programming language support

Comments

@JosephTLyons
Copy link
Collaborator

JosephTLyons commented May 21, 2024

Check for existing issues

  • Completed

Describe the feature

There are a load of python testing frameworks, but of those, the most popular are:

It would be great if we could offer testing support for both unittest and pytest, as I think that would cover most users. We should include commands to run all tests, for both frameworks, and to discover and run single tests, for both frameworks. All that being said, I'm not sure if our task infrastructure is really set up to handle being able to run tests from different testing frameworks. @osiewicz would know more.


Honorable mentions for Python testing frameworks go to ward - its modern, neat, and then one I reach for these days.

If applicable, add mockups / screenshots to help present your vision of the feature

No response

@JosephTLyons JosephTLyons added enhancement [core label] python Python programming language support language An umbrella label for all programming languages syntax behaviors labels May 21, 2024
@JosephTLyons
Copy link
Collaborator Author

After thinking about it, it seems like we should bake unittest testing into the zed python crate, since its part of the standard lib, and then offer pytest testing via an extension, since its a 3rd-party python package.

osiewicz pushed a commit that referenced this issue Jun 1, 2024
Add runnable tasks for Python, starting with `unittest` from the
standard library. Both `TestCase`s (classes meant to be a unit of
testing) and individual test functions in a `TestCase` will have
runnable icons. For completeness, I also included a task that will run
`unittest` on the current file.

The implementation follows the `unittest` CLI. The unittest module can
be used from the command line to run tests from modules, classes or even
individual test methods:

```
python -m unittest test_module.TestClass
python -m unittest test_module.TestClass.test_method
```

```python
import unittest

class TestStringMethods(unittest.TestCase):

    def test_upper(self):
        self.assertEqual('foo'.upper(), 'FOO')

    def test_isupper(self):
        self.assertTrue('FOO'.isupper())
        self.assertFalse('Foo'.isupper())

    def test_split(self):
        s = 'hello world'
        self.assertEqual(s.split(), ['hello', 'world'])
        # check that s.split fails when the separator is not a string
        with self.assertRaises(TypeError):
            s.split(2)

if __name__ == '__main__':
    unittest.main()
```

From the snippet provided by `unittest` docs, a user may want to run
test_split independently of the other test functions in the test case.
Hence, I decided to make each test function runnable despite `TestCase`s
being the unit of testing.

## Example of running a `TestCase`
<img width="600" alt="image"
src="https://github.com/zed-industries/zed/assets/16619392/7be38b71-9d51-4b44-9840-f819502d600a">

## Example of running a test function in a `TestCase`
<img width="600" alt="image"
src="https://github.com/zed-industries/zed/assets/16619392/f0b6274c-4fa7-424e-a0f5-1dc723842046">

`unittest` will also run the `setUp` and `tearDown` fixtures.

Eventually, I want to add the more commonly used `pytest` runnables
(perhaps as an extension instead).

Release Notes:

- Added runnable tasks for Python `unittest`.
([#12080](#12080)).
@ThomAub
Copy link
Contributor

ThomAub commented Jun 3, 2024

@JosephTLyons Should pytest be in an extension or could it be added to the current implementation of @rayduck ?
I think in python pytest is the defacto solutions to testing.

@rayduck
Copy link
Contributor

rayduck commented Jun 3, 2024

A python user is likely to go for pytest as the test runner instead of unittest, hence pytest support should be built-in to the python crate, just like how the non-standard lib pyright LSP is included.

Ideally, zed checks the project's venv for the presence of pytest (perhaps looking at pyproject.toml) and if found, runs that particular pytest so any plugin or user config is preserved. Otherwise, fallback to unittest support.

What I think may be missing is (1) using different test discovery rules (i.e what's in runnables.scm) conditionally or having additional tags like @run-pytest that somehow would not show up in the UI when pytest is not available (pytest's discovery ruleset is a superset of unittest's). @osiewicz do you reckon this is possible today?

Less importantly, (2) a solid detection of venv or conda envs (in the meantime, just let users specify a local config for which test runner to use).

@osiewicz
Copy link
Contributor

osiewicz commented Jul 20, 2024

Yep, we can now adjust tasks to "fit" user system configuration (e.g. installed test runners and so on). Looking at the venv may be a bit tricky, but doable

@lazarust
Copy link

+1 to adding pytest support! It would also be nice to be able to debug a test in the IDE

@NeilRiver
Copy link

oh damn! i'm ready to use zed in production today! give me pytest plz
to be honest i don't know a single living person on this planet who uses unittest (qa) in python
FYI @JosephTLyons

@r-egor
Copy link

r-egor commented Aug 19, 2024

FYI @JosephTLyons @osiewicz
Please, don't forget.
Proszę, nie zapomnijcie.

@BozhenaVlasova
Copy link

FYI @JosephTLyons @osiewicz @rayduck

Drogi programiście, możesz pozwolić lwiej części społeczności swobodnie zająć się tworzeniem testów, jeśli wypuścisz pytest.

Dear developer, you can free up the lion's share of the community to write tests if you release pytest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement [core label] language An umbrella label for all programming languages syntax behaviors python Python programming language support
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants