Feature suggestion: run Django unittests

[DonJayamanne]

From @jvaesteves on January 16, 2017 22:28

Django unittest extends from unittests, but you can't run as the former on this extension. For this, you need to "manage.py test", but it would be awesome if there were those same shortcuts.

Copied from original issue: DonJayamanne/pythonVSCode#648

[DonJayamanne]

From @guhuajun on January 19, 2017 6:51

I did something like following. I think it's supported now.

[DonJayamanne]

From @jvaesteves on January 19, 2017 12:13

I tried your suggestion, copying the same parameters on the settings.json file, but running the "Run Unit Test Method..." command still doesn't work. The Output Console gives me the following message:

ImproperlyConfigured: Requested setting DATABASES, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

And then, when I configure this environment variable to my global_settings.py path, the message change to: ImportError: Import by filename is not supported.

[DonJayamanne]

From @guhuajun on January 20, 2017 0:22

https://github.com/DonJayamanne/pythonVSCode/wiki/unitest-(Standard-Unit-testing)-Framework#test-discovery
Please try to place a __init__.py.

Yes, missing DJANGO_SETTINGS_MODULE is annoying. I encountered this issue when writing tests for a django-cms project.

[DonJayamanne]

@jvaesteves Will be closing this issue as @guhuajun has a solution.

[DonJayamanne]

@jvaesteves please confirm @guhuajun suggestion works, so I can close this issue.

[DonJayamanne]

From @jvaesteves on February 2, 2017 10:46

No, it didn't.

[DonJayamanne]

From @james-perretta on March 5, 2017 4:35

It looks to me like the suggestion from @guhuajun gets us part of the way there for running Django test cases. Unless I'm mistaken, you can't really use unittest to run tests for any serious Django project.
When you run python manage.py test, it first does general setup (figuring out which settings module to use and calling django.setup()). After that, it hands the rest off to a test-runner class that does even more important things like creating and destroying the databases that the test cases need to operate on.

I did a bit of digging, and although I'm not particularly familiar with the implementation details of your plugin, I think this is a starter summary of what it would require to add this:

• Add client modules similar to those for nose and pytest (I have almost zero knowledge of how this part works)
• Add handling for testFx == 'django' in testlauncher.py. This would involve code similar to what you see here in the runtests.py snippet.
• Possibly add a workspace settings option that lets the user specify the value for DJANGO_SETTINGS_MODULE. That environment variable needs to be set before calling django.setup()
• Certain manage.py test options should always be passed, such as --no-input
• Add support for other django-specific unit test options, such as --tag, --exclude-tag, and maybe --parallel

Thoughts?

[DonJayamanne]

From @PlugaruT on October 3, 2017 11:15

Because I have different settings files for each environment (production, local, test), I can't run tests because with this plugin because DJANGO_SETTINGS_MODULE is not set when trying to run the tests. I would be nice if I could set this env variable in in the workspace.json or somewhere else.

[DonJayamanne]

@PlugaruT Environment variables can be set for the workspace. Just create a file named .env in your workspace root and add the necessary variables in there.

[wsantos]

Any update here?, I have a bunch of tests file, I'm not sure where to put django.setup() if that is a real solution.

[Kurara]

You don't need to configure DJANGO_SETTINGS_MODULE if you use the option: "--settings < filesetting >" Just saying because maybe we can make thah unittest plugin doesn't need env variable, just tell him the setting and if not, use default.

And just one question about my closed issue.

My problem is not implementation of new feature for django. But tests are not DISCOVERED at all. I could make them run at least in part, but if it doesn't discover them I can't. When I use python -m unittest it discovers the tests, so the structure must be ok.

[yardensachs]

Any decision on this? This is somewhat of a small ask, but it will make dev work better for those who use django and not pytest/nose/etc.

[brettcannon]

@yardensachs not at the moment (we're tied up with other things that impact all Python developers). If anyone wants to submit a PR to introduce a fix for this we would happily review it.

[chatne]

I have finally figured out a way to make unittests work with VSCode and Django with only one python file and without any additional configuration or setup.

VSCode test runner does test discovery every time, regardless of the amount of tests that are about be run.
We can leverage this discovery by running Django test setup during this phase. File or directory name for the setup file must be set so that the discovery finds it first. I use a directory in my project root called _unittest.

This is only tested on Ubuntu, also you must update filepaths to reflect your environment.

_unittest/__init__.py

"""
Hacks to make unittest work desirably.

VSCode + Python + Django + unittest integration is currently broken.
Django tests can be run using `python3 manage.py test` but this is not enough as VSCode unittest
runner only uses `python3 -m unittest`.

The purpose of this file is to simulate what the manage script is doing before any tests are run.
The trick is to make this load first when unittest does test discovery, hence the path `_unittest`.

This clearly cannot work when directly pointing to a test since test discovery is not used,
for example `python3 -m unittest server.component.test`.
However VSCode points to target tests differently and discovery still gets run making this work.

This all hopefully becomes obsolete if Django support is added to VSCode Python.
The issue: https://github.com/microsoft/vscode-python/issues/73
"""

import inspect
import os

import django
from django.conf import settings
from django.test.utils import get_runner


def _check_stack() -> bool:
    """
    Dig stack for the filepath of the script that imports this file.

    Return True if improting and invoking scripts are as expected.
    """
    # Skip first two calls which are this module and this function
    for frame in inspect.stack()[2:]:
        # Skip internal libraries which start with <
        if frame.filename[0] != "<":
            importing_script = frame.filename
            break
    else:
        raise RuntimeError("Importing script not found")

    # Unittest loader should be used
    if importing_script != "/usr/lib/python3.11/unittest/loader.py":
        return False

    # Django manage script uses the same loader, so check for that as well and skip if used
    first_invoking_script = inspect.stack()[-1].filename
    if first_invoking_script == "/workspace/backend/manage.py":
        return False

    return True


# This script should only be imported, so ignore direct execute
if __name__ == "__main__":
    pass

# Only allow this hack to work with unittest loader
elif _check_stack():
    # Setup Django for testing
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings.testing")
    django.setup()
    settings.ALLOWED_HOSTS += ["testserver"]

    # Setup test database for testing
    get_runner(settings)(interactive=False).setup_databases()

[carltongibson]

Hi. I raised this on mastodon, and @brettcannon suggested to post here.

I'm trying work out what we're missing from Django in order for this to work. The django-admin test command wraps a DiscoverRunner (src) that's based on (or uses) unittest, so the thought is it should work, and I wasn't able to see exactly what's missing.

I asked:

Is this because we need a --discover flag to the management command or something...?

…to which Brett pointed out that you're calling unittest API via Python, and not the CLI, so it becomes, roughly, what API are we missing from DiscoverRunner that would enable test discovery to work here? Presumably we need to be able to call into build_suite…🤔 — I guess my question is, what would this take from VS Code's end? What does it need to look like? If it's just exposing the right API, then surely we can get this working — or is there a structural issue we'd need to resolve first?

Any insight welcomed. Thanks.

[mh-firouzjah]

maybe it would be possible to patch the extension so it could check if a manage.py file is present in the root folder, then use it to run the tests else do the old way(current way)

[djbrown]

on a new, minimal django proect (polls app from tutorial),
the call for discovering tests looks like this for me: C:\Program Files\Python39\python.exe" ~\.vscode\extensions\ms-python.python-2022.18.2\pythonFiles\testing_tools\unittest_discovery.py . test*.py

so the vscode-python plugin calls its own disovery script: unittest_discovery.py

settings.json looks like this:

{
    "python.testing.unittestArgs": [
        "-v",
        "-p",
        "test*.py"
    ],
    "python.testing.pytestEnabled": false,
    "python.testing.unittestEnabled": true
}

full output (vscode > output > python)

> "C:\Program Files\Python39\python.exe" ~\.vscode\extensions\ms-python.python-2022.18.2\pythonFiles\testing_tools\unittest_discovery.py . test*.py
сwd: .
[ERROR 2023-1-9 18:29:1.465]: Error discovering unittest tests:
 Failed to import test module: polls.tests
Traceback (most recent call last):
  File "C:\Program Files\Python39\lib\unittest\loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "C:\Program Files\Python39\lib\unittest\loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "c:\Users\myusername\projects\django-vscode\polls\tests.py", line 6, in <module>
    from .models import Question
  File "c:\Users\myusername\projects\django-vscode\polls\models.py", line 4, in <module>
    class Question(models.Model):
  File "C:\Users\myusername\AppData\Roaming\Python\Python39\site-packages\django\db\models\base.py", line 127, in __new__
    app_config = apps.get_containing_app_config(module)
  File "C:\Users\myusername\AppData\Roaming\Python\Python39\site-packages\django\apps\registry.py", line 260, in get_containing_app_config
    self.check_apps_ready()
  File "C:\Users\myusername\AppData\Roaming\Python\Python39\site-packages\django\apps\registry.py", line 137, in check_apps_ready
    settings.INSTALLED_APPS
  File "C:\Users\myusername\AppData\Roaming\Python\Python39\site-packages\django\conf\__init__.py", line 92, in __getattr__
    self._setup(name)
  File "C:\Users\myusername\AppData\Roaming\Python\Python39\site-packages\django\conf\__init__.py", line 72, in _setup
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: Requested setting INSTALLED_APPS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

similarly as @jvaesteves posted on January 19, 2017 12:13

ImproperlyConfigured: Requested setting DATABASES, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

[brettcannon]

Looping in @eleanorjboyd and @karthiknadig to look into the questions that @carltongibson is asking.

[mh-firouzjah]

By using this fork of LittleFoxTeam.vscode-python-test-adapter extension the problem is fixed.

• there is a pull request

this is working by adding the following code to ~/.vscode/extensions/littlefoxteam.vscode-python-test-adapter-<version>/out/src/unittest/unittestTestRunner.js.

DON'T TRY AT HOME

from pathlib import Path
from ast import literal_eval
if Path(start_dir + "/manage.py").is_file():
    with open(start_dir + "/manage.py", "r") as management_file:
        contents = management_file.readlines()
        if any(True for line in contents if line.strip().replace('"""', '') == "Django\'s command-line utility for administrative tasks."):
            print("django management file found!")
            for line in contents:
                if line.strip().startswith("os.environ.setdefault"):
                    try:
                        literal_eval(line.strip().replace('os.environ.setdefault("DJANGO_SETTINGS_MODULE",', "", 1))
                    except:
                        pass
                    else:
                        eval(line.strip()) # this is the not recommended part!!!
                        try:
                            import django
                            django.setup()
                        except ModuleNotFoundError:
                            pass

[mh-firouzjah]

I have submitted a new pull request

• I'm not sure if it's perfect or not, but it works for my expectations

[carltongibson]

Replying to the recent comments:

import django
django.setup()

I didn't look at the details of the workarounds here but, just from a Django perspective, calling django.setup() is going to be needed. It requires an import path to a settings module, either via DJANGO_SETTINGS_MODULE env var or a --settings CLI flag to django-admin. (The default manage.py file sets a default for DJANGO_SETTINGS_MODULE to the settings file created with the startproject command, so that's not a bad fallback value.)

[mh-firouzjah]

@carltongibson I'm doing my best to make the changes more reliable and stable.(a new commit has been published)

[dimaulupov]

Hi, we have solved this problem in our project.
It needs a bit more than running django.setup().
Our code runs code below and full vscode testing pane works okay.

import django
from django.apps import apps
from django.test.utils import setup_databases, setup_test_environment
if not apps.ready:
    django.setup() # Normal Django setup
    setup_test_environment() # This does a lot of stuff inside Django tests
    # The next one is very important: it creates test databases and changes settings.DATABASES to point to them
    # otherwise tests will run against live database.
    setup_databases(verbosity=1, interactive=False, keepdb=True)
    # keepdb probably should be a setting inside vscode.
    # Our project takes an hour to run migrations from scratch, so we need keepdb, 
    # but normally no one wants to keep test databases around.

As a note if anyone is planning to do it before it is implemented in the extension.
This all is running in every <app>/tests/__init__.py.
Because if you are using something like Django Factory, which touches Django models on import, you'll get an exception just because discovery imported <app>/test/factories.py directly.
But if extension sets everything up - this won't be needed.
Also we run it only if there is "testlauncher.py" or "unittest_discovery" in sys.argv
So this workaround does not kick in under normal ./manage.py test

[carltongibson]

This all is running in every <app>/tests/__init__.py.

In the extension, it should be sufficient to do once this before kicking off the test discovery… 🤔

[eleanorjboyd]

Hello! I have begun to review this question of what VS Code would need to get Django test discovery and run working. So far I had a few questions that came up during exploring documentation and your repo.

It looks like the DiscoverRunner is going to be most useful to us as we could create it then call its methods. What steps need to be taken before we can successfully call DiscoverRunner.run_tests() ? From my understanding , when comparing the functionality found in DiscoverRunner to that of the test method found in manage.py, It seems that the test method handles both interpreting command line args and then calling DiscoverRunner.run_tests() - is this correct? Are these command line args then applied to the DiscoverRunner object? If you could explain a little more how the args are processed that would be helpful.

Next I saw that the method DiscoverRunner.build_suite returns either unittest.TestSuite or ParallelTestSuite what is the difference? Could you also explain more the need for the **kwargs as an argument?

For running tests, I see the method DiscoverRunner.run_tests method which seems to only return the number of tests which failed. What is the return value of DiscoverRunner.run_suite, does it give which tests failed and an error message? If not where could I get that information after run?

Generally I want to understand the key differences between Django tests and Unittests and if the Django test objects can just be stored as test_ids/labels (these seems to be paths) and unittest.TestSuite. Also I want to understand how the environment setup and command line args work- is there an easy method that would allow us to set command line args like how it is done in the test method found in manage.py?

Thank you and I look forward to working with you all!

[carltongibson]

Hi @eleanorjboyd. Thanks for picking this up. It would be very exciting to get this working! 🤩

I want to understand how the environment setup and command line args work- is there an easy method that would allow us to set command line args like how it is done in the test method found in manage.py?

Naïvely, the basics are setting the DJANGO_SETTINGS_MODULE environment variable, and then calling django.setup(). As noted in comments above here, there are a couple of other steps for the tests, like setting up the test database.

If you look at the ManagementUtility class (and the execute_from_command_line helper) — I think you'd have most of what you're asking for there. (See also though the call_command function (docs).)

The ManagementUtility.execute() calls django.setup() and dispatches to the test command. (So, going this route if you set the DJANGO_SETTINGS_MODULE environment variable, the rest is handled for you.)

However you need the results... 🤔

For running tests, I see the method DiscoverRunner.run_tests method which seems to only return the number of tests which failed. What is the return value of DiscoverRunner.run_suite, does it give which tests failed and an error message? If not where could I get that information after run?

So DiscoverRunner.run_suite gives you the TextTestResult that I think you're after?
You're right we discard that currently in run_tests, so for the first attempt I'd likely override run_suite to keep a reference to the result, and then hopefully you could use run_tests() from your subclass, as the test command does.

• This runs the database setup and such.
• We'd certainly look at keeping that result reference for you in Django's development branch if that's what you need to make this work long-term!

Next I saw that the method DiscoverRunner.build_suite returns either unittest.TestSuite or ParallelTestSuite what is the difference? Could you also explain more the need for the **kwargs as an argument?

ParallelTestSuite hides the fact that we're running multiple processes from the test runner, suppressing subprocess output, and combining into a single result. The build_suite() kwargs have never been used, as far as I can tell. (It's looks to me like a case of YAGNI but I'd have to dig a lot to say more... — they should be safe to ignore.)

Generally I want to understand the key differences between Django tests and Unittests and if the Django test objects can just be stored as test_ids/labels (these seems to be paths) and unittest.TestSuite.

At a high-level, I'd say we (cough) just wrap unittest, and hopefully there's all the API you need there (somewhere) — As I say, we're happy to look at adding the hooks you need here.

Generally labels are import paths to tests: myapp.tests.MyTestCase.test_my_method, but could be a directory path too. (build_suite reference).

Also I want to understand how the environment setup and command line args work- is there an easy method that would allow us to set command line args like how it is done in the test method found in manage.py?

Hopefully the ManagementUtility stuff at the top there is what you need?
For running the Django test suite itself we configure INSTALLED_APPS and other settings by hand (because they can vary depending on what tests you're running) — but that is equivalent to running django.setup().

Schematically, I would think something like this would be Pretty Close™:

os.environ["DJANGO_SETTINGS_MODULE"] = "user_project.settings"
django.setup()
runner = DiscoverRunner(**required_kwargs)  # 😬 Need to check
runner.run_tests(["user_app.tests"])

Does that give you the guidance you need? Let me know!

I don't know my way around the extension, but happy to look at a branch if you
can give me the dummy's guide steps to get it running.

Thank you and I look forward to working with you all!

You too! Thanks again. 💃

[eleanorjboyd]

Hello @carltongibson, great to meet you! Thank you for your in-depth reply! I am currently thinking on the next steps and talking with the team so I will get back to you soon. Your answers are extremely helpful in narrowing down what is needed and how to create compatibility. I am excited about the shared types with unittest and how this can make the integration more seamless.

Thanks!

[antoniogamizbadger]

So.. any update on this? I tried setting up our company project with VSCode and the extension is still unable to find django test modules.

[khamaileon]

Apparently the next step is here and we need votes to get things going. If the 300+ people who liked this post could make their voices heard.

[carltongibson]

@khamaileon Nice. Good spot. I didn't even know about that other issue.

[mh-firouzjah]

Apparently the next step is here and we need votes to get things going. If the 300+ people who liked this post could make their voices heard.

As long as the proposed procedure is taking approval, I have tested a new idea, and it worked well for me. I am awaiting feedback and information on whether it works properly for you and others, or if there are any issues.

I expected that after running the tests, there would be a change in the main project database Which it did not and the main database is clear; because in this approach, I did not use a CustomTestRunner.

NOTE: This method requires fewer changes in the vscode extension and is also easier to modify the django_settings_module for each project, especially when we have multiple modules as settings for different scenarios such as development, production, and so on.

1- Vscode Python Extension Requirement
edit the __init__.py module in ~/.vscode/extensions/ms-python.python-VERSION/pythonFiles/unittestadapter/__init__.py and add the code below:

try:
    import django
    django.setup()
except:
    pass

2- Django Project Requirement
create a .env file and inside of it declare DJANGO_SETTINGS_MODULE. an example is:

DJANGO_SETTINGS_MODULE=my_django_project_name.settings

[khamaileon]

@mh-firouzjah I tried but it didn't work out.

Here's my vscode settings:

{
  "editor.codeActionsOnSave": {
    "source.organizeImports": "explicit"
  },
  "editor.formatOnSave": true,
  "python.defaultInterpreterPath": "~/.virtualenvs/acme-api/bin/python",
  "python.envFile": "${workspaceFolder}/.env.local",
  "editor.rulers": [100],
  "python.testing.unittestArgs": ["-v", "-s", "./src", "-p", "test*.py"],
  "python.testing.pytestEnabled": false,
  "python.testing.unittestEnabled": true
}

And here's the kind of output I got:

2024-02-06 10:36:32.276 [info] Telemetry level is off
2024-02-06 10:36:32.276 [info] Experiments are disabled, only manually opted experiments are active.
2024-02-06 10:36:32.276 [info] Default formatter is set to ms-python.black-formatter for workspace /home/me/workspace/acme/mars-api
2024-02-06 10:36:32.276 [info] Test server listening.
2024-02-06 10:36:32.276 [info] VS Code was launched from an activated environment: 'mars-api', selecting it as the interpreter for workspace.
2024-02-06 10:36:32.276 [info] Python interpreter path: ~/.virtualenvs/mars-api/bin/python
2024-02-06 10:36:34.315 [info] Starting Pylance language server.
2024-02-06 10:36:43.381 [info] Discover tests for workspace name: mars-api - uri: /home/me/workspace/acme/mars-api
2024-02-06 10:36:43.405 [info] > . ~/.virtualenvs/mars-api/bin/activate && echo '<some_uuid>' && python ~/.vscode/extensions/ms-python.python-2024.0.0/pythonFiles/printEnvVariables.py
2024-02-06 10:36:43.405 [info] shell: bash
2024-02-06 10:36:43.457 [info] > ~/.virtualenvs/mars-api/bin/python ~/.vscode/extensions/ms-python.python-2024.0.0/pythonFiles/testing_tools/unittest_discovery.py ./src test*.py
2024-02-06 10:36:43.457 [info] cwd: .
2024-02-06 10:36:44.068 [error] Error discovering unittest tests:
 Failed to import test module: account.tests
Traceback (most recent call last):
  File "/usr/lib/python3.12/unittest/loader.py", line 394, in _find_test_path
    module = self._get_module_from_name(name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/unittest/loader.py", line 337, in _get_module_from_name
    __import__(name)
  File "/home/me/workspace/acme/mars-api/src/account/tests.py", line 15, in <module>
    from account.models import Session, User
  File "/home/me/workspace/acme/mars-api/src/account/models.py", line 3, in <module>
    from django.contrib.auth.models import AbstractUser
  File "/home/me/.virtualenvs/mars-api/lib/python3.12/site-packages/django/contrib/auth/models.py", line 3, in <module>
    from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
  File "/home/me/.virtualenvs/mars-api/lib/python3.12/site-packages/django/contrib/auth/base_user.py", line 58, in <module>
    class AbstractBaseUser(models.Model):
  File "/home/me/.virtualenvs/mars-api/lib/python3.12/site-packages/django/db/models/base.py", line 129, in __new__
    app_config = apps.get_containing_app_config(module)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.virtualenvs/mars-api/lib/python3.12/site-packages/django/apps/registry.py", line 260, in get_containing_app_config
    self.check_apps_ready()
  File "/home/me/.virtualenvs/mars-api/lib/python3.12/site-packages/django/apps/registry.py", line 138, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.

DJANGO_SETTINGS_MODULE=project.settings is read correctly, otherwise I'm told it's necessary.

The problem may be that my code is in a src folder and not in the root. 🤷‍♂️

[mh-firouzjah]

hi @khamaileon,
yes, DJANGO_SETTINGS_MODULE is read. but for the error you got it seems that django.setup() had not been called at the beginning of the process.
being in a subdirectory could be the case but if you also have error when using manage.py check/runserver/...
otherwise I don't think thats the problem here.

[khamaileon]

Yet I'm able to launch it with the manage command inside vscode and outside (from the src folder).
Here is my launch.json config file.

{
  // Use IntelliSense to learn about possible attributes.
  // Hover to view descriptions of existing attributes.
  // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Python: Django",
      "type": "python",
      "request": "launch",
      "program": "${workspaceFolder}/src/manage.py",
      "args": ["runserver"],
      "django": true,
      "envFile": "${workspaceFolder}/.env.local",
      "justMyCode": true
    }
  ]
}