Add support for extending core CLI

[ehmatthes]

This update is based on work done implementing support for --vm-size in dsd-flyio.

Pre-tasks

• Restructure plugin_config so it's defined in plugin_config.py.

Tasks

• Update deploy.py to include appropriate hook implementations.
• Add a cli.py file
• Use dummy arg in platform_deployer.py (in e2e test)
• Include an integration test for the dummy arg
• Include an integration test for the plugin-specific help output.
• Include an e2e test that exercises the CLI arg?
• Merge
• Announce

[ehmatthes]

Tests

• django-simple-deploy test suite really needs a CLI arg for passing custom args to test calls.
  • Implement this
  • Update dsd-flyio test to use this flag
  • Use it here for testing custom cli arg.
  • No, just uncomment the test for the CLI arg!
• Update integration tests to pass.
• Should see cli.py in generated project.
• Make sure Fly.io does not appear in final project.
• Write an e2e test that exercises the CLI arg.

[ehmatthes]

Other tasks

• Document current project structure. (here, or in main docs?)

dsd-newfly$ tree -L 4
.
├── CHANGELOG.md
├── LICENSE
├── MANIFEST.in
├── README.md
├── developer_resources
│   └── README.md
├── dsd_newfly
│   ├── __init__.py
│   ├── cli.py
│   ├── deploy.py
│   ├── deploy_messages.py
│   ├── platform_deployer.py
│   ├── plugin_config.py
│   └── templates
│       ├── dockerfile_example
│       └── settings.py
├── pyproject.toml
└── tests
    ├── conftest.py
    ├── e2e_tests
    │   ├── __init__.py
    │   ├── test_deployment.py
    │   └── utils.py
    └── integration_tests
        ├── reference_files
        │   ├── Pipfile
        │   ├── plugin_help_text_sample.txt
        │   ├── pyproject.toml
        │   ├── requirements.txt
        │   └── settings.py
        ├── test_custom_cli_arg.py
        ├── test_help_output.py
        └── test_newfly_config.py

8 directories, 26 files

[ehmatthes]

E2e test

• Modify project to have a CLI arg.
• Test help output.
• Make help test fail.
  • Going to main dsd dev env, install -e dsd-newfly from the temp test env, test for help output passes and then fails if I change the reference file.
  • Why doesn't the plugin test fail when omitting --setup-plugins-only if I change the source reference file? (Probably e2e_utils.get_core_plugin_test_cmd(). Yes, it was the -k test_{platform_name_lower}_config.py.) Fix: add -k test_help_output.py to default call.
  • Why do tests pass with --run-core-tests, but fail without that arg? (See notes below)
  • Try bumping pytest garbage collection threshold? (Helpful anyway, needed to be set on core and plugin generator.) (No effect on test outcome.)
  • Look more closely at e2e_utils.get_core_plugin_test_cmd().
• Make a test call that includes the CLI arg.
• Make assertions about effect of the CLI arg.
• Make sure tests can actually fail.

[ehmatthes]

Make e2e test fail when appropriate

This is a complicated test suite, because we're calling pytest which then builds an environment, and calls pytest from within that environment. The layers get confusing at times. The outer test needs to look at the inner test results to check for failures.

Here's some of the inner test suite output, captured by the outer test suite:

========== test session starts ==========
collected 78 items

tests/integration_tests/platform_agnostic_tests/test_console_output.py s [  1%]
sssss                                                                    [  7%]
...
tests/unit_tests/test_utils.py .................                         [ 75%]
test_help_output.py F                                                    [ 76%]
test_newfly_config.py ..................                                 [100%]
========== FAILURES ==========

I originally made this assertion to check for whether the internal tests failed:

def run_core_plugin_tests(path_dsd, plugin_config, cli_options):
    """Run django-simple-deploy's test suite with a plugin installed."""
    tests_dir = path_dsd / "tests"
    platform_name_lower = _get_platform_name_lower(plugin_config.platform_name)
    cmd = get_core_plugin_test_cmd(path_dsd, cli_options, platform_name_lower)

    output = subprocess.run(cmd, capture_output=True,shell=True)
    stdout = output.stdout.decode()
    print(stdout)

    assert "[100%]" in stdout
    check_core_plugin_tests(stdout, cli_options)

The issue is the assertion here. When a test fails in a way that interrupts execution of the test suite, that assertion fails. But when a test fails without interrupting the test suite execution, this assertion doesn't catch that.

There are two fixes:

• Print stdout after running and examining the internal test suite, so we can examine it by passing the -s flag in the outer pytest call. (done, this is what clarified this whole issue.)
• Make an assertion that FAILURES does not appear in captured output. (Check FAILED instead.)

[ehmatthes]

Future considerations

• There's lots of ways to make the test output useful. For example if the assertion fails, we can grab the most relevant lines from the internal output and print them immediately, or dump them to a report that's printed at the end.

[ehmatthes]

Working notes 9/21

• Consider renaming generator test to be distinct from test in plugin.

[ehmatthes]

Finishing up

• Consider renaming generator test to be distinct from test in plugin.
• Move "future considerations" to parking lot.
• Document current project structure.
• Update integration tests to pass.
• Make sure Fly.io does not appear in final project. (ignore for now)
• Include an integration test for the dummy arg (?) (no, this is in e2e tests)
• Include an integration test for the plugin-specific help output. (no, e2e tests)
• Include an e2e test that exercises the CLI arg?
• Merge
• Announce