Initial support for argcomplete for KVArgParseConfigLoader (#811)

* Initial support for argcomplete for KVArgParseConfigLoader

After enabling argcomplete shell completion for a traitlets.Application
based script (e.g. via activate-global-python-argcomplete), this commit
sets up tab auto-completion for command-line flags and aliases.

Argcomplete completers can be added for a trait by using an "argcompleter"
metadata tag, which should have a function that takes keyword arguments
(passed down from argcomplete) and returns a list of string completions.
Completers are also set up for Bool ("true", "false", "1", "0") & Enum.

This commit does *not* add general support for arbitrary class traits
of the form --Class.trait=xyz, as these are currently not added to
the ArgumentParser instance, but rather directly parsed. It is probably
possible to add this support for the classes in Application.classes.

Issue: #539

Example:

~/dev/traitlets$ python examples/myapp.py --[TAB]
--debug      --disable    --enable     --enabled    --help       --i          --j          --log_level  --mode       --name       --running

~/dev/traitlets$ python examples/myapp.py --running [TAB]
0      1      false  true

~/dev/traitlets$ python examples/myapp.py --running true --[TAB]
--debug      --disable    --enable     --enabled    --help       --i          --j          --log_level  --mode       --name       --running

~/dev/traitlets$ python examples/myapp.py --running true --mode o[TAB]
off    on     other

* Custom argcomplete.CompletionFinder for class traits

This custom finder mainly adds 2 functionalities:

1. When completing options, it will add --Class. to the
list of completions, for each class in Application.classes
that could complete the current option.

2. If it detects that we are currently trying to complete
an option related to --Class., it will add the corresponding
config traits of Class to the ArgumentParser instance,
so that the traits' completers can be used. (This is currently
done in a bit of a hacky manner.)

Note that we are avoiding adding all config traits of all classes
to the ArgumentParser, which would be easier but would add more
runtime overhead and would also make completions
appear more spammy. It also does not support nested class
options like --Class1.Class2.trait.

Example:

~/dev/traitlets$ examples/myapp.py --mode on --[TAB]
--Application.  --Foo.          --debug         --enable        --help          --j             --mode          --running
--Bar.          --MyApp.        --disable       --enabled       --i             --log_level     --name
~/dev/traitlets$ examples/myapp.py --mode on --F[TAB]
~/dev/traitlets$ examples/myapp.py --mode on --Foo.[TAB]
--Foo.i     --Foo.j     --Foo.mode  --Foo.name
~/dev/traitlets$ examples/myapp.py --mode on --Foo.m[TAB]
~/dev/traitlets$ examples/myapp.py --mode on --Foo.mode [TAB]
~/dev/traitlets$ examples/myapp.py --mode on --Foo.mode o[TAB]
off    on     other

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Example application for testing argcomplete

Added an example application for testing argcomplete,
under examples/argcomplete_app.py, with several examples
of completions provided in the docstring.

Fixed using completers

--Class.trait arg1 arg2 [TAB]

for config traits with nargs/multiplicity="+". Note that
currently traitlets does not support multiplicity even though
it is used in the code; refer to issue GH#690 for discussion.

Add more comments since we're using argcomplete internals,
and add argcomplete as dependency for coverage/mypy tests.
Some other minor fixes such as minor mypy annotations fixes.

Another example: add # PYTHON_ARGCOMPLETE_OK to bin/ipython,
and tab-complete away:

    $ ipython --[TAB]
    --Application.               --help
    --BaseFormatter.             --i
    --BaseIPythonApplication.    --ignore-cwd
    --Completer.                 --init
    --HistoryAccessor.           --ipython-dir
    --HistoryManager.            --log-level
    --IPCompleter.               --logappend
    --InteractiveShell.          --logfile
    --InteractiveShellApp.       --m
    ...

    $ ipython --gui=[TAB]
    asyncio  gtk      gtk3     pyglet   qt4      tk
    glut     gtk2     osx      qt       qt5      wx

To-do still: support subcommands. This may still take some work
as traitlets does subcommand parsing independently of argparse.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Initial traitlets subcommand support for argcomplete

argcomplete's strategy is to call the python script with
no arguments e.g. len(sys.argv) == 1, run until the ArgumentParser
is constructed and determine what completions are available.
On the other hand, traitlet's subcommand-handling strategy
is to check sys.argv[1] and see if it matches a subcommand,
and if so then dynamically load the subcommand app and initialize
it with sys.argv[1:].

Write a couple of helper functions to reconcile this by:

1. retrieving tokens from $COMP_LINES, etc, and setting it to argv
2. if traitlets descends into a subcommand, increment index passed
   via env var to argcomplete to mark where command starts

There's quite a few caveats to this approach. For example, it only
is evaluated currently when `App.initialize()` is passed with
`argv=None` (the default). If `argv` is explicitly passed, then
the `argcomplete`-specific handling is skipped currently.

More details in:
#811 (comment)

Some additional minor cleanup with respect to subcommands typing,
examples, and documentation.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Documentation updates

Add docs for argcomplete, and various fixes and improvements
to other docs such as examples for flags/aliases/subcommands.

Update docs for building docs.

Also fix a bug when argcomplete is not installed

* noqa for import argcomplete, more doc updates

* Fix ruff removing argcomplete import check
* Add example scripts corresponding to examples in docs
* Add new sections to docs to further explain Application
configuration, methods, and philosophy.

Closes: #707, #708, #709, #712

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add unit tests for argcomplete & Application

Borrow argcomplete's unit test fixtures to set up
unit testing for completion of Application commands.
Test a few cases of custom completers; caught a bug
with Enum.argcompleter()

* Lint fixes

Fix some errors from hatch run typing:test
and some other minor formatting / CI fixes.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix argcomplete unit tests for argcomplete >= 2.0

argcomplete >= 2.0 was very recently released, and
changes internal output_stream to text mode instead
of bytes; fix unit test for this and some other minor fixes.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Skip failing unit tests?

These tests run for me locally, but currently CI raises
OSError: Bad file descriptor. Something with temp files perhaps?

* More fixes to appease ruff

* Restore argcomplete unit tests

They were failing on some issue with TemporaryFile,
trying again now.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Minor, fix mdformat lint sourcecode -> code-block

* Use StringIO instead of TemporaryFile for argcomplete unit tests

TemporaryFile was causing some OSError(9) Bad file descriptor
on flush, not sure why but using StringIO seems to work
perfectly fine instead.

* Require argcomplete>=2.0 for tests

Only use StringIO instead of BytesIO for argcomplete >= 2.0.
Stop relying on SystemExit which might be messing with pytest,
instead raise a CustomError. Other minor doc fixes.

* Disable _ARC_DEBUG

still trying to get pytest to not crash at the end ..

* Add pytest-mock and mock os.fdopen for argcomplete

argcomplete==2.0.0 always calls fdopen(9, "w") to open a debug stream,
however this could conflict with file descriptors used by pytest
and lead to obscure errors. Since we are not looking at debug stream
in these tests, just mock this fdopen call out.

* Fix pyproject.toml changes, a few more examples/docs

Polish up a bit more docs about Application and commit
corresponding examples.

* Ignore examples/docs/configs/ from pytest, mypy

config.py files have get_config, load_subconfig injected
so have to get them ignored from pytest collection and
mypy checking.

Also minor, added some pathlib support to load_config_file().

* Update corresponding example configs paths in docs

Also link to some examples and add initial CHANGELOG entry.

* Undo changelog.md for release

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

docs/readme-docs.md

@@ -1,47 +1,38 @@
 # Documenting traitlets
 
-[Documentation for `traitlets`](https://traitlets.readthedocs.io/en/latest/)
-is hosted on ReadTheDocs.
+[Documentation for `traitlets`](https://traitlets.readthedocs.io/en/latest/) is hosted on ReadTheDocs.
 
 ## Build documentation locally
 
-#### Change directory to documentation root:
+With [`hatch`](https://hatch.pypa.io/), one can build environments, docs, and serve it in one command:
 
 ```
-cd docs
+pip install hatch
+hatch run docs:build
 ```
 
-#### Create environment
+#### Build documentation manually
 
-- \[**conda**\] Create conda env (and install relevant dependencies):
+Otherwise to build docs manually,
 
-  ```
-    conda env create -f environment.yml
-  ```
+```
+cd docs
+```
 
-- \[**pip**\] Create virtual environment (and install relevant dependencies):
+Create virtual environment (and install relevant dependencies):
 
-  ```
+```
     virtualenv traitlets_docs -p python3
-    pip install -r requirements.txt
-  ```
-
-#### Activate the newly built environment `traitlets_docs`
-
-- \[**conda**\] Activate conda env:
-
-  ```
-    source activate traitlets_docs
-  ```
+    pip install -r traitlets[docs]
+```
 
-- \[**pip**\] The virtualenv should have been automatically activated. If
-  not:
+The virtualenv should have been automatically activated. If not:
 
-  ```
-    source activate
-  ```
+```
+source activate
+```
 
-#### Build documentation using:
+##### Build documentation using:
 
 - Makefile for Linux and OS X:
 
@@ -55,7 +46,7 @@ cd docs
     make.bat html
   ```
 
-#### Display the documentation locally
+##### Display the documentation locally
 
 - Navigate to `build/html/index.html` in your browser.
 
@@ -77,5 +68,3 @@ cd docs
 - `source/conf.py` - Sphinx build configuration file
 - `source` directory - source for documentation
 - `source/index.rst` - Main landing page of the Sphinx documentation
-- `requirements.txt` - list of packages to install when using pip
-- `environment.yml` - list of packages to install when using conda