readline documentation needs work

[ronaldoussoren]

BPO	6953
Nosy	@birkenfeld, @ronaldoussoren, @merwok, @berkerpeksag, @vadmium, @andy-maier
Files	readline-doc.patch readline-doc.v2.patch readline-doc.v3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-04-05.22:11:25.769>
created_at = <Date 2009-09-20.14:13:31.261>
labels = ['type-feature', 'docs']
title = 'readline documentation needs work'
updated_at = <Date 2016-04-05.22:11:25.767>
user = 'https://github.com/ronaldoussoren'

bugs.python.org fields:

activity = <Date 2016-04-05.22:11:25.767>
actor = 'martin.panter'
assignee = 'docs@python'
closed = True
closed_date = <Date 2016-04-05.22:11:25.769>
closer = 'martin.panter'
components = ['Documentation']
creation = <Date 2009-09-20.14:13:31.261>
creator = 'ronaldoussoren'
dependencies = []
files = ['41157', '42281', '42362']
hgrepos = []
issue_num = 6953
keywords = ['patch']
message_count = 14.0
messages = ['92893', '100466', '100757', '100759', '100762', '100767', '222284', '255318', '259589', '262400', '262841', '262890', '262897', '262924']
nosy_count = 11.0
nosy_names = ['georg.brandl', 'ronaldoussoren', 'eric.araujo', 'Alexander.Belopolsky', 'stefanholek', 'python-dev', 'berker.peksag', 'Gabi.Davar', 'martin.panter', 'hideaki_t', 'andymaier']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue6953'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']

[ronaldoussoren]

The documentation for the readline module is a bit too minimal.

1. function 'add_history' is described at the end of the documentation,
   not near the other functions for manipulation the history stack.

2. the index for remove_history_item and replace_history_item is 0-
   based,
   while the index for get_history_item is 1-based.

The latter is a confusing API in the underlying readline library and
should IMO be described in Python's documentation.

(If readline were a new addition to the stdlib I'd prefer to fix the
interface, but that would have too many backward compatibility issues at
this point in time).

[stefanholek]

It is history_base-based, not 1-based. The history_base changes when the history is stifled and old entries "drop off" the list.

http://tiswww.case.edu/php/chet/readline/history.html#IDX36

[stefanholek]

To be zero-based, get_history_item would need to look like:

diff --git a/rl/readline.c b/rl/readline.c
index 33e9905..800bc00 100644
--- a/rl/readline.c
+++ b/rl/readline.c
@@ -559,7 +559,7 @@ get_history_item(PyObject *self, PyObject *args)
 
        if (!PyArg_ParseTuple(args, "i:index", &idx))
                return NULL;
-       if ((hist_ent = history_get(idx)))
+       if ((hist_ent = history_get(history_base + idx)))
                return PyString_FromString(hist_ent->line);
        else {
                Py_RETURN_NONE;

[AlexanderBelopolsky]

On Tue, Mar 9, 2010 at 4:24 PM, stefanholek <report@bugs.python.org> wrote:
..

To be zero-based, get_history_item would need to look like:
..

•       if ((hist_ent = history_get(history_base + idx)))

Did you test this with libedit?

[ronaldoussoren]

Changing get_history_item to be 0-based would be a backward incompatible change.

The point of my report is that the documentation of the readline documentation should mention how the APIs actually behave, you currently have to hunt down that information in the documentation from libreadline.

Stefan's message does point to a potentional issue though: history_base is not exposed to python code, and hence Python code cannot use the correct offset. I don't understand the documentation for history_get in that same document though, it says "get the item at offset, starting from history_base". I don't understand if that means that offset must be at least history_base or something else (and won't read the readline sources to find out).

[stefanholek]

I have read the readline source code, and it does mean just that. :-)

Anyway, this does not really apply to the stdlib because unless someone implements 'stifle_history' and friends 'history_base' is going to be 1 at all times.

[andy-maier]

I would like to revive this issue.

From the discussion, it seems to me that the following changes in the Python Library documentation would make sense:

1. Move add_history() higher up in the sequence of functions, for example to after write_history_file().

2. Clarify that the pos arguments of remove_history_item() and replace_history_item() are 0-based.

3. Clarify that the index argument of get_history_item() is 1-based.
   I do understand Stefan's comment from msg100757 that it is actually 'history_base'-based. On the one hand, history_base is not exposed or implied or even described at the Python level. After searching the GNU readline documentation, it seems that a notion of history base is not described there, either (I may have missed it, though).
   So either we simply state it is 1-based, or we provide in addition the background story mentioning some notion of history_base that is publicly described.

I have the following additional comments:

4. The current documentation is very abridged, probably because it intends to be "just" a description of the Python binding to GNU readline. I think it either needs to evolve into a standalone description (i.e. that does not depend on the description of GNU readline), or it needs to properly reference the description of GNU readline. If we go that route, a simple reference to the document is not sufficient, for one because it is not the only underlying implementation, and second, because it is large and not easy at all to map to the Python readline functions.

5. One needs to understand what the main entities are the module operates on, e.g. init file, history file(s), a (single, global?) history object, the line buffer. Regardless of what we do regarding comment 4., I think the Python docs should describe these main entities in the introduction text.

6. Some more information about the init file is needed. I suspect it is specific to the underlying implementation that is used. If so, add references to the format descriptions for each implementation (by Python OS platform).

7. parse_and_bind(): Change the description to state that it parses and binds the init statement provided in the string argument. That string may or may not come from an init file. The example at the end specifies a statement that is not from an init file.

8. get_line_buffer() talks about "line buffer" and insert_text() talks about "command line". I suspect that means to be the same. If so, use one term consistently.

9. read_init_file(): I suspect it returns a tuple of statements from the init file? In any case, describe how the init file content comes back. Are comments removed? Where is the last filename used remembered, does that survive restarts of the Python runtime?

10. read_history_file(): Add that the history file content is put into a (global?) history object, replacing its prior history.

11. write_history_file(): Add that the (global?) history object is where the information comes from. Is an existing history file replaced? Appended? Exception raised?

12. clear_history(): From the text, I read that if the underlying GNU readline does not support it, this Python function does not exist in the module. If this is not how it works (e.g. if the function exists and raises an exception in the unsupported case), that text should be clarified.
    Also, mention which version of GNU readline is minimally needed in order to support the function.

13. get_history_length(): It says "get the desired length of the history file", but I think it is really the desired number of lines in the history file (i.e. consistent with set_history_length().

14. get_history_item() and remove_history_item() talk about "history item", while all other functions talk about "history line" or "line". Use one term consistently.

15. the completion related functions (from set_completer() to set_completion_display_matches_hook()) are really somewhat short:
    What is the role of a completer function?
    Which completion types are defined?
    What is the beginning index of tab-completion?
    What are word delimiters for tab-completion?

16. Last but not least, the libedit library is mentioned for MacOS. Has that been implemented yet, and is it part of standard Python? If so, mention that.

Andy

[vadmium]

This patch addresses the following points:

1: Moved add_history() into new “History file” section with related functions.

2: Documented that remove_ and replace_history_item() are zero-based.

3: Documented that get_history_item() is one-based.

4: Listed the main underlying Readline function or variable that each Python function accesses. In many cases it should be fairly obvious because the name is essentially the same, but not always. I added them all for consistency.

5: Added six new sections to group functions: Init file, Line buffer, History file, History list, Startup hooks, and Completer.

7: Clarify that the parse_and_bind() line comes directly from the string argument, not a file.

8: Changed “command line” to “line buffer”.

9: read_init_file() also executes the bindings it reads; it does not return anything.

10: read_history_file() appends to the history list, contrary to what Andy suggested.

11: write_history_file() overwrites any existing file, and writes the history list

12: clear_history() is conditionally compiled in

13: Combined get_ and set_history_length() entries

14: Differentiated lines in the history file from history list items, which can be multi-line if an item includes a line break.

15: Added brief description of word completion in the new section. Entry get_completion_type() refers to the rl_completion_type variable, so it should be refer back to the Gnu documentation. Referred beginning and ending indexes to the corresponding callback arguments (these are affected by bpo-16182). Suggested that the word delimiters determine the word completion scope.

16: Whether libedit (Editline) is used or not really only depends on the -lreadline library at run time (out of Python’s hands). I clarified that its detection is what is implemented on OS X. See also bpo-13501 for expanding this support.

Another change:

17: Expanded information on setting up the completion word delimiters for a custom completer, to address bpo-10796.

Not addressed in my patch:

6: Init file information. May be better to just refer to the Gnu documentation <https://cnswww.cns.cwru.edu/php/chet/readline/readline.html#SEC1\> and Editline documentation (does this exist?).

8: I do not know what “the last filename used” means for read_init_file(). It passes a null pointer as the filename in this case. I suspect the answer may actually be it uses the INPUTRC config file instead.

12: I do not know what Readline version clear_history() is available in. Anyway, this might vary between Gnu and Editline implementations.

[berkerpeksag]

Looks good to me. I left a comment on Rietveld.

[vadmium]

Here is a modified patch. Does the new wording for clear_history() clarify things for you Berker? In theory that function could be included or omitted for both the Gnu Readline and Editline cases.

[vadmium]

Here is v3 which adds “. . . in the underlying library” for all the function and variable references.

[berkerpeksag]

readline-doc.v3.patch looks good to me.

[python-dev]

New changeset 6137c46cb8df by Martin Panter in branch '2.7':
Issue bpo-6953: Rearrange and expand Readline module documentation
https://hg.python.org/cpython/rev/6137c46cb8df

New changeset b1acd6cf15b6 by Martin Panter in branch '3.5':
Issue bpo-6953: Rearrange and expand Readline module documentation
https://hg.python.org/cpython/rev/b1acd6cf15b6

New changeset 856d50130154 by Martin Panter in branch 'default':
Issue bpo-6953: Merge readline doc from 3.5
https://hg.python.org/cpython/rev/856d50130154

[vadmium]

Thanks for the reviews.