DocumentationAPI: Add padding

[damngamerz]

Add padding in DocstyleDefinition which
are now acquired from .coalang. Add
top_padding and bottom_padding which
automatically instantiate from
instantiate_padding in DocBaseClass. Add
supporting test cases.

Related to #4200

For short term contributors: we understand that getting your commits well
defined like we require is a hard task and takes some learning. If you
look to help without wanting to contribute long term there's no need
for you to learn this. Just drop us a message and we'll take care of brushing
up your stuff for merge!

Checklist

• I read the commit guidelines and I've followed
  them.
• I ran coala over my code locally. (All commits have to pass
  individually. It is not sufficient to have "fixup commits" on your PR,
  our bot will still report the issues for the previous commit.) You will
  likely receive a lot of bot comments and build failures if coala does not
  pass on every single commit!

After you submit your pull request, DO NOT click the 'Update Branch' button.
When asked for a rebase, consult coala.io/rebase
instead.

Please consider helping us by reviewing other peoples pull requests as well:

• pick up any PR at https://coala.io/review
• review it (check https://coala.io/reviewing for more info)
• if you are sure that it needs work, use cobot mark wip <URL> to get it out
  of the review queue.

The more you review, the more your score will grow at coala.io and we will
review your PRs faster!

[jayvdb]

I see you have not add padding for all other languages.
But you make padding mandatory.
So this would break all languages without a padding in their style?

[damngamerz]

@jayvdb This answers your question :) . .coalang will have official standards defined.
By default top_padding and bottom_padding of DocComment are initialized with zero. If we don't have style defined. It will have default values. Also padding's major motive was if a developer doesn't wants to follow official standard and decides suppose I want to have 2 blank lines at the bottom of docstring. He will set those padding variables and it will be amended by the bear part.

[jayvdb]

Why split this into to variables rather than store a tuple?

[damngamerz]

using separate variables it becomes easily available at bear side. DocComment.docstyle_definition.top_padding/bottom_padding Also they are read as strings from the file(need int). I need to get separate values in the bear part anyhow. So why not split them in the first place.

[jayvdb]

You can used NamedTuple to make it easier for them to be used.
The reason to keep them as a tuple is so that everywhere they are a pair, from style config to bear. consistency.

[jayvdb]

Is it possible to provide a default value for this new arg, thereby avoiding breaking the API?

[damngamerz]

let me see though. But I don't think that would cause any problem because load function provides an empty tuple if it finds nothing. Which I later break into the properties of top_padding and bottom_padding. which is taken care by #4557 (comment)

[damngamerz]

yes this breaks the API. I tried to provide a default value. But the thing is load returns an empty tuple if it finds nothing which overwrites that default value. The best solution to providing default values through https://github.com/coala/coala/pull/4557/files#diff-bb503d77ea6d0498c9865cab08c904f4R64

[damngamerz]

This is important I couldn't find any other way to do this :)

[damngamerz]

@jayvdb We can continue our discussion here.... As default.coalang will take care of the intended blank lines.
So for python I believe is that PEP-257 is old. Even we don't follow it's official multi line docstring. And I feel what we use here at coala is much better(which is an enhanced inherited version of PEP - 257 standards).
atm we don't have any mechanism to describe the docstring type. Is it normal function docstring or class function docstring?
so any ideas what can be top_padding and bottom_padding which can be applied to all docstrings in coala repo, coala-bears? my bets are on (0,1). As generally we need to have a blank line after the docstring(such as in class function).
Or if we don't implement this now, leave it as (0,0) cause this is a trivial task. And find some way to get docstring origin(function docstring, class function docstring....etc).

[jayvdb]

All of the examples seem to be using padding (1, 1).

I re-iterate what Niklas has said at coala/coala-bears#1943 (comment) , you need to be creating your own scenarios which test different valid values, even if there is no defined style with that scenario of values.

[jayvdb]

this should be in method setUpClass or setUp .

[jayvdb]

this (and the lines above) should be in method setUpClass or setUp

[jayvdb]

nope; wrong per https://www.stack.nl/~dimitri/doxygen/manual/docblocks.html#pythonblocks

(and also this fixup must undo the bad changes in the previous patch to default.py : #4549 (review) )

[jayvdb]

So for python I believe is that PEP-257 is old.

No, you are wrong. On https://www.python.org/dev/peps/pep-0257/ , you can see it's status is "Active". It has not been replaced. It is maintained at https://github.com/python/peps/commits/master/pep-0257.txt
And https://github.com/pycqa/pydocstyle implements it. And there are vibrant discussions about the finer details of it, and pydocstyle allowing ignoring some of its rules to allow for alternative styles (and recently has a few extra rules to enforce other styles), but nobody dares say pep 257 is not the official style guide of Python.

And I feel what we use here at coala is much better(which is an enhanced inherited version of PEP - 257 standards).

Again, wrong.
What coala uses is not particularly good. The reason it is bad is because we have not undertaken a project to standardise the docstrings. No bears are enabled. The docstrings can contain anything. Only manual code review can catch problems. Your project is intended to solve that. Or we can work towards enabling PyDocStyleBear.
There is also a bug in QuotesBear which imposes a silly PEP 257 violation.
In fact Stefan and I were complaining about the bears bad behaviour recently on another patch very recently. We were asking for changes without realising that QuotesBear prohibited it.

atm we don't have any mechanism to describe the docstring type.

A design problem, for sure.

[jayvdb]

this blank line is also not acceptable.

[jayvdb]

this blank line is also not acceptable.

[jayvdb]

this blank line is also not acceptable.

[NiklasMM]

uhhh, I still don't understand. The padding is 0

[NiklasMM]

Why do we need getters for all those members?

[damngamerz]

As everything we are fetching from .coalang is a standard. And we don't want anyone to set those up manually. Hence setting them as getters :)
PS- Even the previous things like markers, comment's signature....etc we are fetching from .coalang are introduced in DocstyleDefinition as getters.

[NiklasMM]

hm, ok... since Python doesn't have private members, the whole concept is a bit useless. But if you want to do it like that it's fine.

[NiklasMM]

Comment from previous review unadressed

[NiklasMM]

This is still unchanged and very inefficient. This method should be generator.

You have two options:

1. Change instantiate_padding and instantiate_docstring_type to only process one docstring at a time, so this can become a generator.
2. move the functionality from instantiate_padding and instantiate_docstring_type into extract_documentation_with_markers and use yield from

I prefer the second option.

[damngamerz]

Yes I missed your previous comment we can do that...I see 👍

[NiklasMM]

you don't have to use get with a default value, that's why you're in a try/except block. And why are you casting this to a tuple? isn't it already an iterable?

[damngamerz]

yes no need to use get now 🎉
yup this isn't an iterable. we need to cast into a tuple.

[NiklasMM]

what is it then?

[damngamerz]

when this reads function_padding from .coalang the values are just read as a strings.
suppose function_padding = 1, 0
This will be read as
docstyle_settings = {..., function_padding : '1, 0',...}
we cast then into a tuple ('1', '0') so they get as iterables and then we typecast them into integers.

[NiklasMM]

what is this for?

[damngamerz]

yup we can remove this I guess...
as now I will use docstyle_settings['function_padding'] instead of docstyle_settings.get()
which will throw a KeyError 👍

[SanketDG]

this is what should throw the KeyError right?

This shouldn't be it, I was just showing an example

[gitmate-bot]

This file contains unused source code.

Origin: PyUnusedCodeBear, Section: flakes.

The issue can be fixed by applying the following patch:

--- a/coalib/bearlib/languages/documentation/DocBaseClass.py
+++ b/coalib/bearlib/languages/documentation/DocBaseClass.py
@@ -1,4 +1,3 @@
-import re
 
 from coalib.bearlib.languages.documentation.DocstyleDefinition import (
     DocstyleDefinition)

[NiklasMM]

@damngamerz sorry this took so long. I'm beginning to like the overall design now 👍

Please have a look at my comments once more.

[NiklasMM]

Please mention the name of all these namedtuples in the docstring. Here for example A namedtuple "ClassPadding" consisting..., etc

[NiklasMM]

after missing

[NiklasMM]

after missing

[NiklasMM]

why is the default '', '' and not 0, 0?

[damngamerz]

because 0, 0 will mean that official defined padding is 0, 0. So better leave the defaults to be '', ''

[NiklasMM]

please explain the -2 in a comment

[NiklasMM]

You could remove these two lines by replacing all occurrences of top_padding with doc.top_padding and dito for bottom_padding. There's no real need for a temporary variable.

[NiklasMM]

This comment is inaccurate, you're just compiling regexes here

[NiklasMM]

def some_function():
  """
    documentation
  """
  class myPrivateClass:
     pass

This docstring would falsely be identified as a type class.

I suppose to mitigate this, you'll need to specify the position of the docstring in the coalang files, too.

[damngamerz]

I don't think so....we will require such a thing. Although I will add this in the test cases if it doesn't work we will have to think about the position.

[damngamerz]

Yes this fails. We have two ways to solve this
First one is a tricky but will work, I think we can fix this by little prioritizing the algorithm
something like:

1. we search the line closest to the docstring first. (whichever has least padding will be scanned by both the regexes first.)
2. If paddings are found to be equal scanning the line before starting marker with regexes first(i.e. with both function and class). Than last line will be scanned.
   This will support the generic nature.

Second introducing docstring_type_position.
If we found its written top then we only scan top lines of the docstring with regexes or vice-versa.

[damngamerz]

Im going with the second option looks much more future proof 👍

[NiklasMM]

introducing docstring_type_position is the correct solution. The other will not work, you'll only make it work for Python. Maybe docstring_position should suffice, I doubt that styles will position the docstring on different sides for different types.

[NiklasMM]

In your previous comment you said that docstyle_settings['class_padding'] is a string like "1, 0", if this is true, tuple("1, 0") will be ("1", ",", " ", "0"), which int will choke on. Rather use docstyle_settings['class_padding'].split(",")

Also I'm wondering why this isn't caught by tests.

[damngamerz]

Hmm now even I m wondering the same. You are correct about tuple('1,0') will be ("1", ",", " ", "0") but what I figured out is docstyle_settings is a Setting object. Now when I do tuple(docstyle_settings['class_padding'].value) gives me ("1", ",", " ", "0") but when I only do tuple(docstyle_settings['class_padding']) gives me ('1', '0')(some magic 😛 I m guessing it has to do something with the Setting object). So I think let it be that way.

[NiklasMM]

uh no, please investigate

[damngamerz]

I see why it's like that

coala/coalib/settings/Setting.py

Line 121 in 41e4752

class Setting(StringConverter):

Setting offers common data type conversions. It has list_delimeters for list/tuple conversion. That's why we are getting ('1', '0') while doing tuple(docstyle_settings['class_padding']) 🎉

[NiklasMM]

Wow, talk about overengineering.... anyway... it's ok then

[NiklasMM]

Why is this the desired behavior?

[damngamerz]

we have such a case in DocStyleBear.
I don't wanna break the existing tests 😉
https://github.com/coala/coala-bears/blob/master/tests/documentation/test_files/DocumentationStyleBear/bad_file2.py.test#L21

[NiklasMM]

how is such a case handled by documentation extraction?

[NiklasMM]

What I mean is, if you parse and re-assemble this docstring, what will happen?

[damngamerz]

oh yes! thats because....
The affected range is always taken from starting marker to the end marker(except in case of padding). DocumentationAPI doesn't care about whats after the end marker. Hence if its followed by inline comment...It's still is at the same position after the assembling.

[NiklasMM]

@damngamerz one last thing: The commit message needs an update: kind doesn't exist anymore

[damngamerz]

@NiklasMM Updated commit message 👍

[NiklasMM]

ack d53307c

[SanketDG]

ack 070a19e

[SanketDG]

@rultor merge

[rultor]

@rultor merge

@SanketDG OK, I'll try to merge now. You can check the progress of the merge here

[rultor]

@rultor merge

@SanketDG Done! FYI, the full log is here (took me 2min)