__all__ lists are incomplete #68071
Comments
|
Here is a (perhaps incomplete) list of documented names absent in __all__ lists of respective modules. Perhaps some of them (but not all) are worth to be added to __all__ lists. calendar.Calendar |
serhiy-storchaka
added
stdlib
|
http.client.HTTPMessage: See bpo-23439. There was resistance to adding this (and the status code constants), though IMO they should be added, since they are documented public APIs. http.server.test(): In bpo-23418, I consciously left this function out. It is only mentioned as a place to look for sample code, as far as I can tell. |
|
New changeset ebf3e6332a44 by Berker Peksag in branch 'default': |
|
May be makes sense to add a helper in test.support that implements a test similar to the one in bpo-23411, and add tests for __all__ in multiple modules. |
|
I working on these three. Changes would be the same as https://hg.python.org/cpython/rev/ebf3e6332a44/ for every module? |
|
Serhiy: Yes I was also thinking it might be time for a common helper function. Milap: I think changes like you mentioned (originally by me) would be fine. Another variation was done for bpo-10838: revision 10b0a8076be8, which expects each object that is not a module object (e.g. not from “import sys”), rather than expecting each object that is a function or class defined in the module. It might depend on the particular circumstance which technique is superior. |
|
New changeset 86fbe140e395 by Andrew Kuchling in branch '2.7': |
|
New changeset 717d87c13f0d by Andrew Kuchling in branch '3.4': |
|
I took care of the tarfile module. Added the following according to the first message: tarfile.CompressionError The following were included in __all__ that were not explicitly mentioned in the first message but were denoted as an exported function, exported class, or an exported error. tarfile.main This is my first patch so feedback is highly appreciated. |
|
Regarding tarfile: Two of the extra errors are documented, so I agree they should be added:
However I’m not so sure about main(), TarIter, and the HeaderError subclasses. They aren’t mentioned in the documentation. At least main() and TarIter are just implementation details I think. There are other documented items that should be added in my opinion. These would not be picked up by the proposed test, although they would be picked up by making the test like in revision 10b0a8076be8.
|
|
Thanks for the feedback. I was unsure how to proceed with the undocumented items that seemed to be categorized as exported. Thanks for catching ENCODING & *_FORMAT. |
|
Oh I see, TarIter is listed underneath a comment saying “Exported Classes”, and main() is listed underneath “exported functions”. If they are indeed meant to be exported, they should probably also be documented. Otherwise, maybe we can just add another comment clarifying that they are internal. In the latest patch, I think HeaderError should be added back to __all__; it is just its subclasses that are not documented. Also XHDTYPE is listed twice in the test case. If it were up to me, I would add the TarInfo.type constants (REGTYPE, AREGTYPE, LNKTYPE, SYMTYPE, DIRTYPE, FIFOTYPE, CONTTYPE, CHRTYPE, BLKTYPE, GNUTYPE_SPARSE). But I’m not sure if others would agree. |
|
Put HeaderError back in and removed the extra XHDTYPE. We can get more input on the type constants as well as the undocumented but exported items. Could just be cleared up with some edits to documentation. |
|
I took a stab at the calendar module. Found a few items in the documentation which weren't listed in the above list: LocaleTextCalendar, LocaleHTMLCalendar, and weekheader. I was curious though about week and prweek as month and prmonth are documented and exported should we add week and prweek to the export & docs? |
|
Woops just noticed above in the issue someone else picked up the Calendar __all__. I am genuinely sorry I didn't intend to duplicate the effort. |
|
Hi guys! Here is a patch for the fileinput module, with some names beyond fileinput.fileno: fileinput.hook_compressed, fileinput.hook_encoded as mentioned in the docs https://docs.python.org/3/library/fileinput.html This is my first patch as well, so feedback's appreciated! |
|
Hi! This is my first attempt at contributing so as always, feedback will be well appreciated. :) I meant to start small so I took a shot with csv module. In test, initial expected set contains QUOTE_* because they don't provide __module__ attribute, and __doc/version__ because they are already in csv.__all__ (should they?). I've made a few PEP-8-related fixes just around code I've touched (so they aren't completely unrelated). Is that ok? |
|
Reviews of the patches waiting here: tarfile, calendar (Joel): Look mainly good; I added minor suggestions about the test cases to Reitveld. fileinput (Mauro): Looks pretty good; one minor comment on Rietveld. csv (Jacek): Pretty good; couple minor suggestions. In a perfect world, I don’t think __doc/version__ should be there, but since they are already there it is probably safer to leave them. In general, I think style fixes in related code are okay; although in this case I have no problem with the original single blank lines. There is nothing seriously wrong with the patches so far. They could be committed, perhaps with a few of the tweaks I suggested. Summary of other things mentioned here left to do:
One outstanding question is what to do about module-level constants that are only briefly mentioned in the documentation. IMO they should be included in __all__. Examples: http.client statuses <https://docs.python.org/3.4/library/http.client.html#http.client.HTTPS_PORT\>, tarfile member types <https://docs.python.org/dev/library/tarfile.html#tarfile.TarInfo.type\>, calendar weekdays <https://docs.python.org/dev/library/calendar.html#calendar.setfirstweekday\>. Precedent of similar constants that are already included, in native Python modules: io SEEK_ constants; lzma FORMAT_, CHECK_, PRESET_ etc. |
|
Thank you for feedback, Martin. I've amended the the patch. Next, I've prepared some initial test.support.check__all__ helper, based on generalization of all previous patches. Its name/params' descriptions may be a bit rough - amendments/suggestions for such will be strongly appreciated: Issue23883_support_check__all__.patch I've added missing test.test_gettext.MiscTestCase, based on aforementioned check__all__ helper: Issue23883_test_gettext.patch I've also took the liberty of working on some more modules. These are: csv (using new helper), enum, ftplib, logging, optparse, pickletools, threading and wave: Issue23883_all.patch ftplib and threading have more functions (missing in their __all__ variables) that appear to be documented than mentioned in msg240217 - namely:
so I've added them as well. |
I've meant function and exceptions, of course. Sorry for the noise. |
|
I've added previously missing test and docs for test.support.check__all__ in Issue23883_support_check__all__.v2.patch . Awaiting review. :) |
|
To avoid the list of patches here getting out of control, I suggest opening a fresh issue for any new patches that aren’t a new version of the patches here. We can mark the new issue as a dependency of this one to keep track of it. Nice work with the check__all__() function. I left some comments on Reitveld. Also, it currently ignores items satisfying either of these checks:
The first is largely redundant with the second, because module objects don’t have a __module__ attribute. However I wonder if it would be better to drop the second check and just rely on the ModuleType check, making the test stricter. Or would this be too annoying in some cases (requiring a huge blacklist)? If so, maybe make the name_of_module checking optional. === Serhiy: ftplib.Error does not actually appear to be documented. Perhaps it should not be added to __all__ after all? (excuse the pun) |
Thank you! :)
Could you please elaborate on "making the test stricter"? I'd go with the first check + optional name_of_module. With second one alone, all freshly added test__all__ tests would need additional names in blacklists - not huge ones, but they would otherwise be unnecessary. I've amended the patches and I'm waiting for review. I've also thought of not only making name_of_module param optional, but to make it extra_names_of_module (so such param would be added to module.__name__ used in "getattr(module_object, '__module__', None) in name of module" check. It would account for less typing in general (module.__name__ occurs in almost all cases), but also less explicity. What do you think? |
Agree. The list is only cursorily filtered result of some one-liners and can contain false names. |
|
Adding new names to __all__ can have undesired effect and break user code (by hiding builtins as for tarfile.open). Perhaps not all documented names should be imported with "import *". In any case it is too late for 3.5. |
|
I think names should be in __all__ even if they shadow builtins, at least in a new feature release. There is plenty of precedent, e.g. asyncio.TimeoutError; reprlib.repr(); threading.enumerate(). Modules with open() in __all__ include aifc, bz2, codecs, dbm, dbm.dumb, gzip, lzma, os, shelve, wave and webbrowser. Plus, pydoc ignores things excluded from __all__. |
|
Jacek: If we used the ModuleType check, and somebody adds a module-level constant (like logging.CRITICAL = 50), the test will automatically detect if they forget to update __all__. That is what I meant by the test being stricter. But it looks like you went for the other option, which has its own relative advantages :) Will try to to a proper review later when I get a chance. |
|
Michael: According to bpo-18554, os.__all__ was fixed in 3.5. Can you confirm? It is working for me: Python 3.5.0 (default, Sep 20 2015, 11:28:25)
[GCC 5.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> "getcwd" in os.__all__
True |
|
@martin, my mistake. You're correct. I forgot I was using Python v3.4. |
|
Berker (or anyone else), do you have a preference on how we move forward? I am inclined to use Jacek’s function as it is. I think it is certainly an improvement over the current state. People can propose an alternative version of the function later if they want, though in my opinion the underlying problem is in the architecture of unittest’s assertion methods; see bpo-19645. |
|
I have added comments on Rietveld. Besides few stylistic nitpicks Issue23883_support_check__all__.v5.patch LGTM.
This is not new. There are other testing helpers in test.support that needs passing "self". If the helper is used many times in one test class, I prefer to make a method: class SomeTest(TestCase):
check_something = test.support.check_something
def test_foo():
self.check_something('foo')
def test_bar():
self.check_something('bar')But in this case I'm happy with the current API. |
|
Serhiy, thank you for the review. I've made proposed changes (along with rebasing Issue23883_all patch; Issue23883_test_gettext.v3.patch still applies cleanly). |
|
I like Martin's support.expected_module_api() suggestion in msg247167. I still find passing self to a function just to use assertCountEqual a bit weird, but I can live with that. |
|
The reason why I prefer the current API over my support.expected_module_api() idea is it requires the extra assertCountEqual() boilerplate at each call site. Jacek’s three patches look ready to me. I propose:
Another question that comes to mind: Should we add anything into What’s New, maybe warning of new symbols from “import *”? |
I personally find explicit assert* calls in a test case more readable(e.g. I don't need to check what the helper does every N month), but I guess that's another version of tabs vs. space debate :)
Your plan sounds good to me. Thanks!
I guess it wouldn't hurt to add a sentence :) |
|
New changeset f8fa7bc837a3 by Martin Panter in branch 'default': New changeset 78d67bdc1142 by Martin Panter in branch 'default': New changeset 25a7ceed79d1 by Martin Panter in branch 'default': |
|
Thankyou for sticking with this Jacek. I have committed your three patches. I reworded the documentation a little bit, mainly so it says that it looks for “public” names, rather than documented names, because it does not look at documentation at all. I also added a note to the Changes in the Python API section of What’s New. Next step I think is to finish off those patches for tarfile, calendar and fileinput. |
|
Martin, yay! :) And thank you for the documentation correction. Milap, Joel, Mauro, are you still interested in working on patches for calendar/tarfile/fileinput patches? I intend to finish them up if that's not the case. |
|
Yes, I'm, I have a commitment now but I'll submit a new version later today |
|
New version. |
|
issue23883_fileinput.v2.patch looks good to me. |
|
Week and no response, I'm posting updated patches for calendar and tarfile. |
|
I committed the last three patches to 3.6: 571632315c36: fileinput Please let me know if there are some outstanding patches here that I missed. Otherwise, I think we are up to step 6 in <https://bugs.python.org/issue23883#msg254581\>. |
|
New changeset 571632315c36 by Martin Panter in branch 'default': New changeset a2ffa9eedb1b by Martin Panter in branch 'default': New changeset 48090e08e367 by Martin Panter in branch 'default': New changeset a5d3ebb6ad2a by Martin Panter in branch 'default': |
|
New changeset 62e925be0aff by Serhiy Storchaka in branch 'default': |
|
Thanks for caring for this Martin.
I think yes. |
|
New changeset bd6127a6354f by Martin Panter in branch 'default': |
|
Serhiy: I already added a bullet point at <https://docs.python.org/3.6/whatsnew/3.6.html#changes-in-the-python-api\>. |
|
Per Martin's request, I've created a few new issues for next batch of module's __all__ list updates:
I've also looked at pydoc module, but I'm not sure what to do with it: |
|
I think pydoc could be left alone. The RST documentation doesn’t say anything about importing any functions from the module that I can see. I was surprised that it even defines __all__ = ["help"]. Perhaps pydoc.doc() was another false indication in Serhiy’s list. |
|
In this case I'm proposing a small patch just for testing pydoc module's __all__ list and left the decision to you, whether to apply it or not. :) Test doesn't use test.support.check__all__ (see msg266312) - blacklist would be huge and expected list, as you already pointed out, has only one value. |
|
New changeset a36c7f87eba9 by Martin Panter in branch 'default': |
|
Can this issue be closed now? |
No replies in two years, let's take that as a yes! (We can always re-open if needed :) |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: