New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Markdown for readmes #148

Closed
ewdurbin opened this Issue May 30, 2014 · 44 comments

Comments

Projects
None yet
5 participants
@ewdurbin
Member

ewdurbin commented May 30, 2014

Originally reported by: Hickford (Bitbucket: hickford, GitHub: hickford)


If you write your readme in Restructured Text, PyPI will render it on your project's page (eg. https://pypi.python.org/pypi/requests ). It should be possible to write your readme in Markdown too. Markdown is a pervasive format used on GitHub and StackOverflow. It shouldn't be necessary to learn a new markup language or rewrite your project's documentation to publish a Python package.

This is common frustration. The Python community suffers because it's a barrier to publishing packages and writing documentation . Please consider supporting Markdown!


Update 2014: Here's a pull request! https://bitbucket.org/pypa/pypi/pull-request/59/support-markdown-for-readmes-fixes-148/diff


@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Jun 3, 2014

Member

Original comment by Will McKenzie (Bitbucket: kmd_willmckenzie, GitHub: Unknown):


I'll second this! Much prefer Markdown syntax and it's a pain having to convert before uploading a package.

Member

ewdurbin commented Jun 3, 2014

Original comment by Will McKenzie (Bitbucket: kmd_willmckenzie, GitHub: Unknown):


I'll second this! Much prefer Markdown syntax and it's a pain having to convert before uploading a package.

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Jun 3, 2014

Member

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):


This is unlikely to happen in the near term. Warehouse (PyPI 2.0) is organized so that eventually we can support multiple markdown selections. The primary problem is we don't have any way to know if we should be rendering in markdown or rST. One way around that would be to render Markdown if rST fails, however some markdown is valid rST but it renders incorrectly in that case and looks really bad. It's also not a very particularly elegant solution especially if people want to do something else like asciidoc or some other formatting method. Likely the metadata 2.0 stuff will (eventually maybe?) support saying what markup your description is in and then we can render it using the appropriate renderer. It will also enable us to actually reject and upload if a package doesn't render correctly instead of just falling back to plaintext, because we'll know if it's supposed to be able to render or not.

Member

ewdurbin commented Jun 3, 2014

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):


This is unlikely to happen in the near term. Warehouse (PyPI 2.0) is organized so that eventually we can support multiple markdown selections. The primary problem is we don't have any way to know if we should be rendering in markdown or rST. One way around that would be to render Markdown if rST fails, however some markdown is valid rST but it renders incorrectly in that case and looks really bad. It's also not a very particularly elegant solution especially if people want to do something else like asciidoc or some other formatting method. Likely the metadata 2.0 stuff will (eventually maybe?) support saying what markup your description is in and then we can render it using the appropriate renderer. It will also enable us to actually reject and upload if a package doesn't render correctly instead of just falling back to plaintext, because we'll know if it's supposed to be able to render or not.

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Jun 3, 2014

Member

Original comment by Hickford (Bitbucket: hickford, GitHub: hickford):


Really? I understand PyPI's current algorithm to be

If there's a readme.rst, readme.txt or readme (no extension) file, render that as restructured text. Otherwise, nothing.

Here's a backwards compatible algorithm that would elate markdown authors:

If there's a readme.rst, readme.txt or readme (no extension) file render that as restructured text. Else if there's a readme.md or readme.markdown file render that as Markdown. Otherwise, nothing.

If you're concerned about projects with a readme.txt written in Markdown, they are already broken. This change wouldn't fix that, but it would help the majority of Markdown authors who use the .md or .markdown extensions. They do that because GitHub renders files with those extensions.

Member

ewdurbin commented Jun 3, 2014

Original comment by Hickford (Bitbucket: hickford, GitHub: hickford):


Really? I understand PyPI's current algorithm to be

If there's a readme.rst, readme.txt or readme (no extension) file, render that as restructured text. Otherwise, nothing.

Here's a backwards compatible algorithm that would elate markdown authors:

If there's a readme.rst, readme.txt or readme (no extension) file render that as restructured text. Else if there's a readme.md or readme.markdown file render that as Markdown. Otherwise, nothing.

If you're concerned about projects with a readme.txt written in Markdown, they are already broken. This change wouldn't fix that, but it would help the majority of Markdown authors who use the .md or .markdown extensions. They do that because GitHub renders files with those extensions.

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Jun 4, 2014

Member

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):


No, PyPI doesn't exactly do that. PyPI renders a long_description, if a long_description is not provided it will attempt to discover a README file and will insert that as the long_description.

Member

ewdurbin commented Jun 4, 2014

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):


No, PyPI doesn't exactly do that. PyPI renders a long_description, if a long_description is not provided it will attempt to discover a README file and will insert that as the long_description.

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Jun 4, 2014

Member

Original comment by Will McKenzie (Bitbucket: kmd_willmckenzie, GitHub: Unknown):


A potential interim solution (until the metadata stuff that will let a package maintainer specify the format) could be to have a developer leave the long_description field blank and have a README.(md|markdown) file to be rendered in it's stead? I know with my packages I'm basically just reading the README.txt file and assigning it to the long description field (I didn't know how the algorithm worked).

Member

ewdurbin commented Jun 4, 2014

Original comment by Will McKenzie (Bitbucket: kmd_willmckenzie, GitHub: Unknown):


A potential interim solution (until the metadata stuff that will let a package maintainer specify the format) could be to have a developer leave the long_description field blank and have a README.(md|markdown) file to be rendered in it's stead? I know with my packages I'm basically just reading the README.txt file and assigning it to the long description field (I didn't know how the algorithm worked).

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Jun 5, 2014

Member

Original comment by Hickford (Bitbucket: hickford, GitHub: hickford):


I had a go! Here's a pull request https://bitbucket.org/pypa/pypi/pull-request/33/support-markdown-for-readmes/diff

Member

ewdurbin commented Jun 5, 2014

Original comment by Hickford (Bitbucket: hickford, GitHub: hickford):


I had a go! Here's a pull request https://bitbucket.org/pypa/pypi/pull-request/33/support-markdown-for-readmes/diff

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Nov 27, 2014

Member

Original comment by Marc Abramowitz (Bitbucket: msabramo, GitHub: msabramo):


I like Will's idea. Many, many Python projects have boilerplate in setup.py to set long_description to the contents of their README. That's because people want to have the best chance of their description getting picked up but almost everyone prefers editing a description in a separate file over something embedded in setup.py, plus the file extension tells you what format it's in.

I think it would be great if PyPI/warehouse scanned for a README.rst, README.md, README.markdown and used that as the priority for the description to show. Perhaps setuptools could be modified to warn you in the unlikely event that your long_description has different data than your README (@jaraco: What do you think?).

I think in the next version of the metadata standard I would drop long_description and instead offer something like long_description_filename.

Maybe not a bad idea to add long_description_filename to setuptools now. Then package maintainers could start setting that and PyPI/Warehouse could use that in preference to long_description. That would let the server instantly know the format language from the extension and package maintainers could set it and remove lines from their setup.py to populate long_description from a file.

Member

ewdurbin commented Nov 27, 2014

Original comment by Marc Abramowitz (Bitbucket: msabramo, GitHub: msabramo):


I like Will's idea. Many, many Python projects have boilerplate in setup.py to set long_description to the contents of their README. That's because people want to have the best chance of their description getting picked up but almost everyone prefers editing a description in a separate file over something embedded in setup.py, plus the file extension tells you what format it's in.

I think it would be great if PyPI/warehouse scanned for a README.rst, README.md, README.markdown and used that as the priority for the description to show. Perhaps setuptools could be modified to warn you in the unlikely event that your long_description has different data than your README (@jaraco: What do you think?).

I think in the next version of the metadata standard I would drop long_description and instead offer something like long_description_filename.

Maybe not a bad idea to add long_description_filename to setuptools now. Then package maintainers could start setting that and PyPI/Warehouse could use that in preference to long_description. That would let the server instantly know the format language from the extension and package maintainers could set it and remove lines from their setup.py to populate long_description from a file.

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Nov 27, 2014

Member

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):


@msabramo I'm not sure your suggestion makes sense in the general case. What you and Will are suggesting is to create a new field in the metadata that references content outside the metadata. In particular, long_description_filename references a file that exists in one of the uploaded distribution packages, but which one? What if the distribution package is a wheel or egg or Windows installer? What if there are multiple distributions (Python version specific)? What if the metadata has been uploaded (e.g. through the register command) but no distribution has been uploaded? I see value in having the long description inline with the metadata in all cases, because that means that all metadata is represented in one entity and doesn't require steps to assemble it (and the complication that entails).

Member

ewdurbin commented Nov 27, 2014

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):


@msabramo I'm not sure your suggestion makes sense in the general case. What you and Will are suggesting is to create a new field in the metadata that references content outside the metadata. In particular, long_description_filename references a file that exists in one of the uploaded distribution packages, but which one? What if the distribution package is a wheel or egg or Windows installer? What if there are multiple distributions (Python version specific)? What if the metadata has been uploaded (e.g. through the register command) but no distribution has been uploaded? I see value in having the long description inline with the metadata in all cases, because that means that all metadata is represented in one entity and doesn't require steps to assemble it (and the complication that entails).

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Nov 27, 2014

Member

Original comment by Krisztian Fekete (Bitbucket: krisztianfekete, GitHub: Unknown):


What about a new field: long_description_format defaulting to reST for backward compatibility, but enabling us to have markdown formatted descriptions?

Lifting a README file into the manifest is a different feature, which can be solved with tooling as demonstrated by pbr.

Member

ewdurbin commented Nov 27, 2014

Original comment by Krisztian Fekete (Bitbucket: krisztianfekete, GitHub: Unknown):


What about a new field: long_description_format defaulting to reST for backward compatibility, but enabling us to have markdown formatted descriptions?

Lifting a README file into the manifest is a different feature, which can be solved with tooling as demonstrated by pbr.

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Nov 27, 2014

Member

Original comment by Marc Abramowitz (Bitbucket: msabramo, GitHub: msabramo):


@jaraco: Those are good points - I hadn't considered that there could be multiple distributions or even no distribution if the user registered but did not upload. Maybe the idea from @krisztianfekete is a better one?

Member

ewdurbin commented Nov 27, 2014

Original comment by Marc Abramowitz (Bitbucket: msabramo, GitHub: msabramo):


@jaraco: Those are good points - I hadn't considered that there could be multiple distributions or even no distribution if the user registered but did not upload. Maybe the idea from @krisztianfekete is a better one?

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Nov 27, 2014

Member

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):


I have no objection to supporting markdown or even moving toward preferring markdown. This poses a larger question of what should the metadata reflect to support this ability? Also, does the tool chain that renders documentation support markdown (namely Sphinx or Warehouse)? I'd recommend getting some feedback from pypa-dev on a change of this scope.

Member

ewdurbin commented Nov 27, 2014

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):


I have no objection to supporting markdown or even moving toward preferring markdown. This poses a larger question of what should the metadata reflect to support this ability? Also, does the tool chain that renders documentation support markdown (namely Sphinx or Warehouse)? I'd recommend getting some feedback from pypa-dev on a change of this scope.

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Nov 27, 2014

Member

Original comment by Marc Abramowitz (Bitbucket: msabramo, GitHub: msabramo):


@hickford's patch would add Markdown to legacy PyPI. I'm thinking it could be tweaked a bit to look for a long_description_format property in the metadata and a similar change could be made to Warehouse (GitHub).

Then we could make a setuptools extension to add that property to package metadata (use an extension so we don't have to modify setuptools unless it proves to be wildly popular).

Member

ewdurbin commented Nov 27, 2014

Original comment by Marc Abramowitz (Bitbucket: msabramo, GitHub: msabramo):


@hickford's patch would add Markdown to legacy PyPI. I'm thinking it could be tweaked a bit to look for a long_description_format property in the metadata and a similar change could be made to Warehouse (GitHub).

Then we could make a setuptools extension to add that property to package metadata (use an extension so we don't have to modify setuptools unless it proves to be wildly popular).

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Nov 28, 2014

Member

Original comment by illume NA (Bitbucket: illume, GitHub: illume):


Added an issue to devpi https://bitbucket.org/hpk42/devpi/issue/193/support-markdown-for-readmes

Member

ewdurbin commented Nov 28, 2014

Original comment by illume NA (Bitbucket: illume, GitHub: illume):


Added an issue to devpi https://bitbucket.org/hpk42/devpi/issue/193/support-markdown-for-readmes

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Nov 28, 2014

Member

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):


I'm slighly -1 on adding another metadata field to specify the content type of an existing metadata field (i.e. long_description/long_description_format). If we did adopt that, I would recommend using MIME types (so something like "text/x-markdown; charset=utf-8" for markdown) to encourage alignment with other standards.

Rather than storing the metadata for the long description in another field, perhaps the format could be inferred by a heuristic and specified for cases where the heuristic fails using out-of-band features of the format (i.e. comments). For example, the following two snippets could represent a way to specify the content type of the document:

.. Content-Type: text/x-rst
[//]: # Content-Type: text/x-markdown

In this way, the long_description field becomes content-agnostic, but the tools now can infer the content type and users would have a means to force a content type.

This approach would not require any changes to tooling except for those that render package metadata.

Member

ewdurbin commented Nov 28, 2014

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):


I'm slighly -1 on adding another metadata field to specify the content type of an existing metadata field (i.e. long_description/long_description_format). If we did adopt that, I would recommend using MIME types (so something like "text/x-markdown; charset=utf-8" for markdown) to encourage alignment with other standards.

Rather than storing the metadata for the long description in another field, perhaps the format could be inferred by a heuristic and specified for cases where the heuristic fails using out-of-band features of the format (i.e. comments). For example, the following two snippets could represent a way to specify the content type of the document:

.. Content-Type: text/x-rst
[//]: # Content-Type: text/x-markdown

In this way, the long_description field becomes content-agnostic, but the tools now can infer the content type and users would have a means to force a content type.

This approach would not require any changes to tooling except for those that render package metadata.

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Nov 28, 2014

Member

Original comment by Marc Abramowitz (Bitbucket: msabramo, GitHub: msabramo):


@jaraco: Very interesting ideas to use MIME types and to attempt to deduce the format without adding metadata. @hickford: Want to update your PR?

Member

ewdurbin commented Nov 28, 2014

Original comment by Marc Abramowitz (Bitbucket: msabramo, GitHub: msabramo):


@jaraco: Very interesting ideas to use MIME types and to attempt to deduce the format without adding metadata. @hickford: Want to update your PR?

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Dec 8, 2014

Member

Original comment by Richard Jones (Bitbucket: r1chardj0n3s, GitHub: r1chardj0n3s):


Merged in hickford/pypi/markdown-readme (pull request #33)

Support Markdown for readmes (fixes #148)

Member

ewdurbin commented Dec 8, 2014

Original comment by Richard Jones (Bitbucket: r1chardj0n3s, GitHub: r1chardj0n3s):


Merged in hickford/pypi/markdown-readme (pull request #33)

Support Markdown for readmes (fixes #148)

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Dec 8, 2014

Member

Original comment by Richard Jones (Bitbucket: r1chardj0n3s, GitHub: r1chardj0n3s):


Merged in hickford/pypi/markdown-readme (pull request #33)

Support Markdown for readmes (fixes #148)

Member

ewdurbin commented Dec 8, 2014

Original comment by Richard Jones (Bitbucket: r1chardj0n3s, GitHub: r1chardj0n3s):


Merged in hickford/pypi/markdown-readme (pull request #33)

Support Markdown for readmes (fixes #148)

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Dec 8, 2014

Member

Original comment by Eugene Yunak (Bitbucket: yunake, GitHub: yunake):


@r1chardj0n3s, can you maybe reopen this issue since you reverted the merge?

Member

ewdurbin commented Dec 8, 2014

Original comment by Eugene Yunak (Bitbucket: yunake, GitHub: yunake):


@r1chardj0n3s, can you maybe reopen this issue since you reverted the merge?

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Dec 8, 2014

Member

Original comment by Richard Jones (Bitbucket: r1chardj0n3s, GitHub: r1chardj0n3s):


Needs more thought on the security front.

Member

ewdurbin commented Dec 8, 2014

Original comment by Richard Jones (Bitbucket: r1chardj0n3s, GitHub: r1chardj0n3s):


Needs more thought on the security front.

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Dec 8, 2014

Member

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):


I'm of the opinion that a fallback on Markdown isn't very good.

  • Some Markdown is also valid ReST, but the symbols mean different things, this means that it's not true that a fallback enables proper Markdown support, it only enables markdown support if that markdown happens to not be renderable as ReST. I think depending on the fact that a particular chunk of text doesn't happen to also be valid for ReST is a pretty poor UX.
  • It doesn't scale well beyond Markdown. It might be said that why would anyone choose something else over Markdown and ReST but it seems to me that adopting a solution which makes it harder to add more things in the future is a bad long term solution.
  • It relies on the fact that we're shoving rendered HTML into the database, however Warehouse does not use that HTML and I want to get rid of it eventually. Relying on shoving html in the database also means that things that consume the data from PyPI cannot determine what syntax to render a description with, without downloading the project and trying to determine how PyPI got the long_description and if it's from a file that ends with .md or .rst.

Overall I'd really like not to add more implementation defined stuff to the packaging toolchain. The processes for doing things should be documented in a PEP and someone should be able to replace each part of the toolchain by reading those PEPs and implementing them. It shouldn't require having to inspect the various implementations to figure things out.

Member

ewdurbin commented Dec 8, 2014

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):


I'm of the opinion that a fallback on Markdown isn't very good.

  • Some Markdown is also valid ReST, but the symbols mean different things, this means that it's not true that a fallback enables proper Markdown support, it only enables markdown support if that markdown happens to not be renderable as ReST. I think depending on the fact that a particular chunk of text doesn't happen to also be valid for ReST is a pretty poor UX.
  • It doesn't scale well beyond Markdown. It might be said that why would anyone choose something else over Markdown and ReST but it seems to me that adopting a solution which makes it harder to add more things in the future is a bad long term solution.
  • It relies on the fact that we're shoving rendered HTML into the database, however Warehouse does not use that HTML and I want to get rid of it eventually. Relying on shoving html in the database also means that things that consume the data from PyPI cannot determine what syntax to render a description with, without downloading the project and trying to determine how PyPI got the long_description and if it's from a file that ends with .md or .rst.

Overall I'd really like not to add more implementation defined stuff to the packaging toolchain. The processes for doing things should be documented in a PEP and someone should be able to replace each part of the toolchain by reading those PEPs and implementing them. It shouldn't require having to inspect the various implementations to figure things out.

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Dec 9, 2014

Member

Original comment by Marc Abramowitz (Bitbucket: msabramo, GitHub: msabramo):


@dstufft: Are you more in favor of new metadata like long_description_content_type with a MIME content type? Or something else?

Member

ewdurbin commented Dec 9, 2014

Original comment by Marc Abramowitz (Bitbucket: msabramo, GitHub: msabramo):


@dstufft: Are you more in favor of new metadata like long_description_content_type with a MIME content type? Or something else?

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Dec 11, 2014

Member

Original comment by digitalvapor (Bitbucket: digitalvapor, GitHub: digitalvapor):


Markdown support would be very appreciated! thanks

Member

ewdurbin commented Dec 11, 2014

Original comment by digitalvapor (Bitbucket: digitalvapor, GitHub: digitalvapor):


Markdown support would be very appreciated! thanks

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Dec 11, 2014

Member

Original comment by Will Mayner (Bitbucket: wmayner, GitHub: wmayner):


Big +1. This would be a huge boon to many!

Member

ewdurbin commented Dec 11, 2014

Original comment by Will Mayner (Bitbucket: wmayner, GitHub: wmayner):


Big +1. This would be a huge boon to many!

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Dec 15, 2014

Member

Original comment by Femto Trader (Bitbucket: FemtoTrader, GitHub: FemtoTrader):


+1 I'm also looking for Markdown support!

Member

ewdurbin commented Dec 15, 2014

Original comment by Femto Trader (Bitbucket: FemtoTrader, GitHub: FemtoTrader):


+1 I'm also looking for Markdown support!

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Dec 19, 2014

Member

Original comment by Marc Abramowitz (Bitbucket: msabramo, GitHub: msabramo):


For folks looking for Markdown support, have a look at https://pypi.python.org/pypi/setuptools-markdown which I uploaded to PyPI last night.

This is very, very experimental. Make sure to test thoroughly before you put a package on PyPI that uses it. E.g.: I wouldn't want for folks who install your packages to have to install setuptools-markdown and pandoc; I think it should only be required for the package maintainer. But please check that. You can file GitHub issues, but PRs are much preferred!

Member

ewdurbin commented Dec 19, 2014

Original comment by Marc Abramowitz (Bitbucket: msabramo, GitHub: msabramo):


For folks looking for Markdown support, have a look at https://pypi.python.org/pypi/setuptools-markdown which I uploaded to PyPI last night.

This is very, very experimental. Make sure to test thoroughly before you put a package on PyPI that uses it. E.g.: I wouldn't want for folks who install your packages to have to install setuptools-markdown and pandoc; I think it should only be required for the package maintainer. But please check that. You can file GitHub issues, but PRs are much preferred!

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Mar 9, 2015

Member

Original comment by agh1973 (Bitbucket: agh1973, GitHub: Unknown):


+1 coming from http://stackoverflow.com/questions/4384796/readme-extension-for-python-projects I'll give the setuptools-markdown stuff a try.

Member

ewdurbin commented Mar 9, 2015

Original comment by agh1973 (Bitbucket: agh1973, GitHub: Unknown):


+1 coming from http://stackoverflow.com/questions/4384796/readme-extension-for-python-projects I'll give the setuptools-markdown stuff a try.

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Mar 10, 2015

Member

Original comment by Skuli Arnlaugsson (Bitbucket: arnlaugsson, GitHub: arnlaugsson):


This would be a lovely addition, Markdown is a much nicer format imo.

Member

ewdurbin commented Mar 10, 2015

Original comment by Skuli Arnlaugsson (Bitbucket: arnlaugsson, GitHub: arnlaugsson):


This would be a lovely addition, Markdown is a much nicer format imo.

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Mar 24, 2015

Member

Original comment by laike9m (Bitbucket: laike9m, GitHub: laike9m):


It's hard to imagine in 2015 Pypi still doesn't support markdown. Really need this!!

Member

ewdurbin commented Mar 24, 2015

Original comment by laike9m (Bitbucket: laike9m, GitHub: laike9m):


It's hard to imagine in 2015 Pypi still doesn't support markdown. Really need this!!

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Mar 26, 2015

Member

Original comment by Fletcher Tomalty (Bitbucket: fletom, GitHub: fletom):


+1 Markdown is standard for READMEs and should be supported by PyPI.

Member

ewdurbin commented Mar 26, 2015

Original comment by Fletcher Tomalty (Bitbucket: fletom, GitHub: fletom):


+1 Markdown is standard for READMEs and should be supported by PyPI.

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Apr 3, 2015

Member

Original comment by Tim Santor (Bitbucket: tsantor, GitHub: tsantor):


Support Markdown for READMEs please!

Member

ewdurbin commented Apr 3, 2015

Original comment by Tim Santor (Bitbucket: tsantor, GitHub: tsantor):


Support Markdown for READMEs please!

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Apr 6, 2015

Member

Original comment by Christoph Körner (Bitbucket: chaosmail, GitHub: chaosmail):


+1 i would love to see markdown being supported by pypi

Member

ewdurbin commented Apr 6, 2015

Original comment by Christoph Körner (Bitbucket: chaosmail, GitHub: chaosmail):


+1 i would love to see markdown being supported by pypi

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Apr 6, 2015

Member

Original comment by Naoki INADA (Bitbucket: methane, GitHub: methane):


Use vote feature, not +1 comment.

Member

ewdurbin commented Apr 6, 2015

Original comment by Naoki INADA (Bitbucket: methane, GitHub: methane):


Use vote feature, not +1 comment.

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin May 3, 2015

Member

Original comment by anatoly techtonik (Bitbucket: techtonik, GitHub: techtonik):


This was an official position on any features a year ago:

Please stop submitting pull requests. Development on the existing codebase is halted except for
critical fixes or security issues. You are making extra work for people on this list and it will not be
tolerated. Please consider this your final warning.

http://comments.gmane.org/gmane.comp.python.distutils.devel/20041

Member

ewdurbin commented May 3, 2015

Original comment by anatoly techtonik (Bitbucket: techtonik, GitHub: techtonik):


This was an official position on any features a year ago:

Please stop submitting pull requests. Development on the existing codebase is halted except for
critical fixes or security issues. You are making extra work for people on this list and it will not be
tolerated. Please consider this your final warning.

http://comments.gmane.org/gmane.comp.python.distutils.devel/20041

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin May 3, 2015

Member

Original comment by Hickford (Bitbucket: hickford, GitHub: hickford):


That's unwelcoming :( I am not going to read any further or say any more, because quarrels between people who should be working together make me sad.

Happily I've been fortunate in open source, everyone has been most helpful to me.

Whether Markdown support is added to current PyPI or the new site (https://warehouse.python.org/ ), Markdown support is an important issue to many people. I'm sure we can work it out eventually.

Member

ewdurbin commented May 3, 2015

Original comment by Hickford (Bitbucket: hickford, GitHub: hickford):


That's unwelcoming :( I am not going to read any further or say any more, because quarrels between people who should be working together make me sad.

Happily I've been fortunate in open source, everyone has been most helpful to me.

Whether Markdown support is added to current PyPI or the new site (https://warehouse.python.org/ ), Markdown support is an important issue to many people. I'm sure we can work it out eventually.

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin May 3, 2015

Member

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):


The largest blocker is figuring out how a setup.py should indicate that a project should use Markdown. This is best done on distutils-sig.

Member

ewdurbin commented May 3, 2015

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):


The largest blocker is figuring out how a setup.py should indicate that a project should use Markdown. This is best done on distutils-sig.

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin May 3, 2015

Member

Original comment by Hickford (Bitbucket: hickford, GitHub: hickford):


Hi Donald. You are set that readme and format should be specified explicitly in setup.py ? That would work. I would also be happy with GitHub-style autodiscovery (use readme.rst, readme.md, or readme.txt).

Member

ewdurbin commented May 3, 2015

Original comment by Hickford (Bitbucket: hickford, GitHub: hickford):


Hi Donald. You are set that readme and format should be specified explicitly in setup.py ? That would work. I would also be happy with GitHub-style autodiscovery (use readme.rst, readme.md, or readme.txt).

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin May 3, 2015

Member

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):


There's no way to do auto discovery like Github does because we're not reading from a file (in some cases we are, but not generally), the description gets passed into the long_description field in the setup.py as a string and that gets sent as a POST argument when the file gets uploaded to PyPI. As a way of trying to find something PyPI will also, in the absence of that field, look for a README file in the archive but I personally feel that shouldn't be encouraged as it lives outside the metadata and instead we should focus on a reasonable way to represent it within the metadata.

Member

ewdurbin commented May 3, 2015

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):


There's no way to do auto discovery like Github does because we're not reading from a file (in some cases we are, but not generally), the description gets passed into the long_description field in the setup.py as a string and that gets sent as a POST argument when the file gets uploaded to PyPI. As a way of trying to find something PyPI will also, in the absence of that field, look for a README file in the archive but I personally feel that shouldn't be encouraged as it lives outside the metadata and instead we should focus on a reasonable way to represent it within the metadata.

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin May 3, 2015

Member

Original comment by anatoly techtonik (Bitbucket: techtonik, GitHub: techtonik):


setup.py must die, also because it is impossible to parse correctly in all cases. The README.txt or README,md is already in the .zip archive. No need to reference it. The problem with PyPI is that it tries to provide an online editor for the metadata and mixes metadata with data. The README.txt content is pasted into PKG-INFO file, but it is a hack.

The good way would be to render README file from archive if the Description field in PKG-INFO (long_description in setup.py) is absent. The context can be well detected based on extension.

Member

ewdurbin commented May 3, 2015

Original comment by anatoly techtonik (Bitbucket: techtonik, GitHub: techtonik):


setup.py must die, also because it is impossible to parse correctly in all cases. The README.txt or README,md is already in the .zip archive. No need to reference it. The problem with PyPI is that it tries to provide an online editor for the metadata and mixes metadata with data. The README.txt content is pasted into PKG-INFO file, but it is a hack.

The good way would be to render README file from archive if the Description field in PKG-INFO (long_description in setup.py) is absent. The context can be well detected based on extension.

@ewdurbin

This comment has been minimized.

Show comment
Hide comment
@ewdurbin

ewdurbin Dec 8, 2015

Member

Original comment by Nicholas Chammas (Bitbucket: nickchammas, GitHub: Unknown):


For people looking for an interim solution, you can use pandoc to convert your long_description from Markdown to reStructuredText as part of setup.py.

Here's an example:

import setuptools
import pypandoc

long_description = pypandoc.convert('README.md', 'rst')

setuptools.setup(
    name='example',
    version=find_version('example.py'),
    long_description=long_description,
    ...
)

It's straightforward and should work for most projects. You can add a fallback like in the Stack Overflow answer I linked to so that people running python setup.py without pandoc installed will get the plain Markdown README.

Member

ewdurbin commented Dec 8, 2015

Original comment by Nicholas Chammas (Bitbucket: nickchammas, GitHub: Unknown):


For people looking for an interim solution, you can use pandoc to convert your long_description from Markdown to reStructuredText as part of setup.py.

Here's an example:

import setuptools
import pypandoc

long_description = pypandoc.convert('README.md', 'rst')

setuptools.setup(
    name='example',
    version=find_version('example.py'),
    long_description=long_description,
    ...
)

It's straightforward and should work for most projects. You can add a fallback like in the Stack Overflow answer I linked to so that people running python setup.py without pandoc installed will get the plain Markdown README.

@femtotrader

This comment has been minimized.

Show comment
Hide comment
@femtotrader

femtotrader Jun 22, 2016

Some people don't like this solution
See discussion here femtotrader/pandas_talib#8

Pinging also :
@OiNutter
@nchammas
@e3krisztian

femtotrader commented Jun 22, 2016

Some people don't like this solution
See discussion here femtotrader/pandas_talib#8

Pinging also :
@OiNutter
@nchammas
@e3krisztian

@nchammas

This comment has been minimized.

Show comment
Hide comment
@nchammas

nchammas Jun 22, 2016

@femtotrader - If by "this solution" you are referring to my suggestion just above, I don't see how it is a problem.

The users in the discussion you linked to are complaining that they have to install Pandoc just to install your library. However, at the end of my comment I noted:

It's straightforward and should work for most projects. You can add a fallback like in the Stack Overflow answer I linked to so that people running python setup.py without pandoc installed will get the plain Markdown README.

In other words, only the maintainer who is uploading a package to PyPI will need Pandoc installed.

Perhaps I should have included that in my code sample, rather than just referring to it on Stack Overflow via a link. But to spell things out, if you have something like this in your setup.py:

try:
    import pypandoc
    long_description = pypandoc.convert('README.md', 'rst')
except ImportError:
    long_description = open('README.md').read()

And then you have Pandoc listed as a separate requirement only for maintainers, perhaps in a maintainer.pip file, then you get what you want. Users aren't burdened installing a package they don't need, and maintainers get their README in RST without having to learn that format just for PyPI.

That's how I do it in my project:

It's not the solution we all ultimately want, but I think it's good enough for the time being.

nchammas commented Jun 22, 2016

@femtotrader - If by "this solution" you are referring to my suggestion just above, I don't see how it is a problem.

The users in the discussion you linked to are complaining that they have to install Pandoc just to install your library. However, at the end of my comment I noted:

It's straightforward and should work for most projects. You can add a fallback like in the Stack Overflow answer I linked to so that people running python setup.py without pandoc installed will get the plain Markdown README.

In other words, only the maintainer who is uploading a package to PyPI will need Pandoc installed.

Perhaps I should have included that in my code sample, rather than just referring to it on Stack Overflow via a link. But to spell things out, if you have something like this in your setup.py:

try:
    import pypandoc
    long_description = pypandoc.convert('README.md', 'rst')
except ImportError:
    long_description = open('README.md').read()

And then you have Pandoc listed as a separate requirement only for maintainers, perhaps in a maintainer.pip file, then you get what you want. Users aren't burdened installing a package they don't need, and maintainers get their README in RST without having to learn that format just for PyPI.

That's how I do it in my project:

It's not the solution we all ultimately want, but I think it's good enough for the time being.

@femtotrader

This comment has been minimized.

Show comment
Hide comment
@femtotrader

femtotrader Jun 22, 2016

I understand this correctly. Sorry for being not clear enough...
I really think that solution is to render correctly Markdown on server side.

femtotrader commented Jun 22, 2016

I understand this correctly. Sorry for being not clear enough...
I really think that solution is to render correctly Markdown on server side.

cben added a commit to cben/manageiq-api-client-python that referenced this issue Nov 7, 2016

Switch to reStructuredText README
PyPI doesn't render markdown, it parses README.md assuming reST rules:
pypa/pypi-legacy#148
which currently causes joined lines on
https://pypi.python.org/pypi/manageiq-api-client-python/

Using README.rst with reST syntax is the simplest workaround.
@mec07

This comment has been minimized.

Show comment
Hide comment
@mec07

mec07 Jun 7, 2017

I also think pypi should just support markdown.

mec07 commented Jun 7, 2017

I also think pypi should just support markdown.

@pypa pypa locked and limited conversation to collaborators Jun 8, 2017

@berkerpeksag

This comment has been minimized.

Show comment
Hide comment
@berkerpeksag

berkerpeksag Jun 8, 2017

Member

Since this is a new feature and implementing it is not easy as calling markdown.markdown(long_description) in legacy PyPI and there is already an open issue in Warehouse's issue tracker, I'm going to close this one. Thank you all for the discussion!

Member

berkerpeksag commented Jun 8, 2017

Since this is a new feature and implementing it is not easy as calling markdown.markdown(long_description) in legacy PyPI and there is already an open issue in Warehouse's issue tracker, I'm going to close this one. Thank you all for the discussion!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.