Add Markdown support #3

msabramo · 2014-12-09T19:14:21Z

Fixes: GH-1

$ tox -e py27,py33,py34
GLOB sdist-make: /Users/marca/dev/git-repos/readme/setup.py
py27 inst-nodeps: /Users/marca/dev/git-repos/readme/.tox/dist/readme-0.3.0.zip
py27 runtests: PYTHONHASHSEED='2116262944'
py27 runtests: commands[0] | py.test tests/
============================================================================= test session starts ==============================================================================
platform darwin -- Python 2.7.6 -- py-1.4.26 -- pytest-2.6.4
plugins: capturelog, cov
collected 7 items

tests/test_markdown.py .......

=========================================================================== 7 passed in 0.18 seconds ===========================================================================
py33 inst-nodeps: /Users/marca/dev/git-repos/readme/.tox/dist/readme-0.3.0.zip
py33 runtests: PYTHONHASHSEED='2116262944'
py33 runtests: commands[0] | py.test tests/
============================================================================= test session starts ==============================================================================
platform darwin -- Python 3.3.3 -- py-1.4.26 -- pytest-2.6.4
plugins: capturelog, cov
collected 7 items

tests/test_markdown.py .......

=========================================================================== 7 passed in 0.22 seconds ===========================================================================
py34 inst-nodeps: /Users/marca/dev/git-repos/readme/.tox/dist/readme-0.3.0.zip
py34 runtests: PYTHONHASHSEED='2116262944'
py34 runtests: commands[0] | py.test tests/
============================================================================= test session starts ==============================================================================
platform darwin -- Python 3.4.0 -- py-1.4.26 -- pytest-2.6.4
plugins: capturelog, cov
collected 7 items

tests/test_markdown.py .......

=========================================================================== 7 passed in 0.20 seconds ===========================================================================
___________________________________________________________________________________ summary ____________________________________________________________________________________
  py27: commands succeeded
  py33: commands succeeded
  py34: commands succeeded
  congratulations :)

Passing Travis CI build: https://travis-ci.org/msabramo/readme/builds/43547312

Cc: @dstufft, @r1chardj0n3s, @hickford, @methane

msabramo · 2014-12-10T18:31:27Z

It would be nice to change the tests to use .md and .html files in tests/fixtures, like I did with #4.

I'll try to find some time to do that.

msabramo · 2014-12-10T18:55:42Z

It would be nice to change the tests to use .md and .html files in tests/fixtures, like I did with #4.

Done with 2643da2.

methane · 2014-12-11T04:57:27Z

Most projects using Github Flavored Markdown.
To improve compatibility, could you add some extensions?

msabramo · 2014-12-13T19:48:17Z

@methane: Good point about GitHub Markdown. I'll have to add tests for that and then try out the extensions you mentioned. Feel free to throw it in there if you get a chance.

methane · 2014-12-17T12:53:42Z

By the way, how PyPI can determine that which format long_description written?
Markdown is very loose format. Most reST and asciidoc may be valid markdown.

Add long_description_format option to setuptools
Add magic line like filtype: (rest|markdown)
Heuristic approach: If it starts with #, it may be markdown.

msabramo · 2014-12-17T15:34:48Z

@methane: Good point. You might be interested in the discussion around this at https://bitbucket.org/pypa/pypi/issue/148/support-markdown-for-readmes

msabramo · 2015-01-24T22:47:57Z

@dstufft: If you have comments, I can update this.

dstufft · 2015-01-24T22:49:19Z

I don't think it makes sense to merge any Markdown support until we have an answer for how PyPI (or any other tool) is supposed to know what format it's supposed to use.

berkerpeksag · 2015-01-26T04:31:51Z

tests/test_markdown.py

+
+def read(fn):
+    path = os.path.join(os.path.dirname(__file__), 'fixtures', 'markdown', fn)
+    with codecs.open(path, encoding='utf-8') as f:


Use io.open instead. codecs.open will be deprecated: http://bugs.python.org/issue8796

Updated. Thanks!

bukzor · 2015-01-30T00:34:13Z

I don't think we should block this merge on "how PyPI (or any other tool) is supposed to know what format it's supposed to use"; we can merge this without regressing on that front, and there are a few obvious ways to improve it later.

msabramo · 2015-01-30T17:51:38Z

@bukzor: I agree. I think that the rendering of Markdown vs. the decision of what markup language to render are orthogonal issues.

It's possible that pypa/README could attempt to guess the format from a blob of text using some heuristics, but I think that the more robust solution is to have something at a higher level that guesses the format based on filename extension or even better, a new piece of distribution metadata -- e.g.: long_description_content_type='text/x-markdown; charset=UTF-8'.

I don't think figuring out the mechanism for choosing the renderer needs to block making progress on making Markdown rendering work nicely (which I think largely means having parity with GitHub-flavored Markdown).

sudarkoff · 2015-01-30T17:53:06Z

I completely agree with @msabramo and @bukzor

dstufft · 2015-01-30T18:07:59Z

What's the point of being able to render Markdown without being able to determine if we should render as Markdown? Without some way to determine what we should use this code is just going to sit there doing nothing.

msabramo · 2015-01-30T19:00:34Z

Well yeah, I get that the Markdown stuff won't be used until we figure out how. But I think we should try to work on both.

One could make the reverse argument -- What's the point of having a mechanism for identifying different content types and choosing between them when there is only one markup format?
I could argue that we're going through that now. There was some discussion a while back about how to specify content types and then it kind of stalled (as things in open source often do). But I wonder -- if we had the markdown rendering, that might help motivate the discussion.

I don't like having dead code but my bigger concern would be that if we couple two efforts together then each one stalls the other and increases the chances of a deadlock.

dstufft · 2015-01-30T19:05:02Z

We already have two rendering formats, plaintext and rst. The selection mechanism is implicit and fallback based. This won't work if we add in a third rendering format but anything that works for 3 formats will work for 2 formats.

A better mechanism to choose between them would mean that PyPI could reject uploads which specify rst and which fail at rst rendering (for instance). There's no dependency on Markdown for being able to be explicit about how you want the long description rendered. There is a dependency on being explicit about how to render the long description before we can actually render anything with Markdown.

bukzor · 2015-01-30T19:08:13Z

@dstufft: You make some valid points, but I don't see it as an argument to block this merge, let this work bitrot. As @msabramo noted, once this is merged, everyone will be much happier to work on a content-type switching mechanism. It may not be the order your prefer, but it's certainly the order it's been worked on.

msabramo · 2015-01-30T19:11:47Z

Well put, @bukzor!

I also think @dstufft made some valid points about how the content type switching could be useful without Markdown.

Which just confirms for me that these two things are independent of each other and could happen in either order. So why not just tackle them in the order that they naturally unfold so that we maintain forward momentum?

dstufft · 2015-01-30T19:18:37Z

Content type switching is independent, additional markup language is dependent on content type switching.

Allows markup like: Text with double__underscore__words. And it will render the double underscores within the words instead of taking them to mean to format as strong.

Solves PEP 8 line too long errors

msabramo · 2016-07-29T18:40:15Z

Just fixed the merge conflict in this PR.

msabramo · 2016-07-29T20:20:21Z

@nicktimko: Thank you so much for taking the lead here and finding out the steps that are necessary to make this happen! That is super helpful!

For the first item you listed, "Update the metadata spec for the Python Packaging User Guide (PyPUG)", I just submitted a PR to the PyPUG to propose a new Description-Content-Type metadata field: pypa/packaging.python.org#258

Fixes the following test failure (e.g.: https://travis-ci.org/pypa/readme_renderer/jobs/148390785): ``` if "<" in expected: > assert out == expected E assert 'Here is s...nkey</a>\n' == 'Here is so...nkey</a>\n' E Skipping 734 identical leading characters in diff, use -v to show E - n class="s1">'Ruff!') E ? - E + n class="s">'Ruff!') E E - dog = Dog('Fido') E ? - E + dog = Dog('Fido') E </pre> E Detailed information truncated (6 more lines), use "-vv" to show tests/test_rst.py:31: AssertionError ```

I guess the output of [Markdown](https://pypi.python.org/pypi/Markdown) changed a bit?

msabramo · 2016-07-29T23:33:47Z

Hmm, why are there no Travis CI test results for this PR?

$ tox
...
  py27: commands succeeded
  pypy: commands succeeded
  py33: commands succeeded
  py34: commands succeeded
  py35: commands succeeded
  pep8: commands succeeded
  py2pep8: commands succeeded
  packaging: commands succeeded
  congratulations :)

https://travis-ci.org/msabramo/readme/builds/148455750

msabramo · 2016-07-31T16:11:20Z

@dstufft: Are you able to trigger the Travis tests?

msabramo · 2016-07-31T18:51:07Z

Now that we seem to be gravitating towards making CommonMark the preferred Markdown variant in pypa/packaging.python.org#258, I should probably test how well this renders CommonMark and possibly modify it to be more CommonMark-compliant (perhaps using https://pypi.python.org/pypi/CommonMark if it does a better job of rendering CommonMark than https://pypi.python.org/pypi/Markdown ?)

msabramo · 2016-08-01T04:24:04Z

Checklist with steps that @nicktimko mentioned:

Update metadata spec for the Python Packaging User Guide (PyPUG) to document new field
- Add Description-Content-Type field packaging.python.org#258
pypa/setuptools to support setting it
- ~~WIP: Add new long_description_content_type kwarg setuptools#708~~
- Add new long_description_content_type kwarg setuptools#1075
pypa/warehouse to respect it for all defined rendering formats -- in progress --
- pypa/readme_renderer#3 (this PR)
pypa/pypi-legacy to support respecting it at least for reStructuredText and Markdown (treating others as plain text) (haven't checked, but I'm assuming that the above two call back to this pypa/readme_renderer package/repo?)
PyPUG again to provide usage docs once the default tools support it

AraHaan · 2017-03-21T22:38:43Z

an option is to look at and see if rst and markdown both support some method of commenting stuff out. And then if they both have it we could use it for an advantage and look for comments containing either markdown or restructuredtext (case insensitive) at the top of the files/data sent to pypi.

msabramo · 2017-04-15T22:11:13Z

Time has shown that I haven't been able to pick this up (in part because I've changed roles and Python isn't currently a big part of my new role). Anyone else want to pick this up?

AraHaan · 2017-04-19T03:28:00Z

I got an even better idea on it, why not use the vim style syntax rendering.

like then you could tell it if it is md or rst?

Or better yet, look for HTML tags. Markdown supports HTML in the format, while rst does not support any HTML whatsoever.

What if the markdown file has no HTML to embed in it?

Then they can add an html comment tag that then states whatever they want to not be seen so it renders correctly in PyPi.

JulienPalard · 2017-04-27T13:39:42Z

We now have a way from wharehouse filters to tell appart markdown and rst via https://pypi.python.org/pypi/mdorrst, does something else blocks this PR?

Lucas-C · 2017-06-30T10:43:08Z

There seems to be conflicts now.
But please : finish this PR ! This is an awesome feature !

JulienPalard · 2017-06-30T12:03:50Z

see #51 also.

msabramo mentioned this pull request Dec 9, 2014

Markdown support #1

Closed

msabramo force-pushed the markdown branch 2 times, most recently from d40faf9 to fce3394 Compare December 10, 2014 01:54

msabramo mentioned this pull request Dec 10, 2014

Add tox and travis support #2

Merged

msabramo force-pushed the markdown branch from fce3394 to 40d59fb Compare December 10, 2014 18:22

msabramo force-pushed the markdown branch from 40d59fb to 2643da2 Compare December 10, 2014 18:55

msabramo mentioned this pull request Dec 12, 2014

clean can randomize attribute order mozilla/bleach#121

Closed

msabramo mentioned this pull request Dec 13, 2014

Fix GH-121; Make attribute order deterministic mozilla/bleach#137

Merged

msabramo force-pushed the markdown branch from 2643da2 to 38cef5b Compare December 19, 2014 02:24

berkerpeksag reviewed Jan 26, 2015
View reviewed changes

msabramo force-pushed the markdown branch 3 times, most recently from 0150140 to 75e55a5 Compare January 30, 2015 16:48

msabramo added 4 commits July 29, 2016 11:38

Add support for markdown.extensions.smart_strong

9075696

Allows markup like: Text with double__underscore__words. And it will render the double underscores within the words instead of taking them to mean to format as strong.

MANIFEST.in: recursive-include tests *.md

a2146e3

Add support for markdown.extensions.codehilite

278d3f5

test_github_backtick_syntax_for_code: Use files

bb91f67

Solves PEP 8 line too long errors

msabramo force-pushed the markdown branch from 75e55a5 to bb91f67 Compare July 29, 2016 18:39

msabramo mentioned this pull request Jul 29, 2016

Add Description-Content-Type field pypa/packaging.python.org#258

Merged

msabramo added 3 commits July 29, 2016 13:55

Support rename from readme => readme_renderer

fac5ef3

Update expected HTML for markdown tests

264dbfd

I guess the output of [Markdown](https://pypi.python.org/pypi/Markdown) changed a bit?

msabramo mentioned this pull request Jul 30, 2016

WIP: Add new long_description_content_type kwarg pypa/setuptools#708

Closed

ekohl mentioned this pull request Aug 8, 2016

PyPI should render readmes written in Markdown pypa/packaging-problems#46

Closed

hadim mentioned this pull request Jan 29, 2017

Add markdown support pypi/warehouse#869

Closed

devpi-bot mentioned this pull request Mar 31, 2017

Support Markdown for readmes devpi/devpi#193

Closed

msabramo mentioned this pull request May 9, 2017

DESCRIPTION.RST: Extend underline dcos/dcos-cli#991

Merged

pypa deleted a comment from AraHaan Aug 28, 2017

di mentioned this pull request Sep 5, 2017

Markdown #55

Merged

dstufft closed this in #55 Sep 7, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Markdown support #3

Add Markdown support #3

msabramo commented Dec 9, 2014

msabramo commented Dec 10, 2014

msabramo commented Dec 10, 2014

methane commented Dec 11, 2014

msabramo commented Dec 13, 2014

methane commented Dec 17, 2014

msabramo commented Dec 17, 2014

msabramo commented Jan 24, 2015

dstufft commented Jan 24, 2015

berkerpeksag Jan 26, 2015

msabramo Jan 30, 2015

bukzor commented Jan 30, 2015

msabramo commented Jan 30, 2015

sudarkoff commented Jan 30, 2015

dstufft commented Jan 30, 2015

msabramo commented Jan 30, 2015

dstufft commented Jan 30, 2015

bukzor commented Jan 30, 2015

msabramo commented Jan 30, 2015

dstufft commented Jan 30, 2015

msabramo commented Jul 29, 2016

msabramo commented Jul 29, 2016

msabramo commented Jul 29, 2016 •

edited

msabramo commented Jul 31, 2016

msabramo commented Jul 31, 2016 •

edited

msabramo commented Aug 1, 2016 •

edited by di

AraHaan commented Mar 21, 2017

msabramo commented Apr 15, 2017

AraHaan commented Apr 19, 2017 •

edited

JulienPalard commented Apr 27, 2017

Lucas-C commented Jun 30, 2017

JulienPalard commented Jun 30, 2017

Add Markdown support #3

Add Markdown support #3

Conversation

msabramo commented Dec 9, 2014

msabramo commented Dec 10, 2014

msabramo commented Dec 10, 2014

methane commented Dec 11, 2014

msabramo commented Dec 13, 2014

methane commented Dec 17, 2014

msabramo commented Dec 17, 2014

msabramo commented Jan 24, 2015

dstufft commented Jan 24, 2015

berkerpeksag Jan 26, 2015

Choose a reason for hiding this comment

msabramo Jan 30, 2015

Choose a reason for hiding this comment

bukzor commented Jan 30, 2015

msabramo commented Jan 30, 2015

sudarkoff commented Jan 30, 2015

dstufft commented Jan 30, 2015

msabramo commented Jan 30, 2015

dstufft commented Jan 30, 2015

bukzor commented Jan 30, 2015

msabramo commented Jan 30, 2015

dstufft commented Jan 30, 2015

msabramo commented Jul 29, 2016

msabramo commented Jul 29, 2016

msabramo commented Jul 29, 2016 • edited

msabramo commented Jul 31, 2016

msabramo commented Jul 31, 2016 • edited

msabramo commented Aug 1, 2016 • edited by di

AraHaan commented Mar 21, 2017

msabramo commented Apr 15, 2017

AraHaan commented Apr 19, 2017 • edited

JulienPalard commented Apr 27, 2017

Lucas-C commented Jun 30, 2017

JulienPalard commented Jun 30, 2017

msabramo commented Jul 29, 2016 •

edited

msabramo commented Jul 31, 2016 •

edited

msabramo commented Aug 1, 2016 •

edited by di

AraHaan commented Apr 19, 2017 •

edited