-
-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Support for Vimwiki Syntax #863
Comments
I also want this! Seems close to existing options! |
I want this too. It would be helpful for me to use Jekyll with vimwiki. I'm also interested in implementing it if help is needed. |
I'm going to start working on this. Has there been any progress on it here or somewhere else? |
In the mean time, here's a kludge: https://github.com/LarsEKrueger/pandoc-vimwiki It's a pandoc filter to support some of the extra markup. |
I have written a Vimwiki reader. It is a preliminary version, and I have listed the progress at the end of this post. I also wrote a simple test. The files are I'm rather new to Haskell so any feedbacks are welcome :) I couldn't find a dev branch so the PR is submitted to master. Please let me know what I can do to get it merged. Thanks. progress:
|
To implement everything I have the following questions:
|
+++ Yuchen Pei [May 28 17 18:30 ]:
To implement everything I have the following questions:
1. Indentation. The parsers for blockquotes, bullet lists and ordered
lists all require finding out the indentations at the beginning of
the lines. For example
\t may be width 5 or 3 depending on the tabwidth. Is there a
function in the Pandoc module that can output the tabwidth?
readerTabStop opts (where opts :: ReaderOptions).
The current implementation simply uses the length function (in
bullet/ordered lists) or assume the indentation is all spaces (in
blockquotes) to determine the indentation.
See indentWith in Text.Pandoc.Parsing. This will parse a
certain number of spaces indentation, taking into account
tabs. Is that what you're looking for?
If instead you want to parse a bunch of spaces, then
determine the indent level, you may need to write your
own function. You could use tabFilter from
Text.Pandoc.Shared to convert to spaces, then use length,
but that's inefficient (since you're constructing a new
string solely to count its length). Better to write a
function (perhaps guided by tabFilter) that just outputs
a number.
2. Name-value pair attributes. The parsers for preformatted texts and
images are implemented without taking attributes.
I thought this was probably dealt in other parsers and didn't want
to reinvent the wheels:
Is there a function in the Pandoc module that can convert
attributes written in html to [1]Attr?
Like for example, converting the string "class = "python" style =
"width:150px"" to ("", ["python"], [("width", "150px")])?
There is attributes in the Markdown reader, but it's a bit
different, since it's specialized to pandoc Markdown
attribute specifiers. You could probably reuse some code
from that but you'd have to write your own. In the HTML
reader the attribute parsing is all handled by the TagSoup
HTML tokenizer, but I don't think you can run that just
on attributes. (If you have a whole HTML tag, then TagSoup
might be some help.)
3. Maths. display math with environments like equation or align do not
seem to work. When converting to html the string in display math is
automatically enclosed in $$ delimiter. Any solutions to that?
Two options. (1) you can parse them as raw LaTeX blocks.
This will work fine if you're targeting LaTeX, but it won't
work for other formats. (2) you can parse them as
displayMath, and change the environment name. E.g.
changed align to aligned, which works inside $$..$$.
It's not exactly the same, because you lose the numbering,
but it's as good as we can currently do.
This is what we do in the LaTeX reader:
inlineEnvironments :: PandocMonad m => M.Map String (LP m Inlines)
inlineEnvironments = M.fromList
[ ("displaymath", mathEnvWith id Nothing "displaymath")
, ("math", math <$> mathEnv "math")
, ("equation", mathEnvWith id Nothing "equation")
, ("equation*", mathEnvWith id Nothing "equation*")
, ("gather", mathEnvWith id (Just "gathered") "gather")
, ("gather*", mathEnvWith id (Just "gathered") "gather*")
, ("multline", mathEnvWith id (Just "gathered") "multline")
, ("multline*", mathEnvWith id (Just "gathered") "multline*")
, ("eqnarray", mathEnvWith id (Just "aligned") "eqnarray")
, ("eqnarray*", mathEnvWith id (Just "aligned") "eqnarray*")
, ("align", mathEnvWith id (Just "aligned") "align")
, ("align*", mathEnvWith id (Just "aligned") "align*")
, ("alignat", mathEnvWith id (Just "aligned") "alignat")
, ("alignat*", mathEnvWith id (Just "aligned") "alignat*")
]
4. Table attributes. Is there a way to parse table with attributes?
Not yet. See issue #1024; we plan to add attributes when we
add colspan/rowspan, hopefully for pandoc 2.0.
For now I suggest including the whole table in a Div with the
attributes.
|
Thanks for the answers. I haven't found a convenient way to test when I make changes to the code. Before making the pull request I simply added the following code to the import Control.Monad.Except (throwError)
import Text.Pandoc.Class (PandocIO, runIO)
import Text.Pandoc.Parsing (readWithM, stateOptions, ParserState, ParserT)
import Data.Default -- def is there
import Text.Pandoc.Options (ReaderOptions)
import Control.Monad.Except (throwError)
import Text.Parsec.String (Parser)
import Text.Pandoc.Error (PandocError)
import Text.Parsec.Error (ParseError)
import Text.Parsec (parse)
runParser :: VwParser PandocIO a -> String -> PandocIO a
runParser p s = do
res <- readWithM p def{ stateOptions = def :: ReaderOptions } s
case res of
Left e -> throwError e
Right result -> return result
testP :: VwParser PandocIO a -> String -> IO (Either PandocError a)
testP p s = runIO $ runParser p (s ++ "\n")
simpleParse :: Parser a -> String -> Either ParseError a
simpleParse p s = parse p "" (s ++ "\n") so that I can But I suppose this portion of code shouldn't stay when I commit and update the PR. So I was thinking of putting it in a separate file, say However, I can't find a way to load both files with ghci. I tried I also tried Any solutions how I can solve this problem and quickly test my edits in the code? |
Option 1: Add a unit test. The existing test framework is pretty cool. Check the |
The issue is that you want to interactively test
non-exported functions. You can't do that by loading
another file that imports your module, since it will only
see the exported functions.
One approach is to temporarily remove the list of exported
functions after your module declaration; this will result
in everything being exported. (Obviously, this should
only be temporary.)
Another approach would be to temporarily add the test
functions to the VimWiki reader module, and remove
them before the final version.
|
@LarsEKrueger @jgm: thanks. I have fixed all the problems mentioned earlier and implemented more parsers - here's the progress
It still needs a bit of cleaning up, but functionality-wise except table colspan, rowspan and placeholders, it can satisfy all my needs from a vimwiki reader, so unless there is demand, I am not planning to implement ordered lists with non-# markers, todo lists, definition lists, subscripts or superscripts. Concerning placeholders, I wonder how I can implement it. For example, in Vimwiki syntax %title indicates the title of the document, e.g.
should parse as changing the title metadata to After inspecting the Markdown parser, I wrote the following function: titlePH :: PandocMonad m => VwParser m ()
titlePH = try $ do
string "%title" >> spaceChar
title <- (trimInlines . mconcat <$> (manyTill inline newline))
let meta' = return $ B.setMeta "title" title nullMeta :: F Meta
updateState $ \st -> st { stateMeta' = stateMeta' st <> meta' } and added But running
Why? |
+++ Yuchen Pei [Jun 09 17 06:04 ]:
It still needs a bit of cleaning up, but functionality-wise except
table colspan, rowspan and placeholders, it can satisfy all my needs
from a vimwiki reader, so unless there is demand, I am not planning to
implement ordered lists with non-# markers, todo lists, definition
lists, subscripts or superscripts.
I'd like it if these other features could be supported.
Even if you don't need them, others who use pandoc to
convert from vimwiki might. When they find that these
things aren't supported, they'll submit a bug report.
Unless there's some reason why something can't be supported,
we should aim for full support in all the readers.
Otherwise it ends up being a maintenance burden later.
Concerning placeholders, I wonder how I can implement it. For example,
in Vimwiki syntax %title indicates the title of the document, e.g.
%title hello
should parse as changing the title metadata to hello.
After inspecting the Markdown parser, I wrote the following function:
titlePH :: PandocMonad m => VwParser m ()
titlePH = try $ do
string "%title" >> spaceChar
title <- (trimInlines . mconcat <$> (manyTill inline newline))
let meta' = return $ B.setMeta "title" title nullMeta :: F Meta
updateState $ \st -> st { stateMeta' = stateMeta' st <> meta' }
and added mempty <$ titlePH to the list of block parsers used by the
main parser parseVimwiki.
But running parseVimwiki on "%title hello" still gives empty metadata
(testP is the interactive parser tester I mentioned before):
*Text.Pandoc.Readers.Vimwiki> testP parseVimwiki "%title hello"
Right (Pandoc (Meta {unMeta = fromList []}) [])
Why?
Well (not having looked at the details), I'd guess this is
because you're updating the state, but never retrieving the
metadata from state at the end.
Compare the end of parseMarkdown, where the metadata is
retrieved from state just before the final Pandoc value
is returned. (Of course, the details are a bit different,
because of the use of the "F" Monad in that reader. But
I hope this gives you the clue you need.)
|
@jgm OK thanks. I implemented the non-# markers for ordered lists and the date / title metadata placeholder parsers. Still work in progress though. |
Great, let me know when you think it's ready to go.
|
Any hint on implementing the following? template placeholder:
See the vimwiki doc. It applies an html template when converting to html. An example template is the default one here. nohtml placeholder:
Also can be found in the vimwiki doc near the template placeholder. It forbids conversion to html. |
I realised Pandoc has its own I also realised that I mistook codeblock as |
I assume rawblock is raw html so I have left *:not(pre) > code {
/*style for code...*/
} I think this may be ready to merge now. Let me know if there are any bugs or other problems. |
There's no pandoc element that gets rendered to HTML with
just pre, not pre/code.
If there's a vimwiki element that means "pay attention
to line breaks and leading spaces, but still parse
inline syntax like links," then a pandoc LineBlock
might be the right target.
You should avoid using rawBlock as much as you can,
because a raw block will only render properly in
one output format.
+++ Yuchen Pei [Jun 15 17 13:02 ]:
… I realised Pandoc has its own --template option, so I have changed the
code to ignore the %template placeholder. In any case, the vimwiki
templates and the pandoc templates have different syntax so I think it
is better to just use the pandoc template option.
I also realised that I mistook codeblock as <pre> when converted to
html, but in fact it is <pre><code>. How do I get <pre>? Is it
rawblock? If so, what is the format field, and how do I pass attributes
to the rawblock?
—
You are receiving this because you were mentioned.
Reply to this email directly, [1]view it on GitHub, or [2]mute the
thread.
References
1. #863 (comment)
2. https://github.com/notifications/unsubscribe-auth/AAAL5C1TAt7x46TdF204lJ7PW3oJnqotks5sEY27gaJpZM4ArXxq
|
@jgm OK thanks. I don't think there is a vimwiki element that behaves like LineBlock. I think codeblock with css (see my previous comment) may be a good solution to get preformatted texts in vimwiki, so my current implementation is simply codeblock. Another reason for doing so is that the MediaWiki reader also seems to be using codeblock for pre. |
@ycpei's vimwiki reader has now been merged into master. |
The syntax is defined here: https://code.google.com/p/vimwiki/wiki/Syntax
It's very close to the Google wiki syntax defined here: https://code.google.com/p/support/wiki/WikiSyntax
Is this possible? Are there other vimwiki users that want to take advantage of pandoc??
Thanks!
The text was updated successfully, but these errors were encountered: