.po parser (yecc-based) #3
Conversation
Lists of string tokens are now concatenated into single binaries *by the .yrl parser*. The parser now returns a list of translations as maps, and the Elixir `Gettext.PO.Parser` only has to convert those maps to structs (by setting the `:__struct__` field).
I added a nonterminal "grammar" symbol (in order to remain more flexible in the future) and cleaned up the code a little bit.
I believe this failure is caused by yecc behaving weirdly when `{:string, line,
value}` tokens are used and the value is a binary. I keep getting
`FunctionClauseError`s for `:io_lib.write_string/1` which gets called with the
value of the token, failing because it expects a char list and it gets a binary
instead.
The commit title is bold, but I think I found the bug highlighted by the failing
test from the previous commit. yecc interprets `{string, Line, Contents}` tokens
differently than it handles other tokens as it can be seen at
https://github.com/erlang/otp/blob/maint/lib/parsetools/src/yecc.erl#L2496.
This caused problems when using binaries (common in Elixir) instead of char
lists since yecc tries to "print" them when a syntax error occurs, and for
printing them it uses `io_lib:write_string/1` which expects a char list and
fails with a binary.
The fix to this problem was to just use `{str, Line, Contents}` tokens in
`Gettext.PO.Tokenize.tokenize/1`.
| defmodule Gettext.PO.Parser do | ||
| @moduledoc false | ||
|
|
||
| defmodule Translation do |
There was a problem hiding this comment.
I would move this up to Gettext.PO.Translation and document it. That's because it is likely folks will be using them directly. On the other, you can wait until it happens.
|
This looks great! Feel free to merge this. <3 The next step is a function in |
The `Translation` struct is now documented and ready to be used by users of the library.
|
@josevalim moved |
|
@josevalim with regards to the naming of the parsing functions, I think we should stick to Elixir conventions right now. Parsing a What do you think? I'm going to start working on the implementation of those function (regardless of what their name will be) in the meantime. |
|
@whatyouhide perfect! 馃帀 |
Sorry it took me so long but I've been super busy and I've also been fighting with a nasty bug (read the commit messages if you want to get as angry as I did 馃槃).
The syntax errors should be refined a little but the core functionality is there.
Let me know what you think for now!