New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up quoted text. #2

Open
godber opened this Issue Jan 2, 2015 · 2 comments

Comments

Projects
None yet
1 participant
@godber
Owner

godber commented Jan 2, 2015

Currently the white space and format effectors are included verbatim in quoted text. They should be stripped out. See section 12.5.3.

@godber godber self-assigned this Jan 3, 2015

@godber godber modified the milestone: v0.9 Jan 3, 2015

godber added a commit that referenced this issue Jan 18, 2015

Punting on handling strings at the parser/lexer level.
This grammar will not return properly cleaned up strings as defined in 12.5.3.
Users of this grammar will have to clean up strings at the application code
level.  I haven't managed to figure out a proper way to deal with it in the
parser or lexer.

refs #2
@godber

This comment has been minimized.

Show comment
Hide comment
@godber

godber Jan 18, 2015

Owner

As mentioned in the previous commit message, I am punting on solving this problem. Since I can handle this easily at the application code layer, I am going to do so. But here are some examples of what is not being handled correctly:

The following statement:

COMMENT_2_A    = "Alive Again."

should return the following tokens:

  • COMMENT_2_A - A symbol
  • = - An equal sign
  • Alive Again. - a string

Strings with multiple spaces and line feeds should consolidate those into a single space, so this:

MULTILINE      = "This is a test
                  of the emergency broadcasting system."

Should return the following tokens:

  • MULTILINE - A symbol
  • = - An equal sign
  • This is a test of the emergency broadcasting system. - a string

Hyphenated strings should have the hyphens removed and also be consolidated:

HYPHENATED     = "The planet Jupi-
                 ter is very big"

Should become:

  • HYPHENATED - A symbol
  • = - An equal sign
  • The planet Jupiter is very big - A string.
Owner

godber commented Jan 18, 2015

As mentioned in the previous commit message, I am punting on solving this problem. Since I can handle this easily at the application code layer, I am going to do so. But here are some examples of what is not being handled correctly:

The following statement:

COMMENT_2_A    = "Alive Again."

should return the following tokens:

  • COMMENT_2_A - A symbol
  • = - An equal sign
  • Alive Again. - a string

Strings with multiple spaces and line feeds should consolidate those into a single space, so this:

MULTILINE      = "This is a test
                  of the emergency broadcasting system."

Should return the following tokens:

  • MULTILINE - A symbol
  • = - An equal sign
  • This is a test of the emergency broadcasting system. - a string

Hyphenated strings should have the hyphens removed and also be consolidated:

HYPHENATED     = "The planet Jupi-
                 ter is very big"

Should become:

  • HYPHENATED - A symbol
  • = - An equal sign
  • The planet Jupiter is very big - A string.

@godber godber added the bug label Jan 18, 2015

@godber godber removed this from the v0.9.0 milestone Jan 18, 2015

@godber

This comment has been minimized.

Show comment
Hide comment
@godber

godber Jan 18, 2015

Owner

I am designating this as a known bug and removing it from the v0.9.0 milestone. The workaround is to handle these strings properly at the application code level.

Owner

godber commented Jan 18, 2015

I am designating this as a known bug and removing it from the v0.9.0 milestone. The workaround is to handle these strings properly at the application code level.

godber added a commit that referenced this issue Jan 18, 2015

@godber godber added the help wanted label Jan 18, 2015

@godber godber removed their assignment Nov 10, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment