Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Sieve lexer #1257

Merged
merged 9 commits into from Jan 4, 2020
Merged

Add Sieve lexer #1257

merged 9 commits into from Jan 4, 2020

Conversation

sblondon
Copy link
Contributor

This PR adds a Sieve lexer.
Sieve is a language to filter incoming e-mails into the user mailbox.

@Anteru
Copy link
Collaborator

Anteru commented Nov 23, 2019

Thanks for your contribution -- looks like it's causing some test failures though. Could you please take a look at them? If you need any help, please let me know!

@Anteru Anteru self-assigned this Nov 23, 2019
@Anteru Anteru added the T-feature type: a new feature label Nov 23, 2019
@sblondon
Copy link
Contributor Author

@Anteru I made some improvements but it still does not pass the test. I don't understand why because running:
./pygmentize -v -O full tests/examplefiles/example.sieve
works properly.

I get the same error like Github CI when I run tox:

E           AssertionError: lexer <pygments.lexers.SieveLexer> generated error token for /home/stephane/src/pygments/tests/examplefiles/example.sieve: '.' at position 1109
E           assert Token.Error != Token.Error

The tests are green if the file ./tests/examplefiles/example.sieve is empty.
If example.sieve contains only:

require

I get another error, perhaps it's an hint about the issue:

[...]
>               assert False, absfn
E               AssertionError: /home/stephane/src/pygments/tests/examplefiles/example.sieve
E               assert False

tests/test_examplefiles.py:141: AssertionError
---------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------
--- 

+++ 

@@ -1 +1 @@

-[(Token.Punctuation, '\n')]
+[(Token.Keyword.Namespace, 'require'), (Token.Punctuation, '\n')]

Do you have any idea?

@sblondon
Copy link
Contributor Author

After rm -rf tests/examplefiles/output, the error generated by example.sieve file disappears.

There are two errors in python3.7 with example.praat and example.rkt files which are equivalent (missing token). For example.praat:

- (Token.Name, '#%A|||b #true C\n\\|d\u2002'),
- (Token.Text, ''),
+ (Token.Name, '#%A|||b #true C\n\\|d'),
+ (Token.Text, ''),
+ (Token.Text, '\u2002'),

These errors are also in master branch of pygments.

@birkenfeld
Copy link
Member

You saw the failure generated by cached output in tests/examplefiles/output because a special test mode was active which shouldn't be in the normal case, I've fixed that in d338695.

This shouldn't matter for GitHub CI since it always starts with a clean checkout.

Running the tests locally with your branch, I now get the following error:

E           AssertionError: lexer <pygments.lexers.SieveLexer> generated error token for /home/gbr/devel/pygments/tests/examplefiles/example.sieve: '.' at position 1109

which also occurs on the CI.

@sblondon
Copy link
Contributor Author

@birkenfeld Thanks for your help! I rebased the branch and I get the same error too. I will investigate it.

I wonder if make clean should remove the tests/examplefiles/output/ directory. I can do it in another PR.

@birkenfeld
Copy link
Member

I wonder if make clean should remove the tests/examplefiles/output/ directory. I can do it in another PR.

Yes, that's a good idea in any case.

@sblondon
Copy link
Contributor Author

I fixed the test. $ was not understood as end-of-line in regular expression in 'text' block so I replaced it by \n.

@Anteru Anteru merged commit 3b4a523 into pygments:master Jan 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T-feature type: a new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants