Parser.withSource() parser

[stepancheg]

Parser<T>.withSource() parser applies this parser and returns its
result along with matches string.

.token() parser could be used instead of .withSource(), however
sometimes WithSource is more convenient, because global string
object has to be stored somewhere to resolve token into string.

Also, WithSource class is type-parameterized, unlike Token.

[abailly]

Hi Stepan,
Thanks for your contribution. It is not clear to me what's the use case for this feature. We had months ago a discussion about introducing something similar and we finally decided that the current API was enough to support expected usage (see #5 for more info). How does withSource() differ from the aforementioned issue?

Regards,
Arnaud

[stepancheg]

@abailly

Yes, use case seems to be the same as in #5: I need to get the source of parser along with result of parser, for example, to perform syntax highlighting.

As far as I understood, official recommendation is to use Token. With Token Parser cannot be created with static function, and has refer string stored outside. See comments in this code:

public class ParserWithToken {

  // result of parsing
  static class MyToken {
    private final Object value;
    private final String underlying;

    MyToken(Object value, String underlying) {
      this.value = value;
      this.underlying = underlying;
    }
  }

  // this field and constructor are requires to use token
  private final String string;

  public ParserWithToken(String string) {
    this.string = string;
  }

  // this function cannot be made static
  private Parser<MyToken> myToken() {
    return Scanners.INTEGER.or(...).token().map(new Map<Token, MyToken>() {
      @Override
      public MyToken map(Token token) {
        return new MyToken(token.value(), string.substring(token.index(), token.index() + token.length()));
      }
    });
  }

  private Parser<List<MyToken>> parser() {
    return myToken().many();
  }

  public static List<MyToken> parse(String string) {
    // extra constructor call
    return new ParserWithToken(string).parser().parse(string);
  }
}

Lack of withSource is not critical, of course, but having to use outer variable to resolve token makes code slightly more verbose, not functional, harder to compose etc.

(BTW, now I think better WithSource should provide both source and indices like Locatable from #5).

[fluentfuture]

Another alternative suggested in the thread, is to use the Parsers.INDEX
parser.

static Parser<WithSource> withSource(Parser parser) {
curry(WithSource.class).sequence(INDEX, parser, INDEX);
}

The only difference is that it's a static method, not a Parser instance
method.

On Wed, Feb 19, 2014 at 12:15 PM, Stepan Koltsov
notifications@github.comwrote:

@abailly https://github.com/abailly

Yes, use case seems to be the same as in #5#5:
I need to get the source of parser along with result of parser, for
example, to perform syntax highlighting.

As far as I understood, official recommendation is to use Token. With
Token Parser cannot be created with static function, and has refer string
stored outside. See comments in this code:

public class ParserWithToken {

// result of parsing
static class MyToken {
private final Object value;
private final String underlying;

MyToken(Object value, String underlying) {
  this.value = value;
  this.underlying = underlying;
}

}

// this field and constructor are requires to use token
private final String string;

public ParserWithToken(String string) {
this.string = string;
}

// this function cannot be made static
private Parser myToken() {
return Scanners.INTEGER.or(...).token().map(new Map<Token, MyToken>() {
@OverRide
public MyToken map(Token token) {
return new MyToken(token.value(), string.substring(token.index(), token.index() + token.length()));
}
});
}

private Parser<List> parser() {
return myToken().many();
}

public static List parse(String string) {
// extra constructor call
return new ParserWithToken(string).parser().parse(string);
}
}

Lack of withSource is not critical, of course, but having to use outer
variable to resolve token makes code slightly more verbose, not functional,
harder to compose etc.

(BTW, now I think better WithSource should provide both source and
indices like Locatable from #5#5
).

Reply to this email directly or view it on GitHubhttps://github.com//pull/16#issuecomment-35529753
.

[stepancheg]

@fluentfuture INDEX parser has the same issue as Token parser: it does not directly provide access to parsed string.

[fluentfuture]

That's right. It assumes that you already have the raw string at the time
of constructing the parser.

Can you help us understand your use of the source string a bit more? How is
the source used to do syntax highlighting that the indices can't do?

On Wed, Feb 19, 2014 at 1:15 PM, Stepan Koltsov notifications@github.comwrote:

@fluentfuture https://github.com/fluentfuture INDEX parser has the same
issue as Token parser: it does not directly provide access to parsed
string.

Reply to this email directly or view it on GitHubhttps://github.com//pull/16#issuecomment-35536315
.

[stepancheg]

@fluentfuture everything can be implemented with indices. It just less convenient: I have to explicitly pass original string to parser construction code.

[fluentfuture]

So, I'm just imagining a syntax highlighter. It's possibly too simplistic
but it's why I asked the question how the source is needed:

Parser parser = or(
withColor(keyword, GREEN),
withColor(operator, BLUE),
withColor(word, GRAY),
withColor(blank, WHITE)).many();

class Colored {
private final begin;
private final end;
private final Color color;
}

render(Parser<List> parser, String text) {
for (Colored colored : parser.parse(text)) {
String text = text.substring(colored.begin, colored.end);
renderWithColor(text, colored.color);
}
}

On Wed, Feb 19, 2014 at 1:48 PM, Stepan Koltsov notifications@github.comwrote:

@fluentfuture https://github.com/fluentfuture everything can be
implemented with indices. It just less convenient: I have to explicitly
pass original string to parser construction code.

Reply to this email directly or view it on GitHubhttps://github.com//pull/16#issuecomment-35540032
.