Duplicate tokens captured while parsing. Is that expected?

[dsherret]

Describe the question

I'm getting duplicate tokens in the output. Is that expected?

I think I'm going to sort and de-duplicate the items.

Input code

// full code is here: https://github.com/dsherret/dprint/blob/master/packages/rust-dprint-plugin-typescript/src/parse_swc_ast.rs
let mut ts_config: swc_ecma_parser::TsConfig = Default::default();
ts_config.tsx = should_parse_as_jsx(file_path);
ts_config.dynamic_import = true;
ts_config.decorators = true;
let lexer = Lexer::new(
    session,
    Syntax::Typescript(ts_config),
    JscTarget::Es2019,
    SourceFileInput::from(&source_file),
    Some(&comments)
);
let lexer = Capturing::new(lexer);
let mut parser = Parser::new_from(session, lexer);
let parse_module_result = parser.parse_module();
let tokens = parser.input().take();

for token in tokens.iter() {
    println!("Token: {:?}", token);
}

Code being parsed:

type Test = (
    string | number)

Expected behavior

No duplicate tokens.

Actual behaviour

Token: TokenAndSpan { token: Word(type), had_line_break: true, span: Span { lo: BytePos(0), hi: BytePos(4), ctxt: #0 } }
Token: TokenAndSpan { token: Word(Test), had_line_break: false, span: Span { lo: BytePos(5), hi: BytePos(9), ctxt: #0 } }
Token: TokenAndSpan { token: AssignOp("="), had_line_break: false, span: Span { lo: BytePos(10), hi: BytePos(11), ctxt: #0 } }
Token: TokenAndSpan { token: LParen, had_line_break: false, span: Span { lo: BytePos(12), hi: BytePos(13), ctxt: #0 } }
Token: TokenAndSpan { token: Word(string), had_line_break: true, span: Span { lo: BytePos(18), hi: BytePos(24), ctxt: #0 } }
Token: TokenAndSpan { token: BinOp(BitOr), had_line_break: false, span: Span { lo: BytePos(25), hi: BytePos(26), ctxt: #0 } }
Token: TokenAndSpan { token: Word(string), had_line_break: true, span: Span { lo: BytePos(18), hi: BytePos(24), ctxt: #0 } }
Token: TokenAndSpan { token: BinOp(BitOr), had_line_break: false, span: Span { lo: BytePos(25), hi: BytePos(26), ctxt: #0 } }
Token: TokenAndSpan { token: Word(number), had_line_break: false, span: Span { lo: BytePos(27), hi: BytePos(33), ctxt: #0 } }
Token: TokenAndSpan { token: RParen, had_line_break: false, span: Span { lo: BytePos(33), hi: BytePos(34), ctxt: #0 } }
Token: TokenAndSpan { token: Semi, had_line_break: false, span: Span { lo: BytePos(34), hi: BytePos(35), ctxt: #0 } }

Version

master

[kdy1]

No. It's not intended and seems like a bug related to backtracing.

[kdy1]

I fixed and published a new version

[swc-bot]

This closed issue has been automatically locked because it had no new activity for a month. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.