Comment ranges at wrong location

[neumatho]

Try to run the following code in e.g. RunKit:

var meriyah = require("meriyah")
var escodegen = require("escodegen")
var esprima = require("esprima")

var code = `try{}//
finally{}

try{}
catch(e){}//
finally{}

{
try{}
catch(e){}//
finally{}
}
`;

var merComments = [];
var merTokens = [];
var merTree = meriyah.parse(code, {
    ranges: true,
    onComment: function(type, value, start, end) {
        type = type === 'SingleLine' ? 'Line' : 'Block';
        merComments.push({type: type, value: value, range: [start, end]});
    },
    onToken: function(type, start, end) {
        merTokens.push({ type: type, range: [start, end]});
    }
});

merTree = escodegen.attachComments(merTree, merComments, merTokens);
var merCode = escodegen.generate(merTree, { comment: true });

var espriTree = esprima.parse(code, { comment: true, range: true, tokens: true });
espriTree = escodegen.attachComments(espriTree, espriTree.comments, espriTree.tokens);
var espriCode = escodegen.generate(espriTree, { comment: true });

[merComments, espriTree.comments, merTree, espriTree, merCode, espriCode]

If you then compare the ranges in the comment array for both Meriyah and Esprima, then you will see the values are not the same (start is 2 under, end is 1 under). If you then look at the generated code, which are generated from the Meriyah and Esprima tree and compare them, then you will see the comments are placed at the wrong places from the Meriyah tree.

Generated code from Esprima tree

try {
}    //
finally {
}
try {
} catch (e) {
}    //
finally {
}
{
    try {
    } catch (e) {
    }    //
    finally {
    }
}

Generated code from Meriyah tree

try {
} finally
    //
    {
    }
try {
} catch (e) {
}    //
finally {
}
{
    try {
    } catch (e) {
    }    //
    finally {
    }
}

-Thomas

[KFlash]

@3cp Didn't you solve this before so this could work with ESCodegen?

[3cp]

It was #83. Fixed start of non-comment token.

I will have a look of comment, the range of comment is not constructed by a token.

I think I noticed something wrong with comment before. But didn't pay too much attention because it didn't affect my use cases.

[3cp]

We set start for comment after // or /*, while other parsers set start before them. There is also difference on the ending \n. I see ESTree has no requirement on comment token/node, I can update our implementation to follow other parsers.

But this is not the only issue here. In our ast tree, the finally block (in ast node, not the onToken) has wrong start, it should start at { after keyword finally, not before it. I am going to fix finally block and catch block.

@KFlash why we have html comment? for jsx? But why comment #2 is counted as a comment token?

  it('should extract html comments in array', () => {
    const arr: any[] = [];
    parseScript('<!--comment #1\n--> comment #2', {
      onComment: arr,
      webcompat: true
    });
    t.deepEqual(arr, [
      {
        type: 'HTMLOpen',
        value: 'comment #1\n'
      },
      {
        type: 'HTMLClose',
        value: ' comment #2'
      }
    ]);
  });

[KFlash]

HTML comment is part of the AnnexB extension. Same as other parsers. HTML comment skips either <!-- or --> and then parsed as a single line comment.

[3cp]

That's a weird spec, it's different from comment in html format. Since the spec says

SingleLineHTMLCloseComment::
LineTerminatorSequenceHTMLCloseComment

I will make the start of HTMClose comment before "\n-->"

// For Single, start before "//",
// For HTMLOpen, start before "<!--",
// For HTMLClose, start before "\n-->"