wrong location after line directive

[vicuna]

Original bug ID: 5159
Reporter: Hendrik Tews
Assigned to: @damiendoligez
Status: closed (set by @xavierleroy on 2015-12-11T18:07:30Z)
Resolution: fixed
Priority: normal
Severity: minor
Version: 3.12.0
Fixed in version: 3.13.0+dev
Category: -for Camlp4 use https://github.com/ocaml/camlp4/issues
Monitored by: @bobzhang mehdi

Bug description

After line directives the offsets in the camlp4 locations are wrong.
For instance for a file containing the following two lines

1 "b.ml"

let a = 5

the location for the StVal node is

file = b.ml (OK)
start line = 1 (OK)
start bol = 11 (should be 0)
start offset = 11 (should be 0)
end line = 1 (OK)
end bol = ee (should be 0)
end offset = 20 (should be 9)

[vicuna]

Comment author: @damiendoligez

What makes you think it should be 0 ? The regular compiler does the same thing as camlp4.
These offsets are from the start of the lexbuf, not from the start of an embedded (and possibly nonexistent) file.

[vicuna]

Comment author: Hendrik Tews

The documentation says that Loc.start_off "Returns the number of
characters from the begining of the file of the begining of this
location" and because file_name="b.ml" and in b.ml the let is at
position 0 start_off should return 0.

[vicuna]

Comment author: @damiendoligez

Hm. The documentation needs to be rewritten in a less ambiguous way.
In this case, "the file" is your two-line file that contains the # and the let
(i.e. the real file that OCaml is parsing).

In any case, I don't think you should rely on anything else than the
difference between offset and bol.

[vicuna]

Comment author: Hendrik Tews

doligez wrote:

Hm.  The documentation needs to be rewritten in a less ambiguous way.

I don't see any ambiguity, there is only one file, which name is
returned by Loc.file_name.

In any case, I don't think you should rely on anything else than the
difference between offset and bol. 

You mean file_name and start/stop_line may return invalid values?
Why are these functions present then?

How would you report an error location with only the difference
between offset and bol? Something like

File unknown, line unknown, characters 8-9:
Error: ...

?

[vicuna]

Comment author: @damiendoligez

Hendrik Tews wrote:

I don't see any ambiguity, there is only one file, which name is
returned by Loc.file_name.

Actually, there is only one file, the one that OCaml is parsing (your two-line
file that starts with '# 1 "b.ml"'). The fact that you're interpreting it differently
definitely indicates ambiguity.

You mean file_name and start/stop_line may return invalid values?
Why are these functions present then?

No, I mean as far as offset and bol are concerned, only the difference is
significant. Of course you can use the file name and line number.

[vicuna]

Comment author: @damiendoligez

Fixed by rewording the documentation to refer to the lexbuf rather than "the file".

Fixed in trunk [3.13.0] (commit 12144).

[vicuna]

Comment author: Hendrik Tews

May I suggest to update the documentation in camlp4/Camlp4/Sig.ml
as well? There Loc.start_off is still specified to return the
number of characters from the beginning of the file.

I would further suggest to add a note to camlp4/Camlp4/Sig.ml stating:

Note that character numbers in locations refer to character
numbers in the parsed character stream, while line numbers refer
to line numbers in the source file. The source file and the
parsed character stream differ, for instance, when the parsed
character stream contains a line number directive. The line
number directive will only update the file-name field and the
line-number field of the position. It makes therefore no sense to
use character numbers with the source file if the sources contain
line number directives.

[vicuna]

Comment author: @damiendoligez

Done in trunk [3.13.0] (commit 12150)