Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

line number mapping for debug #558

Closed
pmuellr opened this issue Jul 30, 2010 · 181 comments
Closed

line number mapping for debug #558

pmuellr opened this issue Jul 30, 2010 · 181 comments

Comments

@pmuellr
Copy link

@pmuellr pmuellr commented Jul 30, 2010

There's been some email chit-chat regarding support coffeescript in the existing browser debuggers. The basic idea is to include information in the generated JavaScript files that the debuggers can make use of to display the original coffeescript source, and deal with line mapping issues between the CoffeeScript source and generated JavaScript.

Thought I'd go ahead and open up an issue for discussion.

@pmuellr
Copy link
Author

@pmuellr pmuellr commented Jul 30, 2010

So some thoughts:

  • CoffeeScript source location - somehow we need the coffeescript source. See Issue #557 as one possible solution. The debugger would then be responsible for finding the source given that "URL". Java current just uses the basename() of the file as the file name in it's debug info, but of course it also has the full class name available to it as well. Not sure how we might get some "relative" name in there, so it seems like basename() or abspath() of the input file are the choices. abspath() exposes more information than it should, basename() will cause naming conflicts for same-named files in different directories
  • line number information - The debugger will eventually want this in hash tables, both directions. Go from a cs line # to js line # when setting breakpoints, and js line # to cs line # when the JS vm is communicating to the debugger (eg, "stopped at line 47"). Presumably the actual data for the line number data could be in JSON format (eg, an object literal mapping cs #'s to an array of js #'s).
@mrblonde
Copy link

@mrblonde mrblonde commented Aug 6, 2010

Might be worth looking at this also:
http://nex-3.com/posts/92-firesass-bridges-the-gap-between-sass-and-firebug

Maybe this same type of strategy could be used so that firebug could step through javascript line by line, but display the coffeescript code instead.

Tough project, but could turn into something pretty killer!

@jashkenas
Copy link
Owner

@jashkenas jashkenas commented Aug 17, 2010

I'd be glad to help someone out with this ticket, but I'm not going to be spearheading it myself ... so closing as frozen for the time being. If someone wants to step up and do a port of FireSass for CoffeeScript, do let us know.

@jashkenas
Copy link
Owner

@jashkenas jashkenas commented Dec 12, 2010

Alright -- this ticket now has life again, in Firefox at least:

https://bugzilla.mozilla.org/show_bug.cgi?id=618650

Reopening...

@subtleGradient
Copy link

@subtleGradient subtleGradient commented Dec 13, 2010

Quick helpful workaround:

Include each line of the original coffeescript source inside a block comment before the js code that it generated. You could include the original coffeescript source lune numbers in the comments.

That would at least help simplify the learning and debugging process.

@StanAngeloff
Copy link
Contributor

@StanAngeloff StanAngeloff commented Dec 13, 2010

Mascara has a source-line mapping utility that only works in Firefox. It's quite possible to implement something similar:

try
  console.log 'Hello'
  a for i in [0..10]
catch error
  lines = { 31: 3, 32: 3, 33: 3 }
  error.message = error.message + ' on line ' + lines[error.stack.split('\n').shift().split(':').pop()]
  throw error

Run on Try CoffeeScript. Firefox only, though.

@andrewschaaf
Copy link
Contributor

@andrewschaaf andrewschaaf commented Dec 18, 2010

Related code:

Closure Compiler (JS -> JS)

...can generate source-to-source mappings:

java -jar $CLOSURE_JAR --js example.js --create_source_map ./map --js_output_file compiled.js

SourceMapLegacy.java has comments describing the format.

_PYXC-PJ (PY -> JS)_

...can generate mappings using the same format, but with more file info: [{"sha1":"...",...}] instead of []

commit: "changed emit() return type from a string to list of strings and kids"

commit: "source-to-source mappings! (same format as Closure Compiler)"

** exception-server **

An exception logging/viewing server has been written (in NodeJS) that uses these mappings to convert an exception's stack information into pre-compilation code snippets with TextMate URLs.

(your compilers/scripts send the server (code_sha1, mapping)s and (sha1, code)s at compile-time)

I'll post it [REDACTED TIMEFRAME] when it's more documented/polished.

** common-exception **

...can extract file/line/[col] information from NodeJS, Firefox, Chrome, Safari

@lucaswoj
Copy link

@lucaswoj lucaswoj commented Jan 5, 2011

Crazy idea: What if when the CoffeeScript was compiled to JavaScript, it was done such that the lines corresponded 1:1 by placing multi-line JavaScript conversions on a single line.

@pmuellr
Copy link
Author

@pmuellr pmuellr commented Jan 5, 2011

I suspect it is crazy, but it would be useful. I actually do that with https://github.com/pmuellr/scooj , but only because the source is largely unprocessed JavaScript anyway. The nice thing is that even though you see some grungy code in the debugger, you do know the exact line number to go to in your editor.

@alexandertrefz
Copy link

@alexandertrefz alexandertrefz commented Jan 6, 2011

I suggest a --debug coffee parameter that does something like this:

# Add Core-Utils to Underscore Namespace
_.ducktype = (obj, methods...) ->
  some("code")

to become:

var __slice = Array.prototype.slice;
// Line 2
_.ducktype = function() {
  var methods, obj;
  obj = arguments[0], methods = 2 <= arguments.length ? __slice.call(arguments, 1) : [];
  // Line 3
  return some("code");
};

this would work Browser independent and i think(im not a coffescript compiler expert, though) it could be implement without big hassles.

@andrewschaaf
Copy link
Contributor

@andrewschaaf andrewschaaf commented Jan 7, 2011

Let's define some terms:

Def a "compiler" compiles a non-empty set of "source file"s to exactly one "compiled file"

Defs

  • a "(coord to coord)-mapping" is a function from (line, col) in the compiled file to (source_file_ref, line, col)
  • a "(line to line)-mapping" is a function from (line) in the compiled file to (source_file_ref, line)
  • a compiled file is "{coord,line}-{coord,line}-commented" if it contains comments in some format which provide a ({coord,line} to {coord,line})-mapping
  • a "(...mapping...) file" is a byte-string in some format which contains a representation of a (...mapping...)

Notes:

  • Don't worry, the representations of these mappings are usually much more space-efficient than the ultra-trivial {coord1:info1, ...}
  • source_file_ref should include sha1(source_file) if possible, because that's really useful when used with my not-yet-released exception server

Defs

  • "CCMF" is a (coord to coord)-mapping format. It's what Closure Compiler generates
  • "CCMFH" is a backward-compatible extension with required hashes and optional other info

See my writeup of CCMF[H]: https://gist.github.com/769845

@andrewschaaf
Copy link
Contributor

@andrewschaaf andrewschaaf commented Jan 7, 2011

.lineno

Recall: adding .lineno to each node instance is a solved problem:

#955

With no side effects, that grammar.coffee change could be merged at any time.

@andrewschaaf
Copy link
Contributor

@andrewschaaf andrewschaaf commented Jan 7, 2011

Let's get this done! (Please? Please? Please?)

CoffeeScript should support both

  1. {coord,line}-line-commenting in whatever that format turns out to be (e.g. via --line-commenting)
  2. exporting CCMFH files (e.g. via --ccmfh-out=)

(1) might need to wait for some format finalization, but (2) does not.

And the nodes.coffee work required for (2) will apply to (1).

I'm trying to do a FOSS triple-launch in [REDACTED TIMEFRAME]:

(all three are written in CoffeeScript)

...and it would be really, really awesome to have CoffeeScript CCMFH support before then so I can integrate it.

Imagine being able to hit Command-R and have jashkenas/coffee-script-tmbundle compile, invoke node, and then represent the resulting exception in a beautiful and convenient mapping-following way with CoffeeScript source links and code snippets...

@andrewschaaf
Copy link
Contributor

@andrewschaaf andrewschaaf commented Jan 7, 2011

Some modest proposals

The Good (but more complex and possibly slower)

Something like what PYXC-PJ does.

EDIT: e.g.

Throw
  compileNode: (o) ->
    ...
    something(o, ["throw ", @expression, ";"])

Instead of having the main compilation function return a string, have it return a BAR.

BAR ::= ELEMENT-list

ELEMENT ::= string | FOO | BAR

FOO ::= something which can
    1. provide the code fragment for that subtree
    2. provide the ((line, col) -> (source line))
       mapping information for that fragment

Some function can then use the resulting BAR to generate both

  1. the usual JavaScript
  2. a CCMFH file

PYXC-PJ commits:

I'm not a nodes.coffee expert (yet) so I'm not sure if subnode.compile(o) can be deferred. If so would let FOO be, e.g., {subnode:@expression, o:o}

Would o have been destructively modified by the time foo.subnode(foo.o) gets run?

I think The Good could be done without deferring .compile, but that might be harder.

The Bad (but simple (though verbose) to implement and possibly faster)

Something like what Closure Compiler does.

Instead of the beatifully concise current compileNode methods, do something like:

    something.startNode this
    something.add           "throw {"
    something.addSubnode    @expression, o
    
    # addSubnode: (node, o) ->
    #   ...
    #   node.compile o
    #   ...
    
    something.add           ";"
    something.endNode()

This would invoke the .compiles in the same order as the current implementation.

The Fugly

Have whatever methods invoke compileNode replace ([...].compileNode([...])) with

((() ->
    token = newUniqueToken()
    (
        "/*FUGLY:#{token}:@lineno*/" + 
        [...].compileNode([...]) +
        "/*/FUGLY:#{token}*/"
    )
)())

...and write a crazy function that uses the resulting text monstrosity to generate (comment-free JS, a CCMFH file)

This would be totally insane, but it would work and it would involve the least modification to nodes.coffee

@andrewschaaf
Copy link
Contributor

@andrewschaaf andrewschaaf commented Jan 7, 2011

I'll work on a

something(o, ["throw ", @expression, ";"])

approach

@andrewschaaf
Copy link
Contributor

@andrewschaaf andrewschaaf commented Jan 9, 2011

BAR-ification of nodes.coffee in progress:

https://github.com/andrewschaaf/coffee-script/commits/master

After each change, I run

git checkout -- lib && cake build && cake build && cake test && git status

and confirm that the tests pass and that no lib/*.js files other than nodes.coffee are changed

@andrewschaaf
Copy link
Contributor

@andrewschaaf andrewschaaf commented Jan 9, 2011

From #coffeescript yesterday:

my point about a codegen method's "//#{@child.compile o}" vs "#{@child.compile o}//" was that the codegen method currently discards the information distinguishing the two.

the relativeCharOffset of each kid matters.

one option would be for the codegen method to store that information in its kids

e.g. for Throw ("throw #{@expression...};"),

@expression.relativeCharOffset = "throw ".length

but this could make the codegen methods uglier than with the BAR-returning approach

with the BAR-returning approach, the Base method calling the codegen method can process the BAR immediately, computing the flattenedCodeString as usual and (processing the mapping implications if mapping:true)

BAR-returning codegen methods are a way to abstract out the mapping-handling code

from the outside, you're still calling the top node's .compile or .compileWithMapping

1 similar comment
@andrewschaaf
Copy link
Contributor

@andrewschaaf andrewschaaf commented Jan 9, 2011

From #coffeescript yesterday:

my point about a codegen method's "//#{@child.compile o}" vs "#{@child.compile o}//" was that the codegen method currently discards the information distinguishing the two.

the relativeCharOffset of each kid matters.

one option would be for the codegen method to store that information in its kids

e.g. for Throw ("throw #{@expression...};"),

@expression.relativeCharOffset = "throw ".length

but this could make the codegen methods uglier than with the BAR-returning approach

with the BAR-returning approach, the Base method calling the codegen method can process the BAR immediately, computing the flattenedCodeString as usual and (processing the mapping implications if mapping:true)

BAR-returning codegen methods are a way to abstract out the mapping-handling code

from the outside, you're still calling the top node's .compile or .compileWithMapping

@ozra
Copy link

@ozra ozra commented Jan 13, 2011

About ten years ago I wrote som crude source-to-source compilers for Perl and C++ which I called Pyrl and Cython. They simply gave you pythonish indent-syntax. They had the flag -p for pretty output and some other for line-to-line, which esentially means that it simply made sure that lines in the output ended up in the same line number as in the input, no matter if certain lines became 'ugly' because of code stacking.

Simple solution. Works in all environments (like Jake that compiles the coffee-makefile from inside it's script), any editor, etc., shouldn't be to hard to implement?

@ghost
Copy link

@ghost ghost commented Jan 28, 2011

How hard would it be to add an option to have the js output include a call to a custom trace function on every function call? The param could be the callee function name and optionally the original line number and file. This would really help debug those cases where the code doesn't finish because I forgot to call something else at the end of a callback. These cases drive me crazy and cause me to put log calls all over the place.

The convention could be that if a specific function exists at the top level then the feature is enabled and that function is called.

debugFnCallLog = (funcName, fileName, lineNum) ->
    myCustomLogRoutine ...
@mathieuravaux
Copy link

@mathieuravaux mathieuravaux commented Apr 19, 2011

For reference, here is a related webkit ticket: https://bugs.webkit.org/show_bug.cgi?id=30933

@ghost
Copy link

@ghost ghost commented Apr 19, 2011

In my fantasy world V8 and other engines would be broken into a separate parser and byte-code interpreter while retaining the same JS semantics.

@aseemk
Copy link
Contributor

@aseemk aseemk commented May 3, 2011

Just want to toss out a major +1 to preserving line numbers, i.e. mapping 1:1 from source Coffee line # to generated JS line #. The excellent Streamline.js library has this option for its transformations, and it has proved invaluable.

Btw, if it helps, our platform is Node. We too compile dynamically w/out generating JS files, so things like line-numbers-in-comments aren't helpful. Our use case also isn't e.g. Firebug debugging generally but more "my program threw an error on line 512, where is that?"

Thanks in advance! Excitedly looking forward to seeing what comes of this.

@thelinuxlich
Copy link

@thelinuxlich thelinuxlich commented May 4, 2011

+1 for this

@andrewschaaf
Copy link
Contributor

@andrewschaaf andrewschaaf commented May 4, 2011

Anyone in the NYC area interested in meeting up for a line-mapping mini-hackathon on Sunday the 15th?

@lucaswoj
Copy link

@lucaswoj lucaswoj commented May 4, 2011

No, but I'd personally donate beer money. This needs to be done.

On May 4, 2011, at 10:27 AM, andrewschaafreply@reply.github.com wrote:

Anyone in the NYC area interested in meeting up for a line-mapping mini-hackathon on Sunday the 15th?

Reply to this email directly or view it on GitHub:
#558 (comment)

@tanepiper
Copy link

@tanepiper tanepiper commented May 5, 2011

+1 I'd send some coffee + beer money too for this, it would be the biggest evolution for CoffeeScript

@ghost
Copy link

@ghost ghost commented May 5, 2011

I'm surprised that no one has ever implemented one of the several quick
kludges that have been suggested.

On Wed, May 4, 2011 at 6:14 PM, tanepiper <
reply@reply.github.com>wrote:

+1 I'd send some coffee + beer money too for this, it would be the biggest
evolution for CoffeeScript

Reply to this email directly or view it on GitHub:
#558 (comment)

Mark Hahn
Website Manager
mark@boutiquing.com
949-229-1012

@michaelficarra
Copy link
Collaborator

@michaelficarra michaelficarra commented May 5, 2011

I'll actually be in the NYC area on the 15th with nothing to do. I may be up for this.

@lephyrius
Copy link

@lephyrius lephyrius commented May 6, 2011

+1 for some way of debugging coffeescript. It would be really helpful.

@jashkenas
Copy link
Owner

@jashkenas jashkenas commented Mar 22, 2012

That's fantastic. I'd love to try and get one of the branches merged, and a release cut, for when it lands in Chrome proper.

@jimtla
Copy link

@jimtla jimtla commented Mar 30, 2012

Source maps are in chrome proper - version 18 has support.

There's a nice explanation here which, incidentally, calls us out.

@ehartford
Copy link

@ehartford ehartford commented Apr 5, 2012

Is there a coffee fork that enables source maps compatible with chrome 18's new feature?

@brokenseal
Copy link

@brokenseal brokenseal commented Apr 6, 2012

I'm looking forward to test this in chrome 18 :)

@gimmi
Copy link

@gimmi gimmi commented May 1, 2012

+1

I'm waiting for this feature to start taking seriously coffeescript

@studgeek
Copy link

@studgeek studgeek commented May 11, 2012

Note IDEs that provide their own debugging environment can do the source mapping as well, they just need the sourcemap from coffeescript. For example JetBrains IDEA has a feature request for this (http://youtrack.jetbrains.com/issue/IDEA-84442) and Firefox still has this issue that folks can vote for (https://bugzilla.mozilla.org/show_bug.cgi?id=618650), but neither is going to move until they have something generating sourcemaps for them to play with.

@emilecantin
Copy link

@emilecantin emilecantin commented Jun 4, 2012

Chrome 19 is now out, is there any progress on this? I'm really looking forward to debug directly in CoffeeScript.

@jashkenas
Copy link
Owner

@jashkenas jashkenas commented Jun 4, 2012

'Fraid there's nothing to report yet progress-wise, but it's next on the list.

@vosechu
Copy link

@vosechu vosechu commented Jul 17, 2012

You're all amazing and we love you. I've been using Coffeescript on a production project for the last two months and I'm absolutely in love with it. This would just be a little extra frosting on the cake.

@simonexmachina
Copy link

@simonexmachina simonexmachina commented Jul 22, 2012

Awesome guys, can't wait for this to land.

@gojomo
Copy link

@gojomo gojomo commented Sep 15, 2012

I realize I'm way late to an old and overlong thread, but wanted to throw another offbeat hybrid idea out there:

Indicate the two different "line" numbers by using two different line-end conventions in the same compiled file. For example, the compiled/JS line numbers would be indicated by traditional line-ends (CR/LF/CRLF), and the original-source line-numbers by some other harmless whitespace, such as FF (0x0C) or VB (0x0B).

That is, every JS line has a traditional line-length and line-end (no overlong lines), but only those representing another line forward in the CS source also include the new-convention secondary line-end. (And, this approach also allows for multiple source-line-ends to a single output line-end.)

Code editors and line-reporting code could offer a toggle for how they number lines.. or simply report both when dual conventions are noticed in the file. For example...

underscore_coffee.js:64;83

...where the ';83' indicates the secondary-convention line-number.

@wamatt
Copy link

@wamatt wamatt commented Oct 30, 2012

Given that Redux now supports this. What is the implication for this repo?

Politics aside, and speaking strictly pragmatically for the benefit of new users such as myself, should we be switching to redux as the future compiler of choice?

@michaelficarra
Copy link
Collaborator

@michaelficarra michaelficarra commented Oct 30, 2012

@wamatt: I would recommend sticking with this more stable compiler for serious projects for now as the kinks are worked out of my compiler over the next few months. That said, if you can go over the compiled output manually (or if your program has good test coverage and your tests still pass), and it looks good to you -- which it really should for most programs -- go ahead and take advantage of the source map support.

@wamatt
Copy link

@wamatt wamatt commented Oct 31, 2012

@michaelficarra Thanks, that sounds sensible. Keep up the great work! :)

@devongovett
Copy link

@devongovett devongovett commented Feb 2, 2013

What's the status on this? @michaelficarra's Redux compiler seems to support them and there is at least one pull request for support in this compiler but relatively little discussion from the maintainers. Is sourcemap support still something being planned? Is something holding it back? Just looking for updates.

@jashkenas
Copy link
Owner

@jashkenas jashkenas commented Feb 3, 2013

@devongovett -- I think the status is basically what you said ... they work in Redux, and there are a couple of probably working branches with pull requests over here as well. One in particular I've been keeping my eye on and corresponding via email about, that looks very promising.

That said, I'd like to wrap up one new release of CoffeeScript with official "literate" support before starting to look at 'em in earnest. If you want to help move things along, feel free.

@vendethiel
Copy link
Collaborator

@vendethiel vendethiel commented Mar 4, 2013

PONG, 3 years after

@michaelficarra
Copy link
Collaborator

@michaelficarra michaelficarra commented Mar 4, 2013

Oh, wow, we can finally close this. The oldest open CoffeeScript issue. Well, here goes 😄.

edit: Actually, I forgot about the recently re-opened issue #77. That issue's even older than me.

@andrewschaaf
Copy link
Contributor

@andrewschaaf andrewschaaf commented Mar 5, 2013

Congratulations!

@sheerun
Copy link

@sheerun sheerun commented Mar 5, 2013

Unfortunately 🍰 was a lie ;-) Thank you!

@wamatt
Copy link

@wamatt wamatt commented Mar 6, 2013

Woot!

@terrycojones
Copy link

@terrycojones terrycojones commented Mar 7, 2013

Awesome ticket :-)

@ajumell
Copy link

@ajumell ajumell commented Aug 4, 2013

@JJ
Copy link

@JJ JJ commented Feb 6, 2016

wow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.