Add support for cell magics to specify cell language type (for highlighting / interpreting)

[matthewwardrop]

Greetings all,

Background
I've been working on a project called Omniduct (http://github.com/airbnb/omniduct), which among other things dynamically adds cell magics to the user IPython session that expose certain data sources in a convenient manner.

For example, if one is running multiple HiveServer2 instances somewhere, you can configure as appropriate various connection parameters, after which you can register cell magics associated with these services with base names of "hive1" and "hive2", for example. This then adds line and cell magics to the user session like %hive1.desc for getting table descriptions/schemas and %%hive2 (followed by SQL) for querying the database.

Feature Request
At time of IPython magic registration, I would like to be able to specify the expected language/markup of the cell contents. This would allow appropriate highlighting of cell contents for any cell magic (and should probably default to no highlighting if not specified). Currently, as of PR #2216, there is support for highlighting cell contents, but this support is limited to several pre-defined languages that are hard-coded into the javascript itself. I am imagining code that looks something like this:

from IPython.code.magic import register_cell_magic

@register_cell_magic('mymagic', language='sql')
def mymagic(line, cell):
    return 'Saw your SQL. Nice!'

As a workaround, it looks like it might be possible to hack javascript into the user session using something like that shown in #7699, but this seems a little messy.

Is this something that is in-principle ruled out? Is this something that you would be interested in seeing contributions for?

Extra points
It would be even more amazing if you could optionally pass a callable object as the language, which gets evaluated at runtime based upon the cell contents. This will obviously be much more difficult, since it needs to bridge the kernel/js bridge and perhaps support non-Python languages. But I'm imagining this might look something like:

from IPython.code.magic import register_cell_magic

@register_cell_magic('mymagic', language=lambda cell_contents: 'sql' if 'SELECT' in cell_contents else None)
def mymagic(line, cell):
    return 'Saw your SQL. Nice!'

[Carreau]

Hi @matthewwardrop,

What you ask is a bit tough. The problem is that the frontend is not aware that %%whatver are magics, as far as it knows these are opaque lob of text. We did hardcode some of this knowledge when the notebook was still part of IPython , but we are moving away from that. So we will need to rethink the architecture a bit to have way to get this information through. Though I agree it would be a nice thing.

[carlsmith]

If this feature is added, it should ideally be supported in the terminal too.

[Carreau]

If this feature is added, it should ideally be supported in the terminal too.

It would technically be way easier to implement in the terminal than in the notebook.