Crash when using %macros in sqlite-history branch

[rkern]

I get the following crash when executing a %macro:

[~]
|1> print 10
10

[~]
|2> %macro mm 1
Macro `mm` created. To execute, type its name (without quotes).
Macro contents:
print 10


[~]
|3> mm
--> mm()
---------------------------------------------------------------------------
IntegrityError                            Traceback (most recent call last)
/Users/rkern/<ipython-input-3-a39e8f58efe3> in <module>()
----> 1 mm()

/Users/rkern/git/ipython/IPython/core/macro.pyc in __call__(self, *args)
     33         IPython.utils.io.Term.cout.flush()
     34         self._ip.user_ns['_margv'] = args
---> 35         self._ip.run_cell(self.value)
     36 
     37     def __getstate__(self):

/Users/rkern/git/ipython/IPython/core/interactiveshell.pyc in run_cell(self, cell)
   2100 
   2101         # Store raw and processed history

-> 2102         self.history_manager.store_inputs(self.execution_count, ipy_cell, cell)
   2103 
   2104         self.logger.log(ipy_cell, cell)

/Users/rkern/git/ipython/IPython/core/history.pyc in store_inputs(self, line_num, source, source_raw)
    274         # Trigger to flush cache and write to DB.

    275         if len(self.db_input_cache) >= self.db_cache_size:
--> 276             self.writeout_cache()
    277 
    278         # update the auto _i variables


/Users/rkern/git/ipython/IPython/core/history.pyc in writeout_cache(self)
    304         with self.db:
    305             self.db.executemany("INSERT INTO history VALUES (?, ?, ?, ?)",
--> 306                                 self.db_input_cache)
    307             self.db.executemany("INSERT INTO output_history VALUES (?, ?, ?)",
    308                                 self.db_output_cache)

IntegrityError: columns session, line are not unique

When I try to view the history using %hist immediately after this, I get a full IPython crash with an IntegrityError in the same place.

[takluyver]

It seems that, when you run a macro, both the command calling the macro and the macro itself are stored in history. The database expects a unique line number for each entry, but these are passed with the same line number (shell.execution_count). I think we should only store the macro call, not the content of the macro. I'm not familiar with the various shell.run_* methods, but would it make sense to switch %macro to something like run_source, rather than run_cell?

Nearer to release, it probably makes sense for store_inputs to catch IntegrityError so that it can fail more gracefully. I've left it uncaught for now so that we can pick up on errors like this one.

[rkern]

It might be best for the prefilter mechanism to expand Macro objects explicitly instead of using the IPyAutocall mechanism. Then the history will contain both the raw macro execution and the expanded macro implementation automatically.

[takluyver]

That's not a bad idea. I'll look into it tomorrow.

[takluyver]

It's working a bit. The trouble is that IPython determines that a cell contains a single block before it does the prefiltering. So the macro command gets expanded, but then compiled in single line mode, and only the first line of the macro is executed. Is there an easy way round this, or would it involve substantial changes to IPython's execution machinery?

[rkern]

It looks like a lot of the logic for prefiltering and transformation has been split up (or simply duplicated) between the PrefilterManager and the InputSplitter. I wouldn't be surprised if there's a bunch of dead code in the prefilter module.

I'll think on it.

[takluyver]

The original bug is fixed, and the contents of a macro are now displayed in the translated version of the history. There may well still be duplicate code in prefilter.py and inputsplitter.py - I didn't try to audit them.