Crash when using %macros in sqlite-history branch #287

rkern opened this Issue Mar 7, 2011 · 6 comments


None yet

2 participants

rkern commented Mar 7, 2011

I get the following crash when executing a %macro:

|1> print 10

|2> %macro mm 1
Macro `mm` created. To execute, type its name (without quotes).
Macro contents:
print 10

|3> mm
--> mm()
IntegrityError                            Traceback (most recent call last)
/Users/rkern/<ipython-input-3-a39e8f58efe3> in <module>()
----> 1 mm()

/Users/rkern/git/ipython/IPython/core/macro.pyc in __call__(self, *args)
     34         self._ip.user_ns['_margv'] = args
---> 35         self._ip.run_cell(self.value)
     37     def __getstate__(self):

/Users/rkern/git/ipython/IPython/core/interactiveshell.pyc in run_cell(self, cell)
   2101         # Store raw and processed history

-> 2102         self.history_manager.store_inputs(self.execution_count, ipy_cell, cell)
   2104         self.logger.log(ipy_cell, cell)

/Users/rkern/git/ipython/IPython/core/history.pyc in store_inputs(self, line_num, source, source_raw)
    274         # Trigger to flush cache and write to DB.

    275         if len(self.db_input_cache) >= self.db_cache_size:
--> 276             self.writeout_cache()
    278         # update the auto _i variables

/Users/rkern/git/ipython/IPython/core/history.pyc in writeout_cache(self)
    304         with self.db:
    305             self.db.executemany("INSERT INTO history VALUES (?, ?, ?, ?)",
--> 306                                 self.db_input_cache)
    307             self.db.executemany("INSERT INTO output_history VALUES (?, ?, ?)",
    308                                 self.db_output_cache)

IntegrityError: columns session, line are not unique

When I try to view the history using %hist immediately after this, I get a full IPython crash with an IntegrityError in the same place.


It seems that, when you run a macro, both the command calling the macro and the macro itself are stored in history. The database expects a unique line number for each entry, but these are passed with the same line number (shell.execution_count). I think we should only store the macro call, not the content of the macro. I'm not familiar with the various shell.run_* methods, but would it make sense to switch %macro to something like run_source, rather than run_cell?

Nearer to release, it probably makes sense for store_inputs to catch IntegrityError so that it can fail more gracefully. I've left it uncaught for now so that we can pick up on errors like this one.

rkern commented Mar 8, 2011

It might be best for the prefilter mechanism to expand Macro objects explicitly instead of using the IPyAutocall mechanism. Then the history will contain both the raw macro execution and the expanded macro implementation automatically.


That's not a bad idea. I'll look into it tomorrow.


It's working a bit. The trouble is that IPython determines that a cell contains a single block before it does the prefiltering. So the macro command gets expanded, but then compiled in single line mode, and only the first line of the macro is executed. Is there an easy way round this, or would it involve substantial changes to IPython's execution machinery?

rkern commented Mar 9, 2011

It looks like a lot of the logic for prefiltering and transformation has been split up (or simply duplicated) between the PrefilterManager and the InputSplitter. I wouldn't be surprised if there's a bunch of dead code in the prefilter module.

I'll think on it.


The original bug is fixed, and the contents of a macro are now displayed in the translated version of the history. There may well still be duplicate code in and - I didn't try to audit them.

@takluyver takluyver closed this Mar 28, 2011
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment