HISTIGNORE for IPython #943

Closed
antoine-levitt opened this Issue Oct 29, 2011 · 16 comments

Projects

None yet

2 participants

@antoine-levitt

There's a HISTIGNORE env variable in bash that makes the history ignore certain commands (a common value is to ignore ls / cd commands). Could something like this be added to IPython?

I also have another reason to use such as setting: I'm working on some emacs integration code, and would like to send commands to ipython (typically: run file.py) without it cluttering the history.

@takluyver
Member

One way of doing that might be to use a separate profile when connecting from emacs - history is stored per profile, so commands in one profile won't show up in the other.

If that's not a good solution:

Which kind of history are you concerned about? There's the readline history, which is what you get when you hit the up arrow in the terminal, and the history database, which is what's queried when you use magic commands like %hist or %save.

Also, how flexible would it need to be for your purposes? I'm imagining checking the first word of a command against a set of ignored commands - would that do what you need.

@antoine-levitt

The idea is to have ipython run from emacs, and bindings in python files would send the file to the ipython process. I still want a history, but don't want code run by emacs (like "run file.py") to show up in the history. It's a marginal problem anyway, but still.

I took for granted that the readline and database history were synced, is that not the case? I'm more concerned with readline.

I don't need a lot of flexibility for my purpose, but there's no point in destroying flexibility: ipython could very well expose a variable the user could set to a function that takes the command line as parameter, and decide whether to ignore it or not. Users could set it as they wish. I've got an emacs mindset for these things: if it can be customized, expose a setting so that users can plug their own code in it. (fortunately, in python as in emacs lisp, "expose" simply means "create a variable for it").

@takluyver
Member

I'm not familiar with how emacs integration works, but if it's happening via
the IPython API, you can call ip.run_cell with the parameter
store_history=False. In fact, that's the default in trunk (but not in 0.11).
If you're using the two-process ZMQ messaging, the equivalent field is
'silent' on the execute_request.

readline history works separately within a session, but if you close IPython
and start it again, readline history is restored from the database. But
readline history would normally only pick up what you're actually entering
at a terminal, so if you run code via API calls, they shouldn't be in
readline history at all until you start a new session (when they are loaded
from the database, if they were run with store_history=True).

You might also want to have a look at what Paul Ivanov has done for vim
integration using the two-process model:
https://github.com/ivanov/vim-ipython .

@antoine-levitt

30/10/11 00:49, Thomas

I'm not familiar with how emacs integration works, but if it's happening via
the IPython API, you can call ip.run_cell with the parameter
store_history=False. In fact, that's the default in trunk (but not in
0.11).

Right now, the ipython integration is just sending input to an ipython
process. So calling ip.run_cell will log the call anyway.

If you're using the two-process ZMQ messaging, the equivalent field is
'silent' on the execute_request.

readline history works separately within a session, but if you close IPython
and start it again, readline history is restored from the database. But
readline history would normally only pick up what you're actually entering
at a terminal, so if you run code via API calls, they shouldn't be in
readline history at all until you start a new session (when they are loaded
from the database, if they were run with store_history=True).

You might also want to have a look at what Paul Ivanov has done for vim
integration using the two-process model:
https://github.com/ivanov/vim-ipython .

Ok so here's the situation. Emacs basically runs a terminal emulator,
which runs ipython. I'd like to be able, in a file, to send a file, ie
to execute "run file.py". Right now, this is done by sending the "run
file.py" to the stdin of ipython.

Ideally, that "run file.py" would not echo in the ipython session, nor
would it be stored in history. But the output of "file.py" would still
be visible in the ipython session (in particular, I don't want any
multithreading). From what I see, the problem with vim-python is that it
doesn't run a true ipython session, it just emulates it by displaying
back the result. Also, the codes are run in separate threads. I think
I'll just carry on with the way things are and not ask too many
questions :-)

@antoine-levitt

Ok, another issue is that ipython records duplicate and blank entries. Typically, for matlab-style operation with pylab, you'd often run a script many times, maybe changing a parameter inside it, and then plot the result, running the script again many times, and plotting again. This requires sifting through many "run script.py" before finding that "plot(x,y,options)" line. See also htty/htty@2173fd7

A nice generic solution would be to have a function someplace that gets called to determine whether the current input should be inserted in the history. This function could answer False if the line is the same as the previous line, or a blank line, as well as call a user function (settable from ipython_config.py). This is complicated further by multiline inputs, the various frontends and the fact that readline and ipython history have to be synchronised. I was able to do it by tweaking the "interact" function of TerminalInteractiveShell, but only for ipython history, not readline. Could someone take a look at this ?

@takluyver
Member

Readline stores its own history directly from stdin, and we haven't got any direct control over that. We can modify it later, but it's somewhat awkward, and I think it's probably not worth doing without a pretty good reason.

Just testing in a terminal here, readline only stores consecutive duplicate entries once, so if you %run script.py 20 times, you only see it once as you press up-arrow to go through previous commands.

If you're just using a terminal emulator, there's not really any alternative to sending %run xyz to stdin. The new ZMQ framework can do what you're after - run the code without it appearing as an input - but it would clearly require a very different architecture. If there's a Qt version of Emacs, it might be possible to get most of the architecture for free by embedding the Qt console.

@antoine-levitt

30/10/11 19:21, Thomas

Readline stores its own history directly from stdin, and we haven't
got any direct control over that. We can modify it later, but it's
somewhat awkward, and I think it's probably not worth doing without a
pretty good reason.

The multiline history thing does it. I think it's worth it to avoid
duplicates/blanks/whatever the user wants to avoid, but I'd understand
if you feel it's just not that worth it for most users.

Just testing in a terminal here, readline only stores consecutive
duplicate entries once, so if you %run script.py 20 times, you only
see it once as you press up-arrow to go through previous commands.

Really? Hm. I see it appearing as many times as I entered the
command. However, it appears only once after I exit and run ipython
again, suggesting only the readline history if affected. That's using
the standard conf (no qtconsole), from ipython github, readline
5.2-9. What version are you using?

@takluyver
Member

I've just checked with trunk, and it seems we've broken it since 0.11. I'll file a new issue for that - I've got an idea about what might have caused it. For reference, I've got readline 6.2-2.

There are some things where it's worth dealing with readline history. When we removed the multiline history feature for 0.11, we had a number of users vociferously calling for its return. I certainly want to sort out the regression in duplicates and blanks, but I my feeling is that there isn't enough demand for configurable ignoring of history lines to merit adding that complexity.

@antoine-levitt

30/10/11 23:06, Thomas

I've just checked with trunk, and it seems we've broken it since
0.11. I'll file a new issue for that - I've got an idea about what
might have caused it. For reference, I've got readline 6.2-2.

There are some things where it's worth dealing with readline
history. When we removed the multiline history feature for 0.11, we
had a number of users vociferously calling for its return. I certainly
want to sort out the regression in duplicates and blanks, but I my
feeling is that there isn't enough demand for configurable ignoring of
history lines to merit adding that complexity.

Alright, that's fine. I'll look out for your fix, then see if I can
implement a trivial hack for my personal use case, and if not, just let
it go :) Thanks for your time!

@takluyver
Member

The blanks problem should be fixed by the open pull request #929, and I've commented there asking Julian to fix the problem with duplicates as well.

Thanks for what you're doing - we've had quite a few people looking for better Emacs bindings for IPython, so it should be well received. Our existing ipython.el file is very out of date, because none of the core developers know emacs lisp.

@fperez: just to keep you updated, I felt that there wouldn't be much demand for this feature, at least once we fix some regressions in readline history, so it doesn't warrant adding complexity. Feel free to reopen if you disagree.

@takluyver takluyver closed this Oct 30, 2011
@antoine-levitt

For the record, here's a very simple code you can put in your init scripts to make ipython (at least, the readline part) forget all about lines satisfying a certain predicate. It's so hackish it's almost like emacs lisp. I'm starting to love python. :-)

@antoine-levitt
# our history_ignore predicate
history_ignore = lambda str: str.startswith("run")

# hijack the readline.add_history method
get_ipython().readline.old_add_history = get_ipython().readline.add_history
def my_add_history(str):
    if not history_ignore(str):
        get_ipython().readline.old_add_history(str)

get_ipython().readline.add_history = my_add_history

#force a readline history reload
get_ipython().readline.clear_history()
get_ipython().refill_readline_hist()
@takluyver
Member

Feel free to add it to the cookbook: http://wiki.ipython.org/Cookbook

Note that your code depends on us always removing and re-adding items in readline history. So it won't work if the user has multiline_history set to False, or if we decide to skip doing that dance for single-line cells.

@antoine-levitt

Not really, it only depends on refill_readline_hist working correctly, no matter if it's multiline or single-line. refill_readline_hist exists because you need a persistent history across sessions. I'll add it to the cookbook with a big warning sign.

@takluyver
Member

But to keep working, you'd need to call refill_readline_hist after every input, which is wildly inefficient (it reads 1000 entries from a database, loops over them, encodes them, and feeds them to readline). I'm sure it should be possible to use something like get_ipython().register_post_execute, and inspect readline history. It might be tricky to do reliably (multiline cells, raw_input()s inside the cell...), but a rough implementation should be simple enough.

@antoine-levitt

Ah, true. I guess I should look at how bash does things for its
HISTIGNORE feature (I guess it does just that, inspect the last line and
remove it if it should be). Oh well, works fine for now. :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment