Remove duplicates when refilling readline history

[takluyver]

Readline by default only keeps one copy of consecutive duplicates in history, but when we refill it from the history database, we push those duplicates multiple times. It should be an easy fix in refill_readline_hist.

[antoine-levitt]

Are you saying that the way to fix this is to put duplicate lines in the history, but not refill readline's? Isn't it better to simply not add them in the first place?

[takluyver]

Our history is supposed to be a record of what you've executed - for instance, running a += 1 three times is different from running it once. Also, it's numbered to correspond with the In [1]: prompt numbers you see in a session, so one history entry is stored each time that number is incremented.

[minrk]

Right - we do not want to de-dupe the actual record of your actions, because then it is no longer accurate, and breaks reproducibility. We only want to de-dupe readline, so that we skip repeating entries when hitting the up arrow, etc.

[fperez]

Agreed. In fact, it used to work without the dupes a while back, I'm not exactly sure when it changed. Was it when we introduced the sqlite backend or later, do you guys know?

[takluyver]

I don't think I accounted for it when I did the sqlite stuff initially, so my guess would be then. I can reproduce it in 0.11.

PR coming up in a few minutes.

[fperez]

Closed by #1003, I forgot to put that in the commit comment.

[dashesy]

I see duplicates when going back in history in 3.0.0 (QtConsole), it is not something that can be reproduced with confidence but anyone using IPython for long enough time should have noticed this. Maybe QtConsole is not used as much and thus not get tested like the Notebook?