Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

History can't handle empty lines (was: Keep losing history items) #6032

Closed
jdelStrother opened this issue Aug 7, 2019 · 16 comments
Closed

History can't handle empty lines (was: Keep losing history items) #6032

jdelStrother opened this issue Aug 7, 2019 · 16 comments
Assignees
Labels
bug
Milestone

Comments

@jdelStrother
Copy link

@jdelStrother jdelStrother commented Aug 7, 2019

I keep randomly (?) losing history items. Commands that I know I typed in the past - long before my current shell was opened - are suddenly missing from my shell instance (they're not visible in history or history search --contains 'my-missing-command', but are present in ~/.local/share/fish/fish_history).

Opening a new tab or running history --merge brings back the missing command.

(Anecdotally, it seems like the commands that go missing always have a paths: entry in the history file, but I've not found much on how the paths entry affects the history command.)

What can I try to narrow down the problem?


fish, version 3.0.2
OS X 10.14.5
iTerm2 3.3.20190715-nightly with TERM=xterm-256color

Please tell us if you tried fish without third-party customizations by executing this command and whether it affected the behavior you are reporting:

I didn't, I haven't found a good way of simply reproducing the problem.

@faho
Copy link
Member

@faho faho commented Aug 8, 2019

What can I try to narrow down the problem?

The obvious thing is checking any configuration, in particular that touches $fish_history or does a history merge or save or... If you have any third-party stuff like oh-my-fish, that's often of interest because you don't exactly know if it touches this, so disable it.

(Anecdotally, it seems like the commands that go missing always have a paths: entry in the history file, but I've not found much on how the paths entry affects the history command.)

Could we have the offending entries? Ideally without any changes, so if you have a sample that doesn't include private information that'd be nice (we don't need or want to know your bank details), but if not a censored one might also work.

@faho faho added bug needs more info labels Aug 8, 2019
@jdelStrother
Copy link
Author

@jdelStrother jdelStrother commented Aug 8, 2019

I do have these plugins installed via fisher:

jethrokuan/z
barnybug/docker-fish-completion
laughedelic/brew-completions
arbelt/fish-plugin-scmpuff
oh-my-fish/plugin-foreign-env

although grepping through them for history doesn't show anything likely-sounding. I'm also using fzf (https://github.com/junegunn/fzf/blob/master/shell/key-bindings.fish).

Could we have the offending entries?

My latest one is:

- cmd: docker-compose -f docker-compose.services.yml up
  when: 1565184205
  paths:
    - docker-compose.services.yml

I'll carry on trying to narrow it down, although I think I only run into every couple of days so might take a while.

@faho
Copy link
Member

@faho faho commented Aug 8, 2019

My latest one is:

Okay, so the purpose of the paths: attribute is to not show this entry in suggestions (the greyed-out things in the commandline) if the given paths don't exist.

So you'd expect docker-co to suggest mpose -f docker-compose.services.yml up if the current directory has a docker-compose.services.yml, and something else if it doesn't.

Also, if a history entry is selected to be used as a suggestion, it'll be skipped when navigating history (the idea being that you could have picked it before). Are you sure you're not just seeing that behavior?

And since this is a docker example: Are all the others also docker examples? Can you try without the docker-fish-completion plugin?

@jdelStrother
Copy link
Author

@jdelStrother jdelStrother commented Aug 8, 2019

So you'd expect docker-co to suggest mpose -f docker-compose.services.yml up if the current directory has a docker-compose.services.yml

There's definitely a docker-compose.services.yml file in my current directory.

And since this is a docker example: Are all the others also docker examples? Can you try without the docker-fish-completion plugin?

It's not just docker, but I'll try removing the plugin and start keeping a note of the other commands that seem to frequently go missing.


Something I missed from my original report that seems weird/interesting: I pretty much also use history via fzf (hitting Ctrl-R to invoke it). For a while I assumed it was a fzf bug, but then noticed that the same command is missing from history. However, the missing command does show up as an auto-suggestion if I start to type docker-com...

@faho
Copy link
Member

@faho faho commented Aug 8, 2019

However, the missing command does show up as an auto-suggestion if I start to type docker-com...

Okay, that is indeed weird.

@ridiculousfish Any ideas? Is it possible we're skipping these too hard?

@ridiculousfish
Copy link
Member

@ridiculousfish ridiculousfish commented Aug 10, 2019

I'm at a loss too. It's very hard to think of a reason why it would not be in history but would be offered as a completion.

One remote possibility is that we're rejecting these based on timestamp - we think they're commands from a new session. That could happen if the clock were being set back.

Another possibility is that there's a bug in the history decoding code causing us to fail to decode certain items.

This is the sort of place where we would want a --debug flag to trace what's happening.

@zanchey
Copy link
Member

@zanchey zanchey commented Aug 11, 2019

There was some suggestion in #5136 (comment) that the hardware clock being reset may have caused a similar behaviour.

@mqudsi
Copy link
Contributor

@mqudsi mqudsi commented Sep 7, 2019

@jdelStrother if you are able to reproduce this with a clean (free of sensitive data!) history file (just temporarily point fish's history to a new location) and can induce this behavior by using that file in a future fish session, perhaps you could share that via email or similar with one of us to attempt to debug the issue.

@jaelpark
Copy link

@jaelpark jaelpark commented Sep 7, 2019

The problem occurs when sometimes fish writes an empty command entry to the history. See the following history file as an example, and particularly line 23.

- cmd: sudo pacman -Syu --ignore rofi --ignore root
  when: 1567551190
- cmd: cd build/vim-x11/
  when: 1567551211
  paths:
    - build/vim-x11/
- cmd: vim PKGBUILD
  when: 1567551217
  paths:
    - PKGBUILD
- cmd: cat PKGBUILD
  when: 1567553811
  paths:
    - PKGBUILD
- cmd: makepkg -Acsr
  when: 1567553817
- cmd: sudo pacman -U vim-x11-8.1.1968-1-x86_64.pkg.tar.xz
  when: 1567553908
  paths:
    - vim-x11-8.1.1968-1-x86_64.pkg.tar.xz
- cmd: r
  when: 1567584641
- cmd: 
  when: 1567584666
- cmd: r
  when: 1567584667
- cmd: r
  when: 1567882393

If I add anything to the empty entry, the full history becomes available again.

@ridiculousfish
Copy link
Member

@ridiculousfish ridiculousfish commented Sep 8, 2019

Wild and nice detective work! Any guess on how the empty command got in there? Do you think you entered and executed an empty command or is something else happening?

@jaelpark
Copy link

@jaelpark jaelpark commented Sep 8, 2019

It looks like the offending entry gets written if I manage to put a line return to the prompt without fish registering it. One way to reliably reproduce this is to put fish into vi mode, enter normal mode, and immediately after quickly hit enter to insert the line return. Press enter again to evaluate the empty command. I usually run the previous command by entering normal mode, pressing 'k' and then enter. But sometimes it happens that I "miss" the 'k' key, and only enter is pushed when I quickly attempt this 3-key combo. I think this is why I sometimes randomly lose the command history.

@jdelStrother
Copy link
Author

@jdelStrother jdelStrother commented Sep 9, 2019

I'm still not able to reproduce this at-will (and not using vi mode, FWIW). I'll look for empty entries next time this crops up, though.

@jaelpark
Copy link

@jaelpark jaelpark commented Sep 9, 2019

Another way actually is to enter a single quote character, add two new lines, and then navigate back up and remove the quote. One line break should remain, and evaluating this will result in an empty command.

Edit: So the history might get erased for example when user starts typing a command, enters a quote and adds a new line, but then decides to erase the prompt while ignoring the new line that was also inserted. The new line will remain and gets evaluated when user hits enter for a new prompt line.

@ridiculousfish
Copy link
Member

@ridiculousfish ridiculousfish commented Sep 9, 2019

That reproduces for me. Amazing!

@ridiculousfish ridiculousfish self-assigned this Sep 9, 2019
@floam floam added this to the fish-future milestone Sep 18, 2019
@zanchey
Copy link
Member

@zanchey zanchey commented Sep 25, 2019

I can reproduce with:

Alt-Enter; Enter

and fix with

history --delete '
'

@faho faho changed the title Keep losing history items History can't handle empty lines (was: Keep losing history items) Dec 31, 2019
@ridiculousfish
Copy link
Member

@ridiculousfish ridiculousfish commented Jan 5, 2020

Let's do a targeted fix for 3.1 and fix it properly with a sane history format in 3.2.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 16, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug
Projects
None yet
Development

No branches or pull requests

7 participants