-HistoryNoDuplicates does not have any effect

[joonro]

I have Set-PSReadLineOption -HistoryNoDuplicates in my profile file, and Get-PSReadlineOption shows HistoryNoDuplicates : True. But duplicated entries are being added to the ConsoleHost_history.txt file. I'm using version 1.0.0.13. The following is all the options I have for PSReadLine:

Import-Module PSReadLine

Set-PSReadLineOption -HistoryNoDuplicates 
Set-PSReadLineOption -HistorySearchCursorMovesToEnd
Set-PSReadLineOption -HistorySaveStyle SaveIncrementally
Set-PSReadLineOption -MaximumHistoryCount 4000
# history substring search
Set-PSReadlineKeyHandler -Key UpArrow -Function HistorySearchBackward
Set-PSReadlineKeyHandler -Key DownArrow -Function HistorySearchForward

# Tab completion
Set-PSReadlineKeyHandler -Chord 'Shift+Tab' -Function Complete
Set-PSReadlineKeyHandler -Key Tab -Function MenuComplete

# automatic quotation
Set-PSReadlineKeyHandler -Chord 'Oem7','Shift+Oem7' `
                         -BriefDescription SmartInsertQuote `
                         -LongDescription "Insert paired quotes if not already on a quote" `
                         -ScriptBlock {
    param($key, $arg)

    $line = $null
    $cursor = $null
    [PSConsoleUtilities.PSConsoleReadline]::GetBufferState([ref]$line, [ref]$cursor)

    if ($line[$cursor] -eq $key.KeyChar) {
        # Just move the cursor
        [PSConsoleUtilities.PSConsoleReadline]::SetCursorPosition($cursor + 1)
    }
    else {
        # Insert matching quotes, move cursor to be in between the quotes
        [PSConsoleUtilities.PSConsoleReadline]::Insert("$($key.KeyChar)" * 2)
        [PSConsoleUtilities.PSConsoleReadline]::GetBufferState([ref]$line, [ref]$cursor)
        [PSConsoleUtilities.PSConsoleReadline]::SetCursorPosition($cursor - 1)
    }
}

[lzybkr]

The option is actually working more or less as expected, see issue #74 for a discussion on the current implementation.

That said, when loading history, I've considered ignoring duplicates so that you can load more items before hitting the limit. Is that your concern?

I think I also have a bug where Ctrl+r will recall duplicates but shouldn't. UpArrow will not recall duplicates though.

[joonro]

I see - I was looking at the history file and seeing a lot of duplicates stored, that is why I thought it did not work. But when I actually do the history search, PSReadLine does skip duplicated items. Thank you so much for your quick reply.

That said, when loading history, I've considered ignoring duplicates so that you can load more items before hitting the limit. Is that your concern?

I would prefer to have an option to not store duplicated items in the first place, but I think this would be also great - this was the reason why I was looking the history file in the first place. (I remembered I had one command in the history but PSReadLine did not find it - it was because of the limit).

[vors]

@lzybkr should Set-PSReadLineOption -HistoryNoDuplicates be a default?

[Konstantin-Glukhov]

I think this issue should be re-opened. It is not a big overhead to remove duplicates from the history file on PS session start or end.

[lzybkr]

Shared history relies on the file size to know when to reread history. Trimming history would break that, so another mechanism would be needed to notify other sessions that new history was added.

This would actually be a good thing to add - I just never got around to it.

If I spend any time on history, I'm more likely to replace the simple text file with a sqlite database and would design the schema so duplicates are impossible to add.

[aniketgm]

I don't understand (Or rather would like to understand) what's the issue with not able to get -HistoryNoDuplicates working.
Although, I do not know C# at all, but I believe there must be a Set data structure type which can help remove duplicates (if not from multiple session history, but) from at least one session history.

Also, Instead of trimming the whole file, it can be checked if the command was being run again in the past, let's say within the past 2000 commands. So, If a duplicate command comes again after say 2000 commands, it can be considered fine I think.
Meaning, there should not be any duplicates within the last 2000 commands and if user enters any command which was used before 2000 commands can be allowed again. Now this number 2000 can be made configurable. It's upto the designer and developers.

But, the problem that I still see, for example, is that even if I have say one Windows Terminal console window open. And If I use let's say the ls command more often. I still see the ls multiple times in the history file.