Invalid reads in regexp handling code #3150

ZyX-I · 2015-08-09T23:37:16Z

When checking ShaDa code from #2506 I found what I think are not a ShaDa bugs:

In the following test:

  it('dumps&loads /pattern correctly when &encoding /= UTF-8 when dumping',
  function()
    set_additional_cmd('set encoding=latin1')
    reset()
    -- \171 is U+00AB LEFT-POINTING DOUBLE ANGLE QUOTATION MARK in latin1
    nvim_command('silent! /\171/')
    nvim_command('set shada+=/0')
    set_additional_cmd('')
    nvim_command('qall')
    reset()
    nvim_eval('setline(".", ["«\171"])')
    nvim_command('~&')
    eq('\171', nvim_eval('getline(".")'))
    eq('', nvim_eval('histget("/", -1)'))
  end)

valgrind reports:

==5139== Invalid read of size 1
==5139==    at 0x4E6554: utf_ptr2char (mbyte.c:1361)
==5139==    by 0x4D28DD: find_match_text (regexp_nfa.c:4784)
==5139==    by 0x4D6491: nfa_regexec_both (regexp_nfa.c:6276)
==5139==    by 0x4D698D: nfa_regexec_multi (regexp_nfa.c:6493)
==5139==    by 0x4D6DDC: vim_regexec_multi (regexp.c:7124)
==5139==    by 0x576FD9: next_search_hl (screen.c:5597)
==5139==    by 0x56FA9A: win_line (screen.c:2881)
==5139==    by 0x56B4FB: win_update (screen.c:1429)
==5139==    by 0x568C50: update_screen (screen.c:424)
==5139==    by 0x48B1F8: main_loop (main.c:673)
==5139==    by 0x48ADF2: main (main.c:536)
==5139==  Address 0x71cdbc4 is 0 bytes after a block of size 4 alloc'd
==5139==    at 0x4C2A030: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==5139==    by 0x4B36F9: try_malloc (memory.c:62)
==5139==    by 0x4B376C: xmalloc (memory.c:96)
==5139==    by 0x4B396C: xmallocz (memory.c:171)
==5139==    by 0x4B39A5: xmemdupz (memory.c:189)
==5139==    by 0x4B3D9D: xstrdup (memory.c:381)
==5139==    by 0x57B9B2: vim_strsave (strings.c:54)
==5139==    by 0x47B880: ml_replace (memline.c:2337)
==5139==    by 0x5D7FB3: f_setline (eval.c:14381)
==5139==    by 0x5C7C04: call_func (eval.c:7118)
==5139==    by 0x5C76C1: get_func_tv (eval.c:6967)
==5139==    by 0x5C2B43: eval7 (eval.c:4215)
==5139== 
==5139== Invalid read of size 1
==5139==    at 0x4E655F: utf_ptr2char (mbyte.c:1362)
==5139==    by 0x4D28DD: find_match_text (regexp_nfa.c:4784)
==5139==    by 0x4D6491: nfa_regexec_both (regexp_nfa.c:6276)
==5139==    by 0x4D698D: nfa_regexec_multi (regexp_nfa.c:6493)
==5139==    by 0x4D6DDC: vim_regexec_multi (regexp.c:7124)
==5139==    by 0x576FD9: next_search_hl (screen.c:5597)
==5139==    by 0x56FA9A: win_line (screen.c:2881)
==5139==    by 0x56B4FB: win_update (screen.c:1429)
==5139==    by 0x568C50: update_screen (screen.c:424)
==5139==    by 0x48B1F8: main_loop (main.c:673)
==5139==    by 0x48ADF2: main (main.c:536)
==5139==  Address 0x71cdbc4 is 0 bytes after a block of size 4 alloc'd
==5139==    at 0x4C2A030: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==5139==    by 0x4B36F9: try_malloc (memory.c:62)
==5139==    by 0x4B376C: xmalloc (memory.c:96)
==5139==    by 0x4B396C: xmallocz (memory.c:171)
==5139==    by 0x4B39A5: xmemdupz (memory.c:189)
==5139==    by 0x4B3D9D: xstrdup (memory.c:381)
==5139==    by 0x57B9B2: vim_strsave (strings.c:54)
==5139==    by 0x47B880: ml_replace (memline.c:2337)
==5139==    by 0x5D7FB3: f_setline (eval.c:14381)
==5139==    by 0x5C7C04: call_func (eval.c:7118)
==5139==    by 0x5C76C1: get_func_tv (eval.c:6967)
==5139==    by 0x5C2B43: eval7 (eval.c:4215)
==5139==

. This happens in the third run process, meaning that one of the following lines are responsible:

    nvim_eval('setline(".", ["«\171"])')
    nvim_command('~&')
    eq('\171', nvim_eval('getline(".")'))
    eq('', nvim_eval('histget("/", -1)'))

The following log:

==7819== Invalid read of size 1
==7819==    at 0x4E6554: utf_ptr2char (mbyte.c:1361)
==7819==    by 0x4D28DD: find_match_text (regexp_nfa.c:4784)
==7819==    by 0x4D6491: nfa_regexec_both (regexp_nfa.c:6276)
==7819==    by 0x4D698D: nfa_regexec_multi (regexp_nfa.c:6493)
==7819==    by 0x4D6DDC: vim_regexec_multi (regexp.c:7124)
==7819==    by 0x576FD9: next_search_hl (screen.c:5597)
==7819==    by 0x56FA9A: win_line (screen.c:2881)
==7819==    by 0x56B4FB: win_update (screen.c:1429)
==7819==    by 0x568C50: update_screen (screen.c:424)
==7819==    by 0x48B1F8: main_loop (main.c:673)
==7819==    by 0x48ADF2: main (main.c:536)
==7819==  Address 0x71cfd14 is 0 bytes after a block of size 4 alloc'd
==7819==    at 0x4C2A030: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==7819==    by 0x4B36F9: try_malloc (memory.c:62)
==7819==    by 0x4B376C: xmalloc (memory.c:96)
==7819==    by 0x4B396C: xmallocz (memory.c:171)
==7819==    by 0x4B39A5: xmemdupz (memory.c:189)
==7819==    by 0x4B3D9D: xstrdup (memory.c:381)
==7819==    by 0x57B9B2: vim_strsave (strings.c:54)
==7819==    by 0x47B880: ml_replace (memline.c:2337)
==7819==    by 0x5D7FB3: f_setline (eval.c:14381)
==7819==    by 0x5C7C04: call_func (eval.c:7118)
==7819==    by 0x5C76C1: get_func_tv (eval.c:6967)
==7819==    by 0x5C2B43: eval7 (eval.c:4215)
==7819== 
==7819== Invalid read of size 1
==7819==    at 0x4E655F: utf_ptr2char (mbyte.c:1362)
==7819==    by 0x4D28DD: find_match_text (regexp_nfa.c:4784)
==7819==    by 0x4D6491: nfa_regexec_both (regexp_nfa.c:6276)
==7819==    by 0x4D698D: nfa_regexec_multi (regexp_nfa.c:6493)
==7819==    by 0x4D6DDC: vim_regexec_multi (regexp.c:7124)
==7819==    by 0x576FD9: next_search_hl (screen.c:5597)
==7819==    by 0x56FA9A: win_line (screen.c:2881)
==7819==    by 0x56B4FB: win_update (screen.c:1429)
==7819==    by 0x568C50: update_screen (screen.c:424)
==7819==    by 0x48B1F8: main_loop (main.c:673)
==7819==    by 0x48ADF2: main (main.c:536)
==7819==  Address 0x71cfd14 is 0 bytes after a block of size 4 alloc'd
==7819==    at 0x4C2A030: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==7819==    by 0x4B36F9: try_malloc (memory.c:62)
==7819==    by 0x4B376C: xmalloc (memory.c:96)
==7819==    by 0x4B396C: xmallocz (memory.c:171)
==7819==    by 0x4B39A5: xmemdupz (memory.c:189)
==7819==    by 0x4B3D9D: xstrdup (memory.c:381)
==7819==    by 0x57B9B2: vim_strsave (strings.c:54)
==7819==    by 0x47B880: ml_replace (memline.c:2337)
==7819==    by 0x5D7FB3: f_setline (eval.c:14381)
==7819==    by 0x5C7C04: call_func (eval.c:7118)
==7819==    by 0x5C76C1: get_func_tv (eval.c:6967)
==7819==    by 0x5C2B43: eval7 (eval.c:4215)
==7819==

for

  it('dumps&loads s/pattern correctly when &encoding /= UTF-8 when dumping',
  function()
    set_additional_cmd('set encoding=latin1')
    reset()
    -- \171 is U+00AB LEFT-POINTING DOUBLE ANGLE QUOTATION MARK in latin1
    nvim_command('substitute/\171/./ge')
    set_additional_cmd('')
    nvim_command('qall')
    reset()
    nvim_eval('setline(".", ["«\171"])')
    nvim_command('&')
    eq('.\171', nvim_eval('getline(".")'))
  end)

. Also in the third process, so responsible should be

    nvim_eval('setline(".", ["«\171"])')
    nvim_command('&')
    eq('.\171', nvim_eval('getline(".")'))

What should happen in those tests:

nvim_eval('setline(".", ["«\171"])') in both cases sets current line to "\xC2\xAB\xAB".
nvim_command('~&') in first test runs something like substitute/«//, as well as nvim_command('&') from the second (but in the second case there is a dot in replacement).

I was able to reproduce this with the following input:

 zyx  …/c/neovim/build  valgrind --show-leak-kinds=definite,indirect --leak-check=full --read-var-info=yes --track-origins=yes --num-callers=50 --trace-children=no --log-file=valgrind.log --tool=memcheck bin/nvim -u NONE -i NONE -N --cmd 'call setline(".", ["«\xAB"])' --cmd 'substitute/«/./' -s <(echo :cquit)
 zyx  …/c/neovim/build  cat valgrind.log
==5680== Memcheck, a memory error detector
==5680== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==5680== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==5680== Command: bin/nvim -u NONE -i NONE -N --cmd call\ setline(".",\ ["«\\xAB"]) --cmd substitute/«/./ -s /proc/self/fd/11
==5680== Parent PID: 7981
==5680==
==5680== Invalid read of size 1
==5680==    at 0x4E6554: utf_ptr2char (mbyte.c:1361)
==5680==    by 0x4D28DD: find_match_text (regexp_nfa.c:4784)
==5680==    by 0x4D6491: nfa_regexec_both (regexp_nfa.c:6276)
==5680==    by 0x4D698D: nfa_regexec_multi (regexp_nfa.c:6493)
==5680==    by 0x4D6DDC: vim_regexec_multi (regexp.c:7124)
==5680==    by 0x576FD9: next_search_hl (screen.c:5597)
==5680==    by 0x56EB01: win_line (screen.c:2566)
==5680==    by 0x56B4FB: win_update (screen.c:1429)
==5680==    by 0x568C50: update_screen (screen.c:424)
==5680==    by 0x48B1F8: main_loop (main.c:673)
==5680==    by 0x48ADF2: main (main.c:536)
==5680==  Address 0x6e48a83 is 0 bytes after a block of size 3 alloc'd
==5680==    at 0x4C2A030: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==5680==    by 0x4B36F9: try_malloc (memory.c:62)
==5680==    by 0x4B376C: xmalloc (memory.c:96)
==5680==    by 0x4B396C: xmallocz (memory.c:171)
==5680==    by 0x4B39A5: xmemdupz (memory.c:189)
==5680==    by 0x4B3D9D: xstrdup (memory.c:381)
==5680==    by 0x57B9B2: vim_strsave (strings.c:54)
==5680==    by 0x47B880: ml_replace (memline.c:2337)
==5680==    by 0x451D23: do_sub (ex_cmds.c:3648)
==5680==    by 0x43A91E: do_one_cmd (ex_docmd.c:2196)
==5680==    by 0x4374BB: do_cmdline (ex_docmd.c:622)
==5680==    by 0x436B39: do_cmdline_cmd (ex_docmd.c:294)
==5680==    by 0x48D74E: exe_pre_commands (main.c:1759)
==5680==    by 0x48A978: main (main.c:336)
==5680==
==5680== Invalid read of size 1
==5680==    at 0x4E655F: utf_ptr2char (mbyte.c:1362)
==5680==    by 0x4D28DD: find_match_text (regexp_nfa.c:4784)
==5680==    by 0x4D6491: nfa_regexec_both (regexp_nfa.c:6276)
==5680==    by 0x4D698D: nfa_regexec_multi (regexp_nfa.c:6493)
==5680==    by 0x4D6DDC: vim_regexec_multi (regexp.c:7124)
==5680==    by 0x576FD9: next_search_hl (screen.c:5597)
==5680==    by 0x56EB01: win_line (screen.c:2566)
==5680==    by 0x56B4FB: win_update (screen.c:1429)
==5680==    by 0x568C50: update_screen (screen.c:424)
==5680==    by 0x48B1F8: main_loop (main.c:673)
==5680==    by 0x48ADF2: main (main.c:536)
==5680==  Address 0x6e48a83 is 0 bytes after a block of size 3 alloc'd
==5680==    at 0x4C2A030: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==5680==    by 0x4B36F9: try_malloc (memory.c:62)
==5680==    by 0x4B376C: xmalloc (memory.c:96)
==5680==    by 0x4B396C: xmallocz (memory.c:171)
==5680==    by 0x4B39A5: xmemdupz (memory.c:189)
==5680==    by 0x4B3D9D: xstrdup (memory.c:381)
==5680==    by 0x57B9B2: vim_strsave (strings.c:54)
==5680==    by 0x47B880: ml_replace (memline.c:2337)
==5680==    by 0x451D23: do_sub (ex_cmds.c:3648)
==5680==    by 0x43A91E: do_one_cmd (ex_docmd.c:2196)
==5680==    by 0x4374BB: do_cmdline (ex_docmd.c:622)
==5680==    by 0x436B39: do_cmdline_cmd (ex_docmd.c:294)
==5680==    by 0x48D74E: exe_pre_commands (main.c:1759)
==5680==    by 0x48A978: main (main.c:336)
==5680==
==5680==
==5680== HEAP SUMMARY:
==5680==     in use at exit: 25,897 bytes in 43 blocks
==5680==   total heap usage: 1,465 allocs, 1,422 frees, 6,528,724 bytes allocated
==5680==
==5680== LEAK SUMMARY:
==5680==    definitely lost: 0 bytes in 0 blocks
==5680==    indirectly lost: 0 bytes in 0 blocks
==5680==      possibly lost: 0 bytes in 0 blocks
==5680==    still reachable: 25,897 bytes in 43 blocks
==5680==         suppressed: 0 bytes in 0 blocks
==5680== Reachable blocks (those to which a pointer was found) are not shown.
==5680== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==5680==
==5680== For counts of detected and suppressed errors, rerun with: -v
==5680== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)

Note: problem is not triggered when

I use --cmd cquit to quit.
I use -c cquit to quit.

Quiting with -s <(echo :cquit) is done for this reason.

The text was updated successfully, but these errors were encountered:

ZyX-I · 2015-08-10T05:13:58Z

This has something to do with screen because I can replace -s <(echo :cquit) with --cmd 'redraw!' --cmd cquit:

==14106== Memcheck, a memory error detector
==14106== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==14106== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==14106== Command: bin/nvim -u NONE -i NONE -N --cmd call\ setline(".",\ ["«\\xAB"]) --cmd substitute/«/./ --cmd redraw! --cmd cquit
==14106== Parent PID: 7981
==14106== 
==14106== Invalid read of size 1
==14106==    at 0x4E6554: utf_ptr2char (mbyte.c:1361)
==14106==    by 0x4D28DD: find_match_text (regexp_nfa.c:4784)
==14106==    by 0x4D6491: nfa_regexec_both (regexp_nfa.c:6276)
==14106==    by 0x4D698D: nfa_regexec_multi (regexp_nfa.c:6493)
==14106==    by 0x4D6DDC: vim_regexec_multi (regexp.c:7124)
==14106==    by 0x576FD9: next_search_hl (screen.c:5597)
==14106==    by 0x56EB01: win_line (screen.c:2566)
==14106==    by 0x56B4FB: win_update (screen.c:1429)
==14106==    by 0x568C50: update_screen (screen.c:424)
==14106==    by 0x445F08: ex_redraw (ex_docmd.c:7337)
==14106==    by 0x43A91E: do_one_cmd (ex_docmd.c:2196)
==14106==    by 0x4374BB: do_cmdline (ex_docmd.c:622)
==14106==    by 0x436B39: do_cmdline_cmd (ex_docmd.c:294)
==14106==    by 0x48D74E: exe_pre_commands (main.c:1759)
==14106==    by 0x48A978: main (main.c:336)
==14106==  Address 0x6e48623 is 0 bytes after a block of size 3 alloc'd
==14106==    at 0x4C2A030: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==14106==    by 0x4B36F9: try_malloc (memory.c:62)
==14106==    by 0x4B376C: xmalloc (memory.c:96)
==14106==    by 0x4B396C: xmallocz (memory.c:171)
==14106==    by 0x4B39A5: xmemdupz (memory.c:189)
==14106==    by 0x4B3D9D: xstrdup (memory.c:381)
==14106==    by 0x57B9B2: vim_strsave (strings.c:54)
==14106==    by 0x47B880: ml_replace (memline.c:2337)
==14106==    by 0x451D23: do_sub (ex_cmds.c:3648)
==14106==    by 0x43A91E: do_one_cmd (ex_docmd.c:2196)
==14106==    by 0x4374BB: do_cmdline (ex_docmd.c:622)
==14106==    by 0x436B39: do_cmdline_cmd (ex_docmd.c:294)
==14106==    by 0x48D74E: exe_pre_commands (main.c:1759)
==14106==    by 0x48A978: main (main.c:336)
==14106== 
==14106== Invalid read of size 1
==14106==    at 0x4E655F: utf_ptr2char (mbyte.c:1362)
==14106==    by 0x4D28DD: find_match_text (regexp_nfa.c:4784)
==14106==    by 0x4D6491: nfa_regexec_both (regexp_nfa.c:6276)
==14106==    by 0x4D698D: nfa_regexec_multi (regexp_nfa.c:6493)
==14106==    by 0x4D6DDC: vim_regexec_multi (regexp.c:7124)
==14106==    by 0x576FD9: next_search_hl (screen.c:5597)
==14106==    by 0x56EB01: win_line (screen.c:2566)
==14106==    by 0x56B4FB: win_update (screen.c:1429)
==14106==    by 0x568C50: update_screen (screen.c:424)
==14106==    by 0x445F08: ex_redraw (ex_docmd.c:7337)
==14106==    by 0x43A91E: do_one_cmd (ex_docmd.c:2196)
==14106==    by 0x4374BB: do_cmdline (ex_docmd.c:622)
==14106==    by 0x436B39: do_cmdline_cmd (ex_docmd.c:294)
==14106==    by 0x48D74E: exe_pre_commands (main.c:1759)
==14106==    by 0x48A978: main (main.c:336)
==14106==  Address 0x6e48623 is 0 bytes after a block of size 3 alloc'd
==14106==    at 0x4C2A030: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==14106==    by 0x4B36F9: try_malloc (memory.c:62)
==14106==    by 0x4B376C: xmalloc (memory.c:96)
==14106==    by 0x4B396C: xmallocz (memory.c:171)
==14106==    by 0x4B39A5: xmemdupz (memory.c:189)
==14106==    by 0x4B3D9D: xstrdup (memory.c:381)
==14106==    by 0x57B9B2: vim_strsave (strings.c:54)
==14106==    by 0x47B880: ml_replace (memline.c:2337)
==14106==    by 0x451D23: do_sub (ex_cmds.c:3648)
==14106==    by 0x43A91E: do_one_cmd (ex_docmd.c:2196)
==14106==    by 0x4374BB: do_cmdline (ex_docmd.c:622)
==14106==    by 0x436B39: do_cmdline_cmd (ex_docmd.c:294)
==14106==    by 0x48D74E: exe_pre_commands (main.c:1759)
==14106==    by 0x48A978: main (main.c:336)
==14106== 
==14106== 
==14106== HEAP SUMMARY:
==14106==     in use at exit: 25,651 bytes in 41 blocks
==14106==   total heap usage: 1,244 allocs, 1,203 frees, 4,413,736 bytes allocated
==14106== 
==14106== LEAK SUMMARY:
==14106==    definitely lost: 0 bytes in 0 blocks
==14106==    indirectly lost: 0 bytes in 0 blocks
==14106==      possibly lost: 0 bytes in 0 blocks
==14106==    still reachable: 25,651 bytes in 41 blocks
==14106==         suppressed: 0 bytes in 0 blocks
==14106== Reachable blocks (those to which a pointer was found) are not shown.
==14106== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==14106== 
==14106== For counts of detected and suppressed errors, rerun with: -v
==14106== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)

ZyX-I · 2015-08-10T05:15:48Z

(Though you can guess this from function names: it is not just “something to do with screen”, but “something to do with &hlsearch support”.)

ZyX-I · 2015-08-10T05:19:53Z

One can use /«/ with line .\xAB to get other errors:

 zyx  …/c/neovim/build  valgrind --show-leak-kinds=definite,indirect --leak-check=full --read-var-info=yes --track-origins=yes --num-callers=50 --trace-children=no --log-file=valgrind.log --tool=memcheck bin/nvim -u NONE -i NONE -N --cmd 'call setline(".", [".\xAB"]) | /«/' --cmd cquit
search hit BOTTOM, continuing at TOP%                                                                                   zyx  …/c/neovim/build  cat valgrind.log
==17355== Memcheck, a memory error detector
==17355== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==17355== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==17355== Command: bin/nvim -u NONE -i NONE -N --cmd call\ setline(".",\ [".\\xAB"])\ |\ /«/ --cmd cquit
==17355== Parent PID: 7981
==17355==
==17355== Invalid read of size 1
==17355==    at 0x4E6554: utf_ptr2char (mbyte.c:1361)
==17355==    by 0x4D28DD: find_match_text (regexp_nfa.c:4784)
==17355==    by 0x4D6491: nfa_regexec_both (regexp_nfa.c:6276)
==17355==    by 0x4D698D: nfa_regexec_multi (regexp_nfa.c:6493)
==17355==    by 0x4D6DDC: vim_regexec_multi (regexp.c:7124)
==17355==    by 0x55EA1B: searchit (search.c:535)
==17355==    by 0x560034: do_search (search.c:1121)
==17355==    by 0x43DBF2: get_address (ex_docmd.c:3522)
==17355==    by 0x439260: do_one_cmd (ex_docmd.c:1556)
==17355==    by 0x4374BB: do_cmdline (ex_docmd.c:622)
==17355==    by 0x436B39: do_cmdline_cmd (ex_docmd.c:294)
==17355==    by 0x48D74E: exe_pre_commands (main.c:1759)
==17355==    by 0x48A978: main (main.c:336)
==17355==  Address 0x6e44d33 is 0 bytes after a block of size 3 alloc'd
==17355==    at 0x4C2A030: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==17355==    by 0x4B36F9: try_malloc (memory.c:62)
==17355==    by 0x4B376C: xmalloc (memory.c:96)
==17355==    by 0x4B396C: xmallocz (memory.c:171)
==17355==    by 0x4B39A5: xmemdupz (memory.c:189)
==17355==    by 0x4B3D9D: xstrdup (memory.c:381)
==17355==    by 0x57B9B2: vim_strsave (strings.c:54)
==17355==    by 0x47B880: ml_replace (memline.c:2337)
==17355==    by 0x5D7D91: f_setline (eval.c:14381)
==17355==    by 0x5C79E2: call_func (eval.c:7118)
==17355==    by 0x5C749F: get_func_tv (eval.c:6967)
==17355==    by 0x5BF7F0: ex_call (eval.c:2786)
==17355==    by 0x43A91E: do_one_cmd (ex_docmd.c:2196)
==17355==    by 0x4374BB: do_cmdline (ex_docmd.c:622)
==17355==    by 0x436B39: do_cmdline_cmd (ex_docmd.c:294)
==17355==    by 0x48D74E: exe_pre_commands (main.c:1759)
==17355==    by 0x48A978: main (main.c:336)
==17355==
==17355== Invalid read of size 1
==17355==    at 0x4E655F: utf_ptr2char (mbyte.c:1362)
==17355==    by 0x4D28DD: find_match_text (regexp_nfa.c:4784)
==17355==    by 0x4D6491: nfa_regexec_both (regexp_nfa.c:6276)
==17355==    by 0x4D698D: nfa_regexec_multi (regexp_nfa.c:6493)
==17355==    by 0x4D6DDC: vim_regexec_multi (regexp.c:7124)
==17355==    by 0x55EA1B: searchit (search.c:535)
==17355==    by 0x560034: do_search (search.c:1121)
==17355==    by 0x43DBF2: get_address (ex_docmd.c:3522)
==17355==    by 0x439260: do_one_cmd (ex_docmd.c:1556)
==17355==    by 0x4374BB: do_cmdline (ex_docmd.c:622)
==17355==    by 0x436B39: do_cmdline_cmd (ex_docmd.c:294)
==17355==    by 0x48D74E: exe_pre_commands (main.c:1759)
==17355==    by 0x48A978: main (main.c:336)
==17355==  Address 0x6e44d33 is 0 bytes after a block of size 3 alloc'd
==17355==    at 0x4C2A030: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==17355==    by 0x4B36F9: try_malloc (memory.c:62)
==17355==    by 0x4B376C: xmalloc (memory.c:96)
==17355==    by 0x4B396C: xmallocz (memory.c:171)
==17355==    by 0x4B39A5: xmemdupz (memory.c:189)
==17355==    by 0x4B3D9D: xstrdup (memory.c:381)
==17355==    by 0x57B9B2: vim_strsave (strings.c:54)
==17355==    by 0x47B880: ml_replace (memline.c:2337)
==17355==    by 0x5D7D91: f_setline (eval.c:14381)
==17355==    by 0x5C79E2: call_func (eval.c:7118)
==17355==    by 0x5C749F: get_func_tv (eval.c:6967)
==17355==    by 0x5BF7F0: ex_call (eval.c:2786)
==17355==    by 0x43A91E: do_one_cmd (ex_docmd.c:2196)
==17355==    by 0x4374BB: do_cmdline (ex_docmd.c:622)
==17355==    by 0x436B39: do_cmdline_cmd (ex_docmd.c:294)
==17355==    by 0x48D74E: exe_pre_commands (main.c:1759)
==17355==    by 0x48A978: main (main.c:336)
==17355==
==17355== Invalid read of size 1
==17355==    at 0x55EC4B: searchit (search.c:609)
==17355==    by 0x560034: do_search (search.c:1121)
==17355==    by 0x43DBF2: get_address (ex_docmd.c:3522)
==17355==    by 0x439260: do_one_cmd (ex_docmd.c:1556)
==17355==    by 0x4374BB: do_cmdline (ex_docmd.c:622)
==17355==    by 0x436B39: do_cmdline_cmd (ex_docmd.c:294)
==17355==    by 0x48D74E: exe_pre_commands (main.c:1759)
==17355==    by 0x48A978: main (main.c:336)
==17355==  Address 0x6e44d33 is 0 bytes after a block of size 3 alloc'd
==17355==    at 0x4C2A030: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==17355==    by 0x4B36F9: try_malloc (memory.c:62)
==17355==    by 0x4B376C: xmalloc (memory.c:96)
==17355==    by 0x4B396C: xmallocz (memory.c:171)
==17355==    by 0x4B39A5: xmemdupz (memory.c:189)
==17355==    by 0x4B3D9D: xstrdup (memory.c:381)
==17355==    by 0x57B9B2: vim_strsave (strings.c:54)
==17355==    by 0x47B880: ml_replace (memline.c:2337)
==17355==    by 0x5D7D91: f_setline (eval.c:14381)
==17355==    by 0x5C79E2: call_func (eval.c:7118)
==17355==    by 0x5C749F: get_func_tv (eval.c:6967)
==17355==    by 0x5BF7F0: ex_call (eval.c:2786)
==17355==    by 0x43A91E: do_one_cmd (ex_docmd.c:2196)
==17355==    by 0x4374BB: do_cmdline (ex_docmd.c:622)
==17355==    by 0x436B39: do_cmdline_cmd (ex_docmd.c:294)
==17355==    by 0x48D74E: exe_pre_commands (main.c:1759)
==17355==    by 0x48A978: main (main.c:336)
==17355==
==17355==
==17355== HEAP SUMMARY:
==17355==     in use at exit: 25,651 bytes in 41 blocks
==17355==   total heap usage: 1,151 allocs, 1,110 frees, 4,404,121 bytes allocated
==17355==
==17355== LEAK SUMMARY:
==17355==    definitely lost: 0 bytes in 0 blocks
==17355==    indirectly lost: 0 bytes in 0 blocks
==17355==      possibly lost: 0 bytes in 0 blocks
==17355==    still reachable: 25,651 bytes in 41 blocks
==17355==         suppressed: 0 bytes in 0 blocks
==17355== Reachable blocks (those to which a pointer was found) are not shown.
==17355== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==17355==
==17355== For counts of detected and suppressed errors, rerun with: -v
==17355== ERROR SUMMARY: 5 errors from 3 contexts (suppressed: 0 from 0)

ZyX-I · 2015-08-10T05:23:59Z

And I do not see these errors in vim-7.4.788 (with my patches: bookmark local-default in https://bitbucket.org/ZyX_I/vim), so maybe just porting some patches from Vim is enough to fix the issue.

ZyX-I · 2015-08-10T05:31:38Z

And, by the way, Vim reports E486: Pattern not found, while NeoVim highlights <ab>. &encoding setting is identical.

justinmk · 2015-08-10T06:19:28Z

something to do with &hlsearch support
[...]
I do not see these errors in vim-7.4.788

We set 'hlsearch' and 'incsearch' by default:

0d6877e
63f5286

Maybe it could be related, just a guess.

ZyX-I · 2015-08-10T16:22:06Z

No.

ZyX-I · 2015-08-10T16:22:56Z

At least, this does not trigger the error in Vim. And NeoVim still does not show E486 while it should.

ZyX-I · 2015-08-10T16:23:49Z

And I still see this error reported when I prepend set nohlsearch noincsearch | to call setline….

jebaum · 2015-08-14T02:07:06Z

Possibly related: junegunn/fzf#207 (comment)

tarruda · 2015-08-14T10:57:31Z

@ZyX-I Is

valgrind --show-leak-kinds=definite,indirect --leak-check=full --read-var-info=yes --track-origins=yes --num-callers=50 --trace-children=no --log-file=valgrind.log --tool=memcheck bin/nvim -u NONE -i NONE -N --cmd 'call setline(".", [".\xAB"]) | /«/' --cmd cquit

enough to reproduce the bug consistently? I've tried a few times but got no errors(a94a681)

ZyX-I · 2015-08-14T21:13:12Z

@tarruda This command errors out with

==6438== Memcheck, a memory error detector
==6438== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==6438== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==6438== Command: bin/nvim -u NONE -i NONE -N --cmd call\ setline(".",\ [".\\xAB"])\ |\ /«/ --cmd cquit
==6438== Parent PID: 7981
==6438== 

parse DIE(readdwarf3.c:3607): confused by:
 <2><469>: Abbrev Number: 17 (DW_TAG_subrange_type)
     DW_AT_type        : <471>  
     DW_AT_count       : 1024   
parse_type_DIE:
--6438-- WARNING: Serious error when reading debug info
--6438-- When reading debug info from /home/zyx/a.a/Proj/c/neovim/build/bin/nvim:
--6438-- confused by the above DIE
==6438== Invalid read of size 1
==6438==    at 0x5231DC: utf_ptr2char (mbyte.c:1361)
==6438==    by 0x4D93BB: find_match_text (regexp_nfa.c:4784)
==6438==    by 0x4D8F3F: nfa_regexec_both (regexp_nfa.c:6276)
==6438==    by 0x4D8C7B: nfa_regexec_multi (regexp_nfa.c:6493)
==6438==    by 0x4D86D5: vim_regexec_multi (regexp.c:7124)
==6438==    by 0x61D0E6: searchit (search.c:540)
==6438==    by 0x61EC37: do_search (search.c:1126)
==6438==    by 0x4A09DE: get_address (ex_docmd.c:3521)
==6438==    by 0x48CCF4: do_one_cmd (ex_docmd.c:1555)
==6438==    by 0x48A9E5: do_cmdline (ex_docmd.c:621)
==6438==    by 0x48B995: do_cmdline_cmd (ex_docmd.c:293)
==6438==    by 0x47CBD7: exe_pre_commands (main.c:1756)
==6438==    by 0x47AD57: main (main.c:334)
==6438==  Address 0x6e445c3 is 0 bytes after a block of size 3 alloc'd
==6438==    at 0x4C2A030: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==6438==    by 0x54F047: try_malloc (memory.c:62)
==6438==    by 0x54F184: xmalloc (memory.c:96)
==6438==    by 0x54F4BC: xmallocz (memory.c:171)
==6438==    by 0x54F4F8: xmemdupz (memory.c:189)
==6438==    by 0x54FA1B: xstrdup (memory.c:381)
==6438==    by 0x629F84: vim_strsave (strings.c:54)
==6438==    by 0x46DCFE: ml_replace (memline.c:2337)
==6438==    by 0x57BFE7: f_setline (eval.c:14370)
==6438==    by 0x55ED30: call_func (eval.c:7104)
==6438==    by 0x561A90: get_func_tv (eval.c:6953)
==6438==    by 0x560E49: ex_call (eval.c:2797)
==6438==    by 0x48F2B9: do_one_cmd (ex_docmd.c:2195)
==6438==    by 0x48A9E5: do_cmdline (ex_docmd.c:621)
==6438==    by 0x48B995: do_cmdline_cmd (ex_docmd.c:293)
==6438==    by 0x47CBD7: exe_pre_commands (main.c:1756)
==6438==    by 0x47AD57: main (main.c:334)
==6438== 
==6438== Invalid read of size 1
==6438==    at 0x5231EE: utf_ptr2char (mbyte.c:1362)
==6438==    by 0x4D93BB: find_match_text (regexp_nfa.c:4784)
==6438==    by 0x4D8F3F: nfa_regexec_both (regexp_nfa.c:6276)
==6438==    by 0x4D8C7B: nfa_regexec_multi (regexp_nfa.c:6493)
==6438==    by 0x4D86D5: vim_regexec_multi (regexp.c:7124)
==6438==    by 0x61D0E6: searchit (search.c:540)
==6438==    by 0x61EC37: do_search (search.c:1126)
==6438==    by 0x4A09DE: get_address (ex_docmd.c:3521)
==6438==    by 0x48CCF4: do_one_cmd (ex_docmd.c:1555)
==6438==    by 0x48A9E5: do_cmdline (ex_docmd.c:621)
==6438==    by 0x48B995: do_cmdline_cmd (ex_docmd.c:293)
==6438==    by 0x47CBD7: exe_pre_commands (main.c:1756)
==6438==    by 0x47AD57: main (main.c:334)
==6438==  Address 0x6e445c3 is 0 bytes after a block of size 3 alloc'd
==6438==    at 0x4C2A030: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==6438==    by 0x54F047: try_malloc (memory.c:62)
==6438==    by 0x54F184: xmalloc (memory.c:96)
==6438==    by 0x54F4BC: xmallocz (memory.c:171)
==6438==    by 0x54F4F8: xmemdupz (memory.c:189)
==6438==    by 0x54FA1B: xstrdup (memory.c:381)
==6438==    by 0x629F84: vim_strsave (strings.c:54)
==6438==    by 0x46DCFE: ml_replace (memline.c:2337)
==6438==    by 0x57BFE7: f_setline (eval.c:14370)
==6438==    by 0x55ED30: call_func (eval.c:7104)
==6438==    by 0x561A90: get_func_tv (eval.c:6953)
==6438==    by 0x560E49: ex_call (eval.c:2797)
==6438==    by 0x48F2B9: do_one_cmd (ex_docmd.c:2195)
==6438==    by 0x48A9E5: do_cmdline (ex_docmd.c:621)
==6438==    by 0x48B995: do_cmdline_cmd (ex_docmd.c:293)
==6438==    by 0x47CBD7: exe_pre_commands (main.c:1756)
==6438==    by 0x47AD57: main (main.c:334)
==6438== 
==6438== Invalid read of size 1
==6438==    at 0x61D465: searchit (search.c:614)
==6438==    by 0x61EC37: do_search (search.c:1126)
==6438==    by 0x4A09DE: get_address (ex_docmd.c:3521)
==6438==    by 0x48CCF4: do_one_cmd (ex_docmd.c:1555)
==6438==    by 0x48A9E5: do_cmdline (ex_docmd.c:621)
==6438==    by 0x48B995: do_cmdline_cmd (ex_docmd.c:293)
==6438==    by 0x47CBD7: exe_pre_commands (main.c:1756)
==6438==    by 0x47AD57: main (main.c:334)
==6438==  Address 0x6e445c3 is 0 bytes after a block of size 3 alloc'd
==6438==    at 0x4C2A030: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==6438==    by 0x54F047: try_malloc (memory.c:62)
==6438==    by 0x54F184: xmalloc (memory.c:96)
==6438==    by 0x54F4BC: xmallocz (memory.c:171)
==6438==    by 0x54F4F8: xmemdupz (memory.c:189)
==6438==    by 0x54FA1B: xstrdup (memory.c:381)
==6438==    by 0x629F84: vim_strsave (strings.c:54)
==6438==    by 0x46DCFE: ml_replace (memline.c:2337)
==6438==    by 0x57BFE7: f_setline (eval.c:14370)
==6438==    by 0x55ED30: call_func (eval.c:7104)
==6438==    by 0x561A90: get_func_tv (eval.c:6953)
==6438==    by 0x560E49: ex_call (eval.c:2797)
==6438==    by 0x48F2B9: do_one_cmd (ex_docmd.c:2195)
==6438==    by 0x48A9E5: do_cmdline (ex_docmd.c:621)
==6438==    by 0x48B995: do_cmdline_cmd (ex_docmd.c:293)
==6438==    by 0x47CBD7: exe_pre_commands (main.c:1756)
==6438==    by 0x47AD57: main (main.c:334)
==6438== 
==6438== 
==6438== HEAP SUMMARY:
==6438==     in use at exit: 25,555 bytes in 37 blocks
==6438==   total heap usage: 1,087 allocs, 1,050 frees, 278,864 bytes allocated
==6438== 
==6438== LEAK SUMMARY:
==6438==    definitely lost: 0 bytes in 0 blocks
==6438==    indirectly lost: 0 bytes in 0 blocks
==6438==      possibly lost: 0 bytes in 0 blocks
==6438==    still reachable: 25,555 bytes in 37 blocks
==6438==         suppressed: 0 bytes in 0 blocks
==6438== Reachable blocks (those to which a pointer was found) are not shown.
==6438== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==6438== 
==6438== For counts of detected and suppressed errors, rerun with: -v
==6438== ERROR SUMMARY: 5 errors from 3 contexts (suppressed: 0 from 0)

on a94a681. Using “Linux zyx-desktop 4.0.5-gentoo #1 SMP Mon Jul 20 20:43:27 MSK 2015 x86_64 AMD FX(tm)-6200 Six-Core Processor AuthenticAMD GNU/Linux”. Not using jemalloc.

ZyX-I · 2015-08-14T21:13:47Z

@tarruda Can jemalloc change the outcome?

tarruda · 2015-08-14T22:13:38Z

@tarruda Can jemalloc change the outcome?

It appears so, after disabling jemalloc I saw the errors. AFAIK jemalloc allocates memory from pools of different sizes, which explains why the invalid read is not detected by valgrind.

I think it is reasonable to disable JEMALLOC by default on debug builds.

justinmk · 2015-08-22T19:49:35Z

This week there was a vim_dev patch regarding syntax engine instability with --remote-expr.

Bram's response:

Some kind of global lock, or a queue to put postponed commands on would work better. The netbeans feature already does this, we could build on top of it. Specifically netbeans_parse_messages().

Neovim already does this. Correct?

tarruda · 2015-08-22T20:29:27Z

Neovim already does this. Correct?

Yes, and the main reason is the one described by Bram:

If we are calling breakcheck, then we do not expect all kinds of things
to happen.

tarruda · 2015-08-22T20:31:47Z

For the record, the problem described in the issue is not the same as the one described in that vim_dev thread

Close neovim#3150

tarruda · 2015-09-08T16:07:01Z

FWIW this bug seems present on vim 7.4.52(the ubuntu 14.04 version)

Close neovim#3150

ZyX-I added the bug issues reporting wrong behavior label Aug 9, 2015

justinmk added this to the 0.1-first-public-release milestone Aug 9, 2015

ZyX-I mentioned this issue Aug 13, 2015

[RFC] Replace viminfo with ShaDa files #2506

Merged

tarruda mentioned this issue Aug 22, 2015

Random seg faults #2893

Closed

justinmk mentioned this issue Sep 3, 2015

[RFC] path: check if directories are readable when listing files #3273

Closed

tarruda added a commit to tarruda/neovim that referenced this issue Sep 8, 2015

regexp: Fix invalid multibyte reads at end of strings

9d4ae5e

Close neovim#3150

tarruda added a commit to tarruda/neovim that referenced this issue Sep 9, 2015

regexp: Fix invalid multibyte reads at end of strings

c53335e

Close neovim#3150

tarruda added a commit to tarruda/neovim that referenced this issue Sep 11, 2015

regexp: Fix invalid multibyte reads at end of strings

9ce5dbe

Close neovim#3150

tarruda added a commit to tarruda/neovim that referenced this issue Sep 15, 2015

regexp: Fix invalid multibyte reads at end of strings

4deac02

Close neovim#3150

tarruda added a commit to tarruda/neovim that referenced this issue Sep 16, 2015

regexp: Fix invalid multibyte reads at end of strings

6512028

Close neovim#3150

tarruda added a commit to tarruda/neovim that referenced this issue Sep 18, 2015

regexp: Fix invalid multibyte reads at end of strings

95f2f8d

Close neovim#3150

tarruda closed this as completed in 0a116c8 Sep 18, 2015

mgraczyk pushed a commit to mgraczyk/neovim that referenced this issue Oct 7, 2015

regexp: Fix invalid multibyte reads at end of strings

0d4a3c2

Close neovim#3150

oni-link mentioned this issue Nov 11, 2015

split() function is slow for large strings #3486

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Invalid reads in regexp handling code #3150

Invalid reads in regexp handling code #3150

ZyX-I commented Aug 9, 2015

ZyX-I commented Aug 10, 2015

ZyX-I commented Aug 10, 2015

ZyX-I commented Aug 10, 2015

ZyX-I commented Aug 10, 2015

ZyX-I commented Aug 10, 2015

justinmk commented Aug 10, 2015

ZyX-I commented Aug 10, 2015

ZyX-I commented Aug 10, 2015

ZyX-I commented Aug 10, 2015

jebaum commented Aug 14, 2015

tarruda commented Aug 14, 2015

ZyX-I commented Aug 14, 2015

ZyX-I commented Aug 14, 2015

tarruda commented Aug 14, 2015

justinmk commented Aug 22, 2015

tarruda commented Aug 22, 2015

tarruda commented Aug 22, 2015

tarruda commented Sep 8, 2015

Invalid reads in regexp handling code #3150

Invalid reads in regexp handling code #3150

Comments

ZyX-I commented Aug 9, 2015

ZyX-I commented Aug 10, 2015

ZyX-I commented Aug 10, 2015

ZyX-I commented Aug 10, 2015

ZyX-I commented Aug 10, 2015

ZyX-I commented Aug 10, 2015

justinmk commented Aug 10, 2015

ZyX-I commented Aug 10, 2015

ZyX-I commented Aug 10, 2015

ZyX-I commented Aug 10, 2015

jebaum commented Aug 14, 2015

tarruda commented Aug 14, 2015

ZyX-I commented Aug 14, 2015

ZyX-I commented Aug 14, 2015

tarruda commented Aug 14, 2015

justinmk commented Aug 22, 2015

tarruda commented Aug 22, 2015

tarruda commented Aug 22, 2015

tarruda commented Sep 8, 2015