Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

macOS Catalina 10.15 search tag appear "E432: Tags file not sorted" #11196

Closed
johnlamb opened this issue Oct 10, 2019 · 17 comments · Fixed by #11222

Comments

@johnlamb
Copy link

@johnlamb johnlamb commented Oct 10, 2019

Screenshot 2019-10-10 at 20 27 19

  • nvim --version:
    NVIM v0.4.2
    Build type: Release
    LuaJIT 2.0.5
  • vim -u DEFAULTS (version: ) behaves differently?
    No
  • Operating system/version:
    macOS Cataline 10.15
  • Terminal name/version:
    iTerm 3.3.6 and macOS Terminal
  • $TERM:
    xterm-256color

Steps to reproduce using nvim -u NORC

nvim -u NORC
Use the built in help :h
EIther with or without a topic

Actual behaviour

Gives:
E432: Tags file not sorted: /usr/local/Cellar/neovim/0.4.2/share/nvim/runtime/doc/tags
E432: Tags file not sorted: /usr/local/Cellar/neovim/0.4.2/share/nvim/runtime/pack/dist/opt/matchit/doc/tags
when a topic is supplied and
E432: Tags file not sorted: /usr/local/Cellar/neovim/0.4.2/share/nvim/runtime/doc/tags
when help is started without a topic.

Expected behaviour

No errors.

Notes

This is a marked issue in Vim (vim/vim#5039) as well.
Same issue using macOS builtin ctags, homebrews ctags and homebrews universal-ctags.
Issue persist when --HEAD is installed.

@johnlamb johnlamb added the bug label Oct 10, 2019
@mhinz

This comment has been minimized.

Copy link
Member

@mhinz mhinz commented Oct 10, 2019

FWIW, I can't reproduce it at all, no matter which ctags implementation is used. So, there's probably other conditions needed as well.

I'm really just guessing here, but did you do the typical xcode-select --install after upgrading? Xcode often effects a lot of parts in magical ways and it's the only other thing I did after upgrading.

@dm1try

This comment has been minimized.

Copy link
Contributor

@dm1try dm1try commented Oct 11, 2019

reproducible example -> echo "a\ta\ta\nb\tb\tb\nc\tc\tc\nd\td\td\n" > tags && nvim -c 'tag a' on my side

@mhinz

This comment has been minimized.

Copy link
Member

@mhinz mhinz commented Oct 11, 2019

Hmm, that works for me, too. I just get the expected E429: File "a" does not exist.

@dm1try

This comment has been minimized.

Copy link
Contributor

@dm1try dm1try commented Oct 11, 2019

@mhinz that's interesting)
Screen Shot 2019-10-11 at 03 50 48

@mhinz

This comment has been minimized.

Copy link
Member

@mhinz mhinz commented Oct 11, 2019

Correction: echo "a\ta\ta\nb\tb\tb\nc\tc\tc\nd\td\td\n" > tags && nvim -u NORC -c 'tag a' does throw E432: Tags file not sorted: tags as well.

(But I still can't reproduce it with just nvim -u NORC after a fresh brew install neovim.)

@mhinz

This comment has been minimized.

Copy link
Member

@mhinz mhinz commented Oct 11, 2019

Ah.. of course.. can't reproduce it with:

nvim -u NORC -c 'set ignorecase | tag a'
@dm1try

This comment has been minimized.

Copy link
Contributor

@dm1try dm1try commented Oct 11, 2019

offset calculation in binary search algorithm is broken somehow
Screen Shot 2019-10-11 at 04 50 11

though, I'm not sure; will continue tomorrow

@KevinSjoberg

This comment has been minimized.

Copy link

@KevinSjoberg KevinSjoberg commented Oct 11, 2019

I'm seeing the exact same problem with a setup that worked flawlessly for over ten years. I'm trying to lookup documentation using :help <keyword> and are instantly greeted with the error message.

Command

:help surround

Output

E432: Tags file not sorted: /Users/kevinsjoberg/.local/share/nvim/plugged/lightline.vim/doc/tags
E432: Tags file not sorted: /Users/kevinsjoberg/.local/share/nvim/plugged/vim-dirvish/doc/tags
E432: Tags file not sorted: /Users/kevinsjoberg/.local/share/nvim/plugged/vim-abolish/doc/tags
E432: Tags file not sorted: /Users/kevinsjoberg/.local/share/nvim/plugged/vim-commentary/doc/tags
E432: Tags file not sorted: /Users/kevinsjoberg/.local/share/nvim/plugged/vim-dispatch/doc/tags
E432: Tags file not sorted: /Users/kevinsjoberg/.local/share/nvim/plugged/vim-rails/doc/tags
E432: Tags file not sorted: /usr/local/Cellar/neovim/0.4.2/share/nvim/runtime/doc/tags
E426: tag not found: surround@en

Neovim

NVIM v0.4.2
Build type: Release
LuaJIT 2.0.5
Compilation: /usr/local/Homebrew/Library/Homebrew/shims/mac/super/clang -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -DNDEBUG -DMIN_LOG_LEVEL=3 -Wall -Wextra -pedanti
c -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wshadow -Wconversion -Wmissing-prototypes -Wimplicit-fallthrough -Wvla -fstack-protector-strong -fdiagn
ostics-color=auto -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -DNVIM_MSGPACK_HAS_FLOAT32 -DNVIM_UNIBI_HAS_VAR_FROM -I/tmp/neovim-20190916-88369-176iaxs/neov
im-0.4.2/build/config -I/tmp/neovim-20190916-88369-176iaxs/neovim-0.4.2/src -I/usr/local/include -I/tmp/neovim-20190916-88369-176iaxs/neovim-0.4.2/deps-build/i
nclude -I/usr/local/opt/gettext/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -I/tm
p/neovim-20190916-88369-176iaxs/neovim-0.4.2/build/src/nvim/auto -I/tmp/neovim-20190916-88369-176iaxs/neovim-0.4.2/build/include
Compiled by brew@Mojave-2.local

This began happening after updating to Mac OS Catalina. Never had this problem before. I've tried generating helptags automatically using vim-plug, manually using :helptags ALL, and the result is the same. I've also manually deleted all tags and then regenerated them. The outcome is the same.

@dm1try

This comment has been minimized.

Copy link
Contributor

@dm1try dm1try commented Oct 11, 2019

the problem is here, on Catalina the returned value of vim_ftell is -1
Screen Shot 2019-10-11 at 17 22 45

Screen Shot 2019-10-11 at 17 57 37

@dm1try

This comment has been minimized.

Copy link
Contributor

@dm1try dm1try commented Oct 11, 2019

ok guys,
something is going wrong in underlying FIO
this dirty patch works(re-open the tag file for reading), but I have no time; will continue tomorrow

diff --git a/src/nvim/tag.c b/src/nvim/tag.c
index 6fe3efbaa..d64abd76c 100644
--- a/src/nvim/tag.c
+++ b/src/nvim/tag.c
@@ -1372,7 +1372,11 @@ find_tags (
         if (state == TS_BINARY || state == TS_SKIP_BACK) {
           /* Adjust the search file offset to the correct position */
           search_info.curr_offset_used = search_info.curr_offset;
-          vim_fseek(fp, search_info.curr_offset, SEEK_SET);
+          int res = vim_fseek(fp, search_info.curr_offset, SEEK_SET);
+          if(res == -1){
+            fp = freopen((char *)tag_fname, "r", fp);
+            vim_fseek(fp, search_info.curr_offset, SEEK_SET);
+          }
           eof = vim_fgets(lbuf, LSIZE, fp);
           if (!eof && search_info.curr_offset != 0) {
             /* The explicit cast is to work around a bug in gcc 3.4.2

EDIT: updated the patch

@mhinz

This comment has been minimized.

Copy link
Member

@mhinz mhinz commented Oct 11, 2019

What I don't get is why vim_fseek(fp, 12, SEEK_SET) (and thus fseeko()) returns -1 at all. Errno gets set to invalid argument.

But as far as I see, all the arguments are valid. I checked the stream with ferror()/feof() and the tags file is certainly bigger than 12 bytes.

@dm1try

This comment has been minimized.

Copy link
Contributor

@dm1try dm1try commented Oct 11, 2019

EDITED:

@mhinz maybe fp is not "valid" anymore? I'm not sure how to properly check this, but it looks like re-opening the file helps.

I checked the stream with ferror()/feof()

looks like a bug in the file system interface of new osx

@dm1try

This comment has been minimized.

Copy link
Contributor

@dm1try dm1try commented Oct 11, 2019

I've made the minimal reproducible example outside Neovim codebase; see fgets/lseek

#include <stdio.h>
#include <unistd.h>

int main(){
  char lbuf[256];

  FILE *p = fopen("/Users/dmitrydedov/projects/sandbox/tags", "r");
  // those calls lead to "broken"(?) file pointer
  fgets(lbuf,255, p);
  lseek(fileno(p), 0L, SEEK_SET);

  fseek(p, 10, SEEK_SET);

  int pos = ftell(p);
  printf("pos: %d\n", pos);
  return 0;
}

Screen Shot 2019-10-11 at 23 17 24

eof = vim_fgets(lbuf, LSIZE, fp);

vim_lseek(fileno(fp), (off_T)0L, SEEK_SET);

@dm1try

This comment has been minimized.

Copy link
Contributor

@dm1try dm1try commented Oct 11, 2019

using fseek instead of lseak fixes the problem, btw
the new patch:

diff --git a/src/nvim/tag.c b/src/nvim/tag.c
index 6fe3efbaa..fd0a8e057 100644
--- a/src/nvim/tag.c
+++ b/src/nvim/tag.c
@@ -1512,7 +1512,7 @@ line_read_in:
             if ((filesize = vim_lseek(fileno(fp), (off_T)0L, SEEK_END)) <= 0) {
               state = TS_LINEAR;
             } else {
-              vim_lseek(fileno(fp), (off_T)0L, SEEK_SET);
+              vim_fseek(fp, 0, SEEK_SET);

               /* Calculate the first read offset in the file.  Start
                * the search in the middle of the file. */
@dm1try

This comment has been minimized.

Copy link
Contributor

@dm1try dm1try commented Oct 12, 2019

I sent the bug report to Apple, though it's not clear when the bug will be fixed(it exits in 10.15.1 Beta too).

btw fseek works because it calls __sflush while doing its work, while lseek looks like more low-level thing.

so either fseek or flush+lseek will help. should I open a PR with that workaround for Catalina or we will wait for the action from Apple?

@mhinz

This comment has been minimized.

Copy link
Member

@mhinz mhinz commented Oct 13, 2019

Catalina clearly has lots of issues around I/O. E.g. Homebrew/homebrew-core#44715 or fink/fink#197.

Why was lseek() instead of fseek() used in the first place? (aka. Why mixing standard I/O with file I/O?) I don't see much point in that opposed to other uses of lseek() in fileio.c or memline.c, which work on file descriptors anyway.

No matter how long Apple needs to fix their stuff, if there are no pressing arguments against replacing both vim_lseek() in tag.c by vim_fseek(), I'd go that route.

@mhinz

This comment has been minimized.

Copy link
Member

@mhinz mhinz commented Oct 13, 2019

@dm1try

Just make a PR for it. If Vim later decides to wait for Apple or goes another route, we can change it again to stay in sync.

It's an annoying bug that probably a lot of users experience at the moment, so let's just get that fix in now. Maybe we can even get it into the next 0.4.3 release, so it gets picked up by Homebrew as well.

dm1try added a commit to dm1try/neovim that referenced this issue Oct 13, 2019
this also fixes neovim#11196 as using lseek leads to file system misbehaviour on osx Catalina
dm1try added a commit to dm1try/neovim that referenced this issue Oct 13, 2019
this also fixes neovim#11196 as using lseek leads to file system misbehaviour on osx Catalina
Gasol added a commit to Gasol/neovim that referenced this issue Oct 14, 2019
…1196 as using lseek leads to file system misbehaviour on osx Catalina
@mhinz mhinz closed this in #11222 Oct 14, 2019
mhinz added a commit that referenced this issue Oct 14, 2019
I/O in Catalina is currently known to be broken. This commit works around a pesky bug and also makes the code more consistent by removing the mix of C file and standard I/O.

Fixes #11196
blueyed added a commit that referenced this issue Oct 14, 2019
I/O in Catalina is currently known to be broken. This commit works
around a pesky bug and also makes the code more consistent by removing
the mix of C file and standard I/O.

Fixes #11196

(cherry picked from commit d0efc1c)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.