segfault when trying to list a not indexed directory #236
Assignees
Comments
|
What is your duc version?
$ duc --version
Quoting deathtrip (2020-03-25 21:04:15)
… When i'm inside a directory not in the database, and type `duc ls` i get:
> Path /home/deathtrip not found in database
Requested path not found
zsh: segmentation fault (core dumped) duc ls
>
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
#236
--
:wq
^X^Cy^K^X^C^C^C^C
|
|
duc version: 1.4.4 |
|
>>>> "deathtrip" == deathtrip ***@***.***> writes:
deathtrip> duc version: 1.4.4
deathtrip> options: cairo x11 ui leveldb
deathtrip> Running on Arch Linux.
Hmm... I think we might need to add more info to the --version command
to give actual versions of the various libraries if we can...
I don't have any real concrete suggestions, but I do have some
questions:
What release of arch linux and the output of 'uname -a' please.
Can you give us a test case DB that shows the problem?
How about the output of: duc info
Are you using a different $LANG setting maybe? Can you share details
of your linux setup with us?
I just built duc 1.4.4 with leveldb on linux Mint 19.3 Tricia and I
had it work just find inside and outside the indexed path using the ls
command. But I didn't index anything big.
I'd love to figure out what's wrong here and add some test cases to
make sure we don't have these in the future.
I even installed zsh to make sure it wasn't something wierd there
either. No luck. We're going to need more details from you. At
worse, if you can run it under gdb and get a back trace, that might
help. But you'll have to build with debugging info:
make distclean
CFLAGS="-g" ./configure --with-db-backend=leveldb
make -j
Then try to reproduce the crash, and if you can, then do:
bt
when it crashes at the (gdb) prompt, that might help us.
Thanks!
John
|
|
Arch Linux is a rolling distro, so there is no "release", the segfault happens on both the 5.5 and 5.6 kernels. duc info shows: Yes, i use a different $LANG (than en_US i assume). |
|
>>>> "deathtrip" == deathtrip ***@***.***> writes:
deathtrip> Arch Linux is a rolling distro, so there is no "release",
deathtrip> the segfault happens on both the 5.5 and 5.6 kernels.
Ok, good to kow.
deathtrip> duc info shows:
deathtrip> Date Time Files Dirs Size Path
deathtrip> 2020-03-26 16:28:17 2.8K 1.7K 4.3G /home/deathtrip/makepkg/sources
deathtrip> Yes, i use a different $LANG (than en_US i assume).
Can you tell us whay $LANG you are using? It's probably something to
do with that causing the segfault.
And would you mind sharing your DB with us privately so we can look it
over as a test case?
Were you able to run duc under GDB and get a backtrace?
Please give us more details, even those you think might not matter.
For example, if you do the following commands:
$ cd /var/tmp/duc
$ ./configure --with-backend-db=leveldb
$ make -j
$ ./duc index -d /var/tmp/test.db .
$ cd /tmp
$ /var/tmp/duc/duc ls -d /var/tmp/test.db
does it segfault then? If it does, can you please send us a copy of
the db and your LANG* environment variables, or anything else you can
think of about your environment so we can re-produce this issue?
Also, just for testing, does this seg fault happen under bash instead
of zsh? just trying to get the smallest test case I can for this.
Cheers,
John
|
|
I also have a similar issue under ubuntu 16.4 when I use * in my query duc built from source |
|
>>>> "Stephane" == Stephane Plaisance ***@***.***> writes:
Stephane> I also have a similar issue under ubuntu 16.4 when I use *
Stephane> in my query
Stephane> duc ls -D -d /opt/biotools/duc/db/freenas.db /mnt/freenas/NC_projects/*
This isn't the right way to do this command, since the '*' is expanded
by the shell before it's fed to duc. If you do:
ls /mnt/freenas/NC_projects/*
what do you get back?
Stephane> ...
Stephane> expected output then at the end
Stephane> ...
Stephane> Requested path not found
Stephane> Segmentation fault (core dumped)
This is a bug which we should handle better. If you can reduce this
down to a smaller test case, and share the DB with us, that would be
awesome. Or even build duc with debugging and run it under 'gdb' to
get a back trace would help as well.
Stephane> duc built from source
Stephane> duc --version
Stephane> duc version: 1.4.4
Stephane> options: cairo x11 ui tokyocabinet
Hopefully this is a known bug. I'm busy with work/life right now, but
will try to poke at this more this week.
|
|
ls /mnt/freenas/NC_projects/* returns a whole bunch of folders and content. |
|
Stephane> ls /mnt/freenas/NC_projects/* returns a whole bunch of folders and content.
I'm going to need to see that list to try and figure out what's going
wrong here.
Just a thought though, when you do your 'ls' do you get directories
first?
Stephane> Just gut feeling but since I get some results, could it be
Stephane> that the tool fails when it tries to report '.' and or '..'
Stephane> ?
Nope, that's not it, at least in my testing here.
Stephane> I have the impression that the failure comes after a
Stephane> substantial number of good results like if DUC would reach
Stephane> the end of its work-list and chew on something tasting bad.
Stephane> again, just a gut feeling :-)
Again, we need to see the data. If you can copy that top level of
NC_Projects/ to /tmp/NC_Projects and then index that to a new
directory and see if the problem is still there, that would be a help.
I haven't been able to re-create the error with a directory with
spaces, but you also haven't given us a good enough report to know
what's happening.
If you're worried about sharing private directory names, you can send
them to me directly if you like: john@stoffel.org
|
|
I haven't been able to re-produce this crash, if you can still
reproduce it, can you please run duc undex gdb and get a back trace?
Basically, do:
gdb ./duc .....
and when you get a crash, do the 'bt' command at the gdb> prompt to
get a back trace. Any other info you can provide would be helpful.
If you can, maybe copy just the files and directories from the level
having the problem into a new directory somewhere completely else,
then try to re-create the problem. If it does, then try to delete
files/dirs one by one until it works again.
I'll keep poking at things here, but something strange is going on.
Also, are you mounting this from another system? If so, please give
us the full details of the mount command you're using. Ideally do:
mount | grep mnt
So we can see what's going on here.
Thanks,
John
|
|
I get a segmentation fault on Ubuntu 18.04 with as well as with when trying to maybe the output of that is how it's mounted: |
|
>>>> "Alex" == Alex Myczko ***@***.***> writes:
Hi Alex,
Can you try the latest verision from github? It's at version 1.4.4 so
this crash might be fixed already. We're slowly slowly slowly working
on getting a new release out, but life keeps getting in the way.
If you could replicate this crash on the latest version, that would be
a big help.
Cheers,
John
Alex> I get a segmentation fault on Ubuntu 18.04 with duc --version
Alex> duc version: 1.4.3 options: ui tokyocabinet
Alex> when trying to duc index /scratch
Alex> maybe the output of strace duc index -x -p /scratch is helpful, the last few lines
Alex> openat(AT_FDCWD, "tests", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 13
Alex> fstat(13, {st_mode=S_IFDIR|0755, st_size=104, ...}) = 0
Alex> chdir("tests") = 0
Alex> getdents(13, /* 6 entries */, 32768) = 192
Alex> lstat("__init__.py", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
Alex> lstat("__init__.pyc", {st_mode=S_IFREG|0644, st_size=155, ...}) = 0
Alex> lstat("test_denton.py", {st_mode=S_IFREG|0644, st_size=1256, ...}) = 0
Alex> lstat("test_denton.pyc", {st_mode=S_IFREG|0644, st_size=1760, ...}) = 0
Alex> getdents(13, /* 0 entries */, 32768) = 0
Alex> chdir("..") = 0
Alex> sched_yield() = 0
Alex> sched_yield() = 0
Alex> sched_yield() = 0
Alex> close(13) = 0
Alex> getdents(12, /* 0 entries */, 32768) = 0
Alex> chdir("..") = 0
Alex> sched_yield() = 0
Alex> --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=NULL} ---
Alex> +++ killed by SIGSEGV +++
Alex> Segmentation fault
Alex> —
Alex> You are receiving this because you were assigned.
Alex> Reply to this email directly, view it on GitHub, or unsubscribe.
Alex> Triage notifications on the go with GitHub Mobile for iOS or Android.*
|
|
@l8gravely I had tried with 1.4.4, you maybe mean a master/main checkout (current)? Funnily, now I get a different output: as usual i get the upstream source, and oh well it ships debian/, is that needed, or can it be dropped? ok now it looks different: ok so i remove /root/.duc.db and run it again, and it looks good. time for a new release, pity it won't make it into and what should $XDG_CACHE_HOME if it's not defined? what happens on gnu/kfreebsd or macOS? |
|
>>>> "Alex" == Alex Myczko ***@***.***> writes:
Alex> @l8gravely I had tried with 1.4.4, you maybe mean a master/main checkout (current)?
Alex> Funnily, now I get a different output:
Alex> # duc index /scratch
Alex> free(): invalid next size (normal)
Alex> Aborted
Interesting. Can you give us more details about what filesystem
you're trying to index? And what backend and kernel version you're
using? The more details the better. 'duc --version' is a good start.
And yes, if you can try master/main from github as a test, that would
be great.
You might need to do:
make distclean
CFLAGS="-Og" ./configure
make
and then try to get a back trace when/if it crashes again.
John
|
|
it was 4.15.0 kernel of ubuntu 18.04, on btrfs with live file compression, see my previous two messages. fun fact: it doesn't crash anymore since indexed with 1.4.5-rc1 (yesterdays master), now 1.4.3 is back installed... |
|
>>>> "Alex" == Alex Myczko ***@***.***> writes:
Alex> it was 4.15.0 kernel of ubuntu 18.04, on btrfs with live file
Alex> compression, see my previous two messages. fun fact: it doesn't
Alex> crash anymore since indexed with 1.4.5-rc1 (yesterdays master),
Alex> now 1.4.3 is back installed...
Fun fun fun... If you could find some way to replicate this issue,
that would be awesome. I don't run BTRFS at all, so it's hard for me
to spin up a test system without a bunch of work.
I'll see what I can do.
John
|
|
Well, I have like 200 workstations (2 with 18.04, the rest with 20.04 of Ubuntu) with /scratch mostly being btrfs, and some are xfs, I'll need to rerun it on all to find the 1% that fail. I guess it's not related to filesystem generally, but more likely the files/directories where some just make it fail, for reasons |
|
no idea if this could play here but I have had bad times long ago with windows users naming files or folders with prohibited characters like / \ | ( ) : that got somewhere interpreted and made my life miserable |
|
>>>> "Alex" == Alex Myczko ***@***.***> writes:
Alex> Well, I have like 200 workstations with /scratch mostly being
Alex> btrfs, and some are xfs, I'll need to rerun it on all to find
Alex> the 1% that fail. I guess it's not related to filesystem
Alex> generally, but more likely the files/directories where some just
Alex> make it fail, for reasons I can't imagine yet. Will keep posting
Alex> here if I find out more...
Ah.... users! Can't live with em, can't LART em. Bless their hearts
for doing all the strange things to make stuff break. *grin*
Thanks for helping chase this down, it's no fun, but I suspect they
have a symlink loop or something like that in the filesystem. We've
seen some problems like that in the past, but I don't recall the
details right now.
John
|
|
Thanks for the hint, it could of course very well be symlink loop(s). |
|
>>>> "Alex" == Alex Myczko ***@***.***> writes:
Alex> Thanks for the hint, it could of course very well be symlink loop(s).
I think there's a test for that now, but I've been very slow in my
checks of the code recently. Sorry.
Would love for you to find a test case which causes duc to crash
reliably so we can fix it and put in a test case.
John
|
|
Hi, I just run into the same segfault issue. $ uname -a Full compilation-logs.txt. |
|
Sorry for the late reply, and thanks for the bug report. Can you pull
down the latest release and check it? I'll try to get at it this
weekend, but not promises.
John
|
|
Hi, I compiled it from source (git clone, commit d2d1dfb), and I cannot reproduce the error. It looks like ok. I also had similar issue on larger database, which I cannot reproduce too. |
|
>>>> "iksnalybok" == iksnalybok ***@***.***> writes:
iksnalybok> Hi, I compiled it from source (git clone, commit d2d1dfb), and I cannot reproduce the error.
iksnalybok> $ ./duc index --database=my.db .
iksnalybok> $ ./duc ls --database=my.db doesNotExist
iksnalybok> The requested path 'doesNotExist' was not found in the database,
iksnalybok> Please run 'duc info' for a list of available directories.
iksnalybok> $ ./duc ls --database=my.db /
iksnalybok> Path / not found in database
iksnalybok> The requested path '/' was not found in the database,
iksnalybok> Please run 'duc info' for a list of available directories.
iksnalybok> It looks like ok. I also had similar issue on larger
iksnalybok> database, which I cannot reproduce too.
Thanks for the report and update. I need to cut a new release sooner
rather than later. Hopefully this coming weekend.
John
|
|
Just build with the latest version (1.4.5-rc1) and have not had this again thankfully! Note I did remove my .duc.db first too, just in case with the version change. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When i'm inside a directory not in the database, and type
duc lsi get:The text was updated successfully, but these errors were encountered: