Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PendingFileCompleter failures lead to dirty index #2148

Closed
vladak opened this issue Jun 8, 2018 · 2 comments
Closed

PendingFileCompleter failures lead to dirty index #2148

vladak opened this issue Jun 8, 2018 · 2 comments
Labels
bug stopper

Comments

@vladak
Copy link
Member

vladak commented Jun 8, 2018

When indexing Linux repositories, the indexer ends with:

2018-06-08 14:45:12.947+0200 INFO t1 Indexer.main: Indexer version 1.1-rc28 (dbcad1993409e04b41712fdcf0b2d90534e45c9a)

... (thousands of 'Add' lines omitted)

2018-06-08 14:48:14.192+0200 INFO t3415 DefaultIndexChangedListener.fileAdd: Add: /linux-mainline/drivers/gpu/drm/armada/armada_fb.h (CAnalyzer)
2018-06-08 14:48:14.205+0200 INFO t3415 DefaultIndexChangedListener.fileAdd: Add: /linux-mainline/drivers/gpu/drm/armada/armada_fbdev.c (CAnalyzer)
2018-06-08 14:48:14.239+0200 INFO t3415 DefaultIndexChangedListener.fileAdd: Add: /linux-mainline/drivers/gpu/drm/armada/armada_gem.c (CAnalyzer)
2018-06-08 14:48:34.523+0200 WARNING t3515 PendingFileCompleter.doLink: Failed to link: /opengrok/data/xref/linux-mainline/scripts/dtc/include-prefixes/powerpc -> ../../../arch/powerpc/boot/dts
2018-06-08 14:48:34.523+0200 WARNING t3528 PendingFileCompleter.doLink: Failed to link: /opengrok/data/xref/linux-mainline/scripts/dtc/include-prefixes/h8300 -> ../../../arch/h8300/boot/dts
2018-06-08 14:48:34.523+0200 WARNING t3517 PendingFileCompleter.doLink: Failed to link: /opengrok/data/xref/linux-mainline/scripts/dtc/include-prefixes/arm64 -> ../../../arch/arm64/boot/dts
2018-06-08 14:48:34.523+0200 WARNING t3519 PendingFileCompleter.doLink: Failed to link: /opengrok/data/xref/linux-mainline/scripts/dtc/include-prefixes/dt-bindings -> ../../../include/dt-bindings
2018-06-08 14:48:34.523+0200 WARNING t3537 PendingFileCompleter.doLink: Failed to link: /opengrok/data/xref/linux-mainline/scripts/dtc/include-prefixes/nios2 -> ../../../arch/nios2/boot/dts
2018-06-08 14:48:34.523+0200 WARNING t3534 PendingFileCompleter.doLink: Failed to link: /opengrok/data/xref/linux-mainline/scripts/dtc/include-prefixes/sh -> ../../../arch/sh/boot/dts
2018-06-08 14:48:34.523+0200 WARNING t3525 PendingFileCompleter.doLink: Failed to link: /opengrok/data/xref/linux-mainline/scripts/dtc/include-prefixes/c6x -> ../../../arch/c6x/boot/dts
2018-06-08 14:48:34.523+0200 WARNING t3532 PendingFileCompleter.doLink: Failed to link: /opengrok/data/xref/linux-mainline/scripts/dtc/include-prefixes/mips -> ../../../arch/mips/boot/dts
2018-06-08 14:48:34.523+0200 WARNING t3527 PendingFileCompleter.doLink: Failed to link: /opengrok/data/xref/linux-mainline/scripts/dtc/include-prefixes/arc -> ../../../arch/arc/boot/dts
2018-06-08 14:48:34.523+0200 WARNING t2283 PendingFileCompleter.doLink: Failed to link: /opengrok/data/xref/linux-mainline/scripts/dtc/include-prefixes/xtensa -> ../../../arch/xtensa/boot/dts
2018-06-08 14:48:34.523+0200 WARNING t3536 PendingFileCompleter.doLink: Failed to link: /opengrok/data/xref/linux-mainline/scripts/dtc/include-prefixes/arm -> ../../../arch/arm/boot/dts
2018-06-08 14:48:34.523+0200 WARNING t3543 PendingFileCompleter.doLink: Failed to link: /opengrok/data/xref/linux-mainline/scripts/dtc/include-prefixes/microblaze -> ../../../arch/microblaze/boot/dts
2018-06-08 14:48:34.523+0200 WARNING t3513 PendingFileCompleter.doLink: Failed to link: /opengrok/data/xref/linux-mainline/scripts/dtc/include-prefixes/openrisc -> ../../../arch/openrisc/boot/dts
2018-06-08 14:48:34.557+0200 WARNING t2283 IndexDatabase.finishWriting: An error occurred while finishing writer and completer
java.io.IOException: 13 failures (100.0%) while linking pending paths
	at org.opensolaris.opengrok.index.PendingFileCompleter.completeLinkages(PendingFileCompleter.java:306)
	at org.opensolaris.opengrok.index.PendingFileCompleter.complete(PendingFileCompleter.java:169)
	at org.opensolaris.opengrok.index.IndexDatabase.finishWriting(IndexDatabase.java:1613)
	at org.opensolaris.opengrok.index.IndexDatabase.update(IndexDatabase.java:516)
	at org.opensolaris.opengrok.index.Indexer$1.run(Indexer.java:1080)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.NoSuchFileException: /opengrok/data/xref/linux-mainline/scripts/dtc/include-prefixes/h8300
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixFileSystemProvider.createSymbolicLink(UnixFileSystemProvider.java:457)
	at java.nio.file.Files.createSymbolicLink(Files.java:1043)
	at org.opensolaris.opengrok.index.PendingFileCompleter.doLink(PendingFileCompleter.java:353)
	at org.opensolaris.opengrok.index.PendingFileCompleter.lambda$completeLinkages$6(PendingFileCompleter.java:289)
	at java.util.stream.Collectors.lambda$groupingByConcurrent$51(Collectors.java:1070)
	at java.util.stream.ReferencePipeline.lambda$collect$1(ReferencePipeline.java:496)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
	at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

2018-06-08 14:48:34.559+0200 SEVERE t2283 Indexer$1.run: An error occurred while updating index
java.io.IOException: 13 failures (100.0%) while linking pending paths
	at org.opensolaris.opengrok.index.PendingFileCompleter.completeLinkages(PendingFileCompleter.java:306)
	at org.opensolaris.opengrok.index.PendingFileCompleter.complete(PendingFileCompleter.java:169)
	at org.opensolaris.opengrok.index.IndexDatabase.finishWriting(IndexDatabase.java:1613)
	at org.opensolaris.opengrok.index.IndexDatabase.update(IndexDatabase.java:516)
	at org.opensolaris.opengrok.index.Indexer$1.run(Indexer.java:1080)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.NoSuchFileException: /opengrok/data/xref/linux-mainline/scripts/dtc/include-prefixes/h8300
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixFileSystemProvider.createSymbolicLink(UnixFileSystemProvider.java:457)
	at java.nio.file.Files.createSymbolicLink(Files.java:1043)
	at org.opensolaris.opengrok.index.PendingFileCompleter.doLink(PendingFileCompleter.java:353)
	at org.opensolaris.opengrok.index.PendingFileCompleter.lambda$completeLinkages$6(PendingFileCompleter.java:289)
	at java.util.stream.Collectors.lambda$groupingByConcurrent$51(Collectors.java:1070)
	at java.util.stream.ReferencePipeline.lambda$collect$1(ReferencePipeline.java:496)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
	at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

2018-06-08 14:48:34.567+0200 INFO t1 Statistics.report: Done indexing data of all repositories (took 0:03:11)
2018-06-08 14:48:34.568+0200 INFO t1 Indexer.refreshSearcherManagers: Refreshing searcher managers to: localhost
2018-06-08 14:48:34.591+0200 INFO t1 Statistics.report: Total time: 0:03:41
2018-06-08 14:48:34.795+0200 INFO t1 Statistics.report: Final Memory: 164M/6,422M

which results in:

$ ls -al /opengrok/data/index/linux-mainline
total 39
drwxr-xr-x   2 webservd webservd       4 Jun  8 14:48 .
drwxr-xr-x 406 webservd webservd     407 Jun  8 14:42 ..
-rw-r--r--   1 webservd webservd       0 May 31 05:25 dirty
-rw-r--r--   1 webservd webservd      69 May 31 02:36 segments_1

Is this perhaps the result of the changes in cset e253612 ?
@vladak
Copy link
Member Author

vladak commented Jun 8, 2018 •
edited
Loading

For reference, this seemed to work in 1.1-rc26:

$ ls -al /opengrok-old/data/index/linux-mainline
total 2026649
drwxr-xr-x   2 webservd webservd      15 Jun  4 00:09 .
drwxr-xr-x 403 webservd webservd     417 May 28 10:22 ..
-rw-r--r--   1 webservd webservd 372373503 Jun  4 00:09 _1nwg.fdt
-rw-r--r--   1 webservd webservd   91183 Jun  4 00:09 _1nwg.fdx
-rw-r--r--   1 webservd webservd    2390 Jun  4 00:09 _1nwg.fnm
-rw-r--r--   1 webservd webservd  385601 Jun  4 00:09 _1nwg.nvd
-rw-r--r--   1 webservd webservd     168 Jun  4 00:09 _1nwg.nvm
-rw-r--r--   1 webservd webservd     559 Jun  4 00:09 _1nwg.si
-rw-r--r--   1 webservd webservd 113504824 Jun  4 00:09 _1nwg_Lucene50_0.doc
-rw-r--r--   1 webservd webservd 453475679 Jun  4 00:09 _1nwg_Lucene50_0.pos
-rw-r--r--   1 webservd webservd 213647183 Jun  4 00:09 _1nwg_Lucene50_0.tim
-rw-r--r--   1 webservd webservd 3578559 Jun  4 00:09 _1nwg_Lucene50_0.tip
-rw-r--r--   1 webservd webservd 1382007 Jun  4 00:09 _1nwg_Lucene54_0.dvd
-rw-r--r--   1 webservd webservd     232 Jun  4 00:09 _1nwg_Lucene54_0.dvm
-rw-r--r--   1 webservd webservd     142 Jun  4 00:09 segments_2lje

@vladak
Copy link
Member Author

vladak commented Jun 8, 2018

For now I workarounded this by removing said symlinks in post-mirror script.

idodeclare added a commit to idodeclare/OpenGrok that referenced this issue Jun 9, 2018
@idodeclare
Fix oracle#2148 : mkdirs() in doLink() in case no other related content
f0ed1fa 
When a leaf directory consists solely of symlinks
to other local directories -- i.e. with no other
content or child content to xref, then the xref/
directory would not have been created to contain
the duplicated symlinks.
@vladak vladak mentioned this issue Jun 22, 2018
Closed
@xiangp126 xiangp126 mentioned this issue Jun 22, 2018
Closed
@vladak vladak mentioned this issue Jun 25, 2018
Closed
@vladak vladak closed this as completed in dd8120b Jul 9, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug stopper
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@vladak