NullPointerException on File indexing (IgnoreFilesIndex), infinite loop of failing re-indexes.

[feuerwagen]

Prerequisites

• Plugin is in the latest version
• Issue was not reported yet
• Stack trace (if provided) contains mobi.hsz.idea.gitignore package name

Description

Infinite loops of index runs (every ~10 sec), each accompanied with the following log entries:

2018-04-04 10:02:51,358 [ 165995] INFO - il.indexing.FileBasedIndexImpl - Rebuild requested for index IgnoreFilesIndex
java.lang.NullPointerException
at com.intellij.util.indexing.impl.ValueContainerImpl.saveTo(ValueContainerImpl.java:410)
at com.intellij.util.indexing.impl.ChangeTrackingValueContainer.saveTo(ChangeTrackingValueContainer.java:174)
at com.intellij.util.indexing.impl.ValueContainerMap$1.append(ValueContainerMap.java:58)
at com.intellij.util.io.PersistentHashMap.doAppendData(PersistentHashMap.java:481)
at com.intellij.util.io.PersistentHashMap.appendData(PersistentHashMap.java:458)
at com.intellij.util.indexing.impl.ValueContainerMap.doPut(ValueContainerMap.java:55)
at com.intellij.util.indexing.impl.ValueContainerMap.doPut(ValueContainerMap.java:28)
at com.intellij.util.io.PersistentHashMap.put(PersistentHashMap.java:377)
at com.intellij.util.indexing.impl.MapIndexStorage$3.onDropFromCache(MapIndexStorage.java:132)
at com.intellij.util.indexing.impl.MapIndexStorage$3.onDropFromCache(MapIndexStorage.java:99)
at com.intellij.util.containers.SLRUMap.clear(SLRUMap.java:148)
at com.intellij.util.indexing.impl.MapIndexStorage.flush(MapIndexStorage.java:156)
at com.intellij.util.indexing.VfsAwareMapIndexStorage.flush(VfsAwareMapIndexStorage.java:106)
at com.intellij.util.indexing.MemoryIndexStorage.flush(MemoryIndexStorage.java:127)
at com.intellij.util.indexing.impl.MapReduceIndex.doFlush(MapReduceIndex.java:148)
at com.intellij.util.indexing.VfsAwareMapReduceIndex.doFlush(VfsAwareMapReduceIndex.java:192)
at com.intellij.util.indexing.impl.MapReduceIndex.flush(MapReduceIndex.java:127)
at com.intellij.util.indexing.FileBasedIndexImpl.a(FileBasedIndexImpl.java:600)
at com.intellij.util.indexing.FileBasedIndexImpl.access$4600(FileBasedIndexImpl.java:117)
at com.intellij.util.indexing.FileBasedIndexImpl$FileIndexDataInitialization$1.run(FileBasedIndexImpl.java:2443)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at com.intellij.util.concurrency.SchedulingWrapper$MyScheduledFutureTask.run(SchedulingWrapper.java:230)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Steps to Reproduce

1. Update .ignore Plugin to latest version (2.5)
2. Open any Project in PHPStorm

Expected behavior: Index runs once, then finishes, as nothing changed

Actual behavior: Infinite index reruns (every ~10 sec)

Reproduces how often: 100%

Versions

Plugin: 2.5

IDE: PHPStorm 2018.1 / 2017.3.6

OS: macOS 10.13.3 / 10.13.4

Additional Information

Removing .ignore Plugin and manual reinstall of version 2.4 fixed the index problem. does not fix the problem completely, but the NullPointerException with subsequent index rebuild occurs only occasionally.

idea.log

[MaXal]

We had to disable .ignore plugin versions 2.4.0 and 2.5.0 in 2018.1.1 released due to the issue since it causes serious problems for many customers: https://youtrack.jetbrains.com/v2/issue/IDEA-189189

[nicity]

Btw,
IgnoreEntryOccurrence.deserialize seems to catch all IOExceptions and return null when deserializing the value from index

[ogonkov]

Affects PyCharm 2018.1

[hsz]

@MaXal
Thanks for letting me know.

@nicity
Ok, so simply passing exception should do the work, right?
It is interesting that for the previous IDE versions we didn't face this kind of issue. :(

[nicity]

@hsz Previous IDE versions have the same NPE, number 1 in my previous post #491 (comment)

[nicity]

@hsz Please take a look.
I found more serious problem in IgnoreEntryOccurrence: the value of an index should be immutable with persistent hashCode / equals (value is used as Key in a Map), serializing / deserializing should produce equals value instances.
But IgnoreEntryOccurrence contains a list of Pair<java.util.regex.Matcher,Boolean> that a) is mutable (Matcher instance is) b) serializing Matcher and deserializing it back will produce nonequal Matcher instances (Matcher has identity equals)!

[hsz]

@nicity Thanks for this summary. I've made a few changes - available in separated branch: 24d0b06

I'd also appreciate if someone would install following package and verify if issues are resolved.
idea-gitignore-2.6.0-RC.1.zip

[Frederick888]

@hsz I installed the attached plugin in PhpStorm 2018.1.1. The exceptions no longer show up but every time I run some commands that update ignored files, e.g. phpunit, they get re-indexed.

2018-04-11 10:57:29,184 [ 116065]   INFO - g.FileBasedIndexProjectHandler - Reindexing refreshed files: 143 to update, calculated in 4ms 
2018-04-11 10:57:32,695 [ 119576]   INFO - .diagnostic.PerformanceWatcher - Reindexing refreshed files took 3509ms; general responsiveness: ok; EDT responsiveness: ok 

Is it related to this issue?

Edit: nvm, I manually added it excluded directories and it's fine now. I don't think I've ever touched this setting before tho...

[nicity]

@hsz Please release 2.6.0 once this issue is fixed, current 2.6.0 RC doesn't address serious problem discovered and the plugin will continue producing NPEs => will be disabled again

[hsz]

@nicity

IgnoreEntryOccurrence is now immutable:

• instead of storing items in ArrayList - it's array now,
• I've moved from Matcher to Pattern which is serializable
• equals/hashCode are correctly calculated

Regarding NPE - I've removed try-catch and IOException will be handled by IDE directly - so I don't return null anymore. I didn't link this commit properly before - here it is: 8a4797a
Correct me if I'm wrong, but it should resolve this NPE problem, right?

[nicity]

@hsz Changes for Try / catch / IOException aren't enough for the originally reported problem: 'value' can be null inside indices. I commented on code because it masks other problems: e.g. readUTF throws exception for large strings.

Mutable IgnoreEntryOccurrence (as it breaks index contract) leads to NPE problem because Value are keys inside Map and mutability breaks key's hashcode.

Added several comments for your changes in 24d0b06

[hsz]

All right - everything is resolved, refactored and tesed.

v2.6.0 is built and released.
Thanks @nicity !