Active page cache warmup #10957
Active page cache warmup #10957
Conversation
| @@ -197,6 +197,8 @@ public String threadName( Map<String, String> metadata ) | |||
| */ | |||
| public static Group cypherWorker = new Group( "CypherWorker" ); | |||
|
|
|||
| public static Group pageCacheIOHelper = new Group( "PageCacheIOHelper" ); | |||
| @Internal | ||
| @Description( "The profiling frequency for the page cache. Accurate profiles allow the page cache to do active " + | ||
| "warmup after a restart, reducing the mean time to performance." ) | ||
| public static Setting<Duration> pagecache_warmup_profiling_interval = |
| @@ -159,6 +159,13 @@ public static String duration( long durationMillis, TimeUnit highestGranularity, | |||
| } | |||
| } | |||
|
|
|||
| if ( builder.length() == 0 ) | |||
|
|
||
| try ( OutputStream outputStream = compressedOutputStream( outputNext ); | ||
| PageCursor cursor = file.io( 0, PF_SHARED_READ_LOCK | PF_NO_FAULT ) ) | ||
| { |
There was a problem hiding this comment.
While correct, all this shifting makes me dizzy, might I suggest:
{
int stepper = 0;
int b = 0;
for ( ; ; )
{
if ( stopped )
{
return pagesInMemory;
}
if ( !cursor.next() )
{
break; // Exit the loop if there are no more pages.
}
if ( cursor.getCurrentPageId() != PageCursor.UNBOUND_PAGE_ID )
{
pagesInMemory++;
b |= (1 << stepper);
}
stepper++;
if ( stepper == 8 )
{
outputStream.write( b );
b = 0;
stepper = 0;
}
}
outputStream.write( b );
outputStream.flush();
}
instead?
|
|
||
| private CompressorStreamFactory compressors() | ||
| { | ||
| return new CompressorStreamFactory( true, 1024 ); |
| private File profileOutputFileFinal( PagedFile file ) | ||
| { | ||
| File mappedFile = file.file(); | ||
| String profileOutputName = "." + mappedFile.getName() + SUFFIX_CACHEPROF; |
There was a problem hiding this comment.
Do we want the files to be hidden?
There was a problem hiding this comment.
I don't think people want to see these by default since they're not super important. I hope hiding them means we'll get fewer support cases that are just about asking what these files are.
| { | ||
| if ( profilingStarted.compareAndSet( false, true ) ) | ||
| { | ||
| // At this point, we are currently executing inside a *scheduled* executor thread. There are, at this time |
There was a problem hiding this comment.
This relies on too much undocumented implementation details of the Neo4jJobScheduler. If the problem is that we only have two threads, we can increase that.
If the purpose of the scheduled executor is to just spawn new jobs at a delay, refactor Neo4jJobScheduler to do this instead of implementing it here. If you had this problem others will surely follow, and this hidden relationship with the causal clustering is very easy to miss.
There was a problem hiding this comment.
Let's do that in a different PR.
| import java.io.Closeable; | ||
| import java.io.IOException; | ||
|
|
||
| abstract class PageLoader implements Closeable |
| import static org.hamcrest.Matchers.is; | ||
| import static org.junit.Assert.assertThat; | ||
|
|
||
| public class PageCacheWarmupCcIT extends PageCacheWarmupTestSupport |
There was a problem hiding this comment.
It really nice we get warmup for cluster members as well! 🎉
| for ( int i = 0; i < 1000; i++ ) | ||
| { | ||
| Node node = db.createNode( label ); | ||
| node.setProperty( "Niels", "Borh" ); |
|
Also, shouldn't this be tagged as |
|
@klaren We don't have very much feeling about how good a default |
With this, client code does not have to depend on, and call out to, the cursor tracer. So allows entanglement between modules to be reduced.
… for a value of zero. Now we instead produce a string that specifies the value zero of the lowest unit.
…ache data, in an attempt at warming the page cache up faster after a restart.
…uster store copies.
|
@klaren I believe I've addressed all of your comments. |
[cl] Introduces a new active page cache warmup feature in the Enterprise Edition. The memory contents of the Neo4j page cache are periodically profiled, and these profiles will then be used to quickly reheat the cache on startup. This decreases the time-to-performance for restarts or crashes.