Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update java version and fix memory leak #714

Merged
merged 20 commits into from
May 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
85717a7
Update java version and fix memory leak
JohannesLichtenberger Apr 15, 2024
cfbd90e
Update gradle.properties
JohannesLichtenberger Apr 15, 2024
5c4764c
Update Kotlin
JohannesLichtenberger Apr 16, 2024
2589471
Merge remote-tracking branch 'origin/update-to-java22' into update-to…
JohannesLichtenberger Apr 16, 2024
12529bb
Merge branch 'main' into update-to-java22
JohannesLichtenberger Apr 17, 2024
cd6651d
Add a test for N-read-only trxs traversing a resource in parallel...
JohannesLichtenberger May 4, 2024
dc6890f
Merge remote-tracking branch 'origin/update-to-java22' into update-to…
JohannesLichtenberger May 4, 2024
775dc41
Update build.gradle
JohannesLichtenberger May 21, 2024
6586af2
Merge branch 'main' into update-to-java22
JohannesLichtenberger May 21, 2024
7ced6ff
Update gradle.yml
JohannesLichtenberger May 21, 2024
85cba02
Update gradle-wrapper.properties
JohannesLichtenberger May 21, 2024
4a34d8c
Merge branch 'main' into update-to-java22
JohannesLichtenberger May 21, 2024
9587b00
Update gradle-wrapper.properties
JohannesLichtenberger May 21, 2024
347965c
Update JsonShredderTest.java
JohannesLichtenberger May 22, 2024
3d7961d
Update JsonShredderTest.java
JohannesLichtenberger May 22, 2024
5883846
Update JsonShredderTest.java
JohannesLichtenberger May 22, 2024
a384288
Update JsonShredderTest.java
JohannesLichtenberger May 22, 2024
a877ff5
Update build.gradle
JohannesLichtenberger May 22, 2024
b2bc723
Update NodePageReadOnlyTrx.java
JohannesLichtenberger May 22, 2024
d1d895b
Fix tests
JohannesLichtenberger May 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/gradle.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
uses: actions/setup-java@v2
with:
distribution: 'temurin'
java-version: '21'
java-version: '22'
- uses: actions/cache@v3
with:
path: |
Expand Down Expand Up @@ -73,7 +73,7 @@ jobs:
uses: actions/setup-java@v2
with:
distribution: 'temurin'
java-version: '21'
java-version: '22'
- name: Gradle deploy
run: ./gradlew travisReleaseSnapshot --info
- name: Login to Docker Hub
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -441,7 +441,7 @@ git clone https://github.com/sirixdb/sirix.git

or use the following dependencies in your Maven or Gradle project.

**SirixDB uses Java 21, thus you need an up-to-date Gradle (if you want to work on SirixDB) and an IDE (for instance IntelliJ or Eclipse). Also make sure to use the provided Gradle wrapper.**
**SirixDB uses Java 22, thus you need an up-to-date Gradle (if you want to work on SirixDB) or simply use the gradle wrapper and an IDE (for instance IntelliJ or Eclipse). Also make sure to use the provided Gradle wrapper.**

### Maven artifacts
At this stage of development, you should use the latest SNAPSHOT artifacts from [the OSS snapshot repository](https://oss.sonatype.org/content/repositories/snapshots/io/sirix/) to get the most recent changes.
Expand Down
16 changes: 9 additions & 7 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -96,15 +96,15 @@ subprojects {
withSourcesJar()
withJavadocJar()
toolchain {
languageVersion = JavaLanguageVersion.of(21)
languageVersion = JavaLanguageVersion.of(22)
}
}


javadoc {
failOnError = false
options {
addStringOption('-release', '21')
addStringOption('-release', '22')
addBooleanOption('-enable-preview', true)
}
}
Expand Down Expand Up @@ -260,11 +260,13 @@ subprojects {
"--add-opens=java.base/java.io=ALL-UNNAMED",
"--add-opens=java.base/java.util=ALL-UNNAMED",
"--add-opens=java.base/java.lang.reflect=ALL-UNNAMED",
"-XX:+UnlockDiagnosticVMOptions",
"-XX:+DebugNonSafepoints",
//"-XX:+UseShenandoahGC",
"-Xlog:gc*=debug:file=g1.log",
"-XX:+UseZGC",
"-XX:+ZGenerational",
"-verbose:gc",
// "-Xlog:gc*=debug:file=g1.log",
// "-XX:+UseZGC",
// "-XX:+ZGenerational",
// "-verbose:gc",
"-XX:+HeapDumpOnOutOfMemoryError",
//"-XX:HeapDumpPath=heapdump.hprof",
"-XX:+UseStringDeduplication",
Expand All @@ -274,7 +276,7 @@ subprojects {
"-XX:MaxDirectMemorySize=1g",
"-XX:+UnlockExperimentalVMOptions",
"-XX:+AlwaysPreTouch",
//"-XX:+UseLargePages",
"-XX:+UseLargePages",
"-XX:ReservedCodeCacheSize=1000m",
"-XX:+UnlockDiagnosticVMOptions",
// "-XX:+PrintInlining",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -197,15 +197,14 @@ private Page loadPage(final PageReference reference) {
}

if (page != null) {
putIntoPageCache(reference, page);
reference.setPage(page);
putIntoPageCacheIfItIsNotAWriteTrx(reference, page);
}
return page;
}

private void putIntoPageCacheIfItIsNotAWriteTrx(PageReference reference, Page page) {
assert reference.getLogKey() == Constants.NULL_ID_INT;
if (trxIntentLog == null && !(page instanceof UberPage)) {
private void putIntoPageCache(PageReference reference, Page page) {
if (!(page instanceof UberPage)) {
// Put page into buffer manager.
resourceBufferManager.getPageCache().put(reference, page);
}
Expand Down Expand Up @@ -449,9 +448,11 @@ public Page getRecordPage(@NonNull IndexLogKey indexLogKey) {
}

// Fourth: Try to get from resource buffer manager.
Page recordPageFromBuffer = getFromBufferManager(indexLogKey, pageReferenceToRecordPage);
if (recordPageFromBuffer != null) {
return recordPageFromBuffer;
if (trxIntentLog == null || indexLogKey.getIndexType() != IndexType.PATH_SUMMARY) {
Page recordPageFromBuffer = getFromBufferManager(indexLogKey, pageReferenceToRecordPage);
if (recordPageFromBuffer != null) {
return recordPageFromBuffer;
}
}

if (pageReferenceToRecordPage.getKey() == Constants.NULL_ID_LONG) {
Expand Down Expand Up @@ -528,9 +529,7 @@ private Page loadDataPageFromDurableStorageAndCombinePageFragments(@NonNull Inde
final VersioningType versioningApproach = resourceConfig.versioningType;
final Page completePage = versioningApproach.combineRecordPages(pages, maxRevisionsToRestore, this);

if (trxIntentLog == null) {
resourceBufferManager.getRecordPageCache().put(pageReferenceToRecordPage, (KeyValueLeafPage) completePage);
}
resourceBufferManager.getRecordPageCache().put(pageReferenceToRecordPage, (KeyValueLeafPage) completePage);

pageReferenceToRecordPage.setPage(completePage);
setMostRecentlyReadRecordPage(indexLogKey, completePage);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -592,10 +592,14 @@ private PageContainer prepareRecordPage(final @NonNegative long recordPageKey, f
indexType,
getResourceSession().getResourceConfig(),
pageRtx.getRevisionNumber());
final KeyValueLeafPage modifyPage = new KeyValueLeafPage(completePage);
final KeyValueLeafPage modifyPage = new KeyValueLeafPage(recordPageKey,
indexType,
getResourceSession().getResourceConfig(),
pageRtx.getRevisionNumber());
pageContainer = PageContainer.getInstance(completePage, modifyPage);
} else {
pageContainer = dereferenceRecordPageForModification(reference);
return pageContainer;
}

assert pageContainer != null;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ public PageTrx createPageTrx(

final TreeModifierImpl treeModifier = new TreeModifierImpl();
final TransactionIntentLogFactory logFactory = new TransactionIntentLogFactoryImpl();
final TransactionIntentLog log = logFactory.createTrxIntentLog(resourceConfig);
final TransactionIntentLog log = logFactory.createTrxIntentLog(bufferManager, resourceConfig);

// Create revision tree if needed. Note: This must happen before the page read trx is created.
if (uberPage.isBootstrap()) {
Expand Down Expand Up @@ -187,11 +187,6 @@ public PageTrx createPageTrx(
log.put(newRevisionRootPage.getDeweyIdPageReference(), PageContainer.getInstance(deweyIDPage, deweyIDPage));
}

// final Page indirectPage =
// pageRtx.dereferenceIndirectPageReference(newRevisionRootPage.getIndirectDocumentIndexPageReference());
// log.put(newRevisionRootPage.getIndirectDocumentIndexPageReference(),
// PageContainer.getInstance(indirectPage, indirectPage));

final var revisionRootPageReference = new PageReference();
log.put(revisionRootPageReference, PageContainer.getInstance(newRevisionRootPage, newRevisionRootPage));
uberPage.setRevisionRootPageReference(revisionRootPageReference);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
package io.sirix.access.trx.page;

import io.sirix.access.ResourceConfiguration;
import io.sirix.cache.BufferManager;
import io.sirix.cache.TransactionIntentLog;

/**
Expand All @@ -37,11 +38,12 @@ public interface TransactionIntentLogFactory {

/**
* Create a new transaction intent log
*
*
* @param bufferManager the buffer manager
* @param resourceConfig the resource configuration to get the path for the log and a byte handler
* pipeline
* @return the transaction intent log instance
* @throws NullPointerException if the {@code resourceConfig} parameter is null
*/
TransactionIntentLog createTrxIntentLog(ResourceConfiguration resourceConfig);
TransactionIntentLog createTrxIntentLog(BufferManager bufferManager, ResourceConfiguration resourceConfig);
}
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
package io.sirix.access.trx.page;

import io.sirix.access.ResourceConfiguration;
import io.sirix.cache.BufferManager;
import io.sirix.cache.TransactionIntentLog;

/**
Expand All @@ -42,7 +43,7 @@ public TransactionIntentLogFactoryImpl() {
}

@Override
public TransactionIntentLog createTrxIntentLog(final ResourceConfiguration resourceConfig) {
return new TransactionIntentLog(1 << 12);
public TransactionIntentLog createTrxIntentLog(final BufferManager bufferManager, final ResourceConfiguration resourceConfig) {
return new TransactionIntentLog(bufferManager, 1 << 12);
}
}
15 changes: 7 additions & 8 deletions bundles/sirix-core/src/main/java/io/sirix/cache/PageCache.java
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
import com.github.benmanes.caffeine.cache.Caffeine;
import com.github.benmanes.caffeine.cache.RemovalCause;
import com.github.benmanes.caffeine.cache.RemovalListener;
import io.sirix.page.IndirectPage;
import io.sirix.page.PageReference;
import io.sirix.page.*;
import io.sirix.page.interfaces.Page;
import io.sirix.settings.Constants;

import java.util.Map;
import java.util.concurrent.TimeUnit;
Expand Down Expand Up @@ -35,17 +35,16 @@ public void clear() {
@Override
public Page get(PageReference key) {
var page = pageCache.getIfPresent(key);

// if (page instanceof IndirectPage indirectPage) {
// page = new IndirectPage(indirectPage);
// }

return page;
}

@Override
public void put(PageReference key, Page value) {
pageCache.put(key, value);
if (!(value instanceof RevisionRootPage) && !(value instanceof PathSummaryPage) && !(value instanceof PathPage)
&& !(value instanceof CASPage) && !(value instanceof NamePage)) {
assert key.getKey() != Constants.NULL_ID_LONG;
pageCache.put(key, value);
}
}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,11 @@ public final class TransactionIntentLog implements AutoCloseable {
*/
private final List<PageContainer> list;

/**
* The buffer manager.
*/
private final BufferManager bufferManager;

/**
* The log key.
*/
Expand All @@ -28,7 +33,8 @@ public final class TransactionIntentLog implements AutoCloseable {
*
* @param maxInMemoryCapacity the maximum size of the in-memory map
*/
public TransactionIntentLog(final int maxInMemoryCapacity) {
public TransactionIntentLog(final BufferManager bufferManager, final int maxInMemoryCapacity) {
this.bufferManager = bufferManager;
logKey = 0;
list = new ArrayList<>(maxInMemoryCapacity);
}
Expand Down Expand Up @@ -56,6 +62,9 @@ public PageContainer get(final PageReference key) {
* @param value a value to be associated with the specified key
*/
public void put(final PageReference key, final PageContainer value) {
bufferManager.getRecordPageCache().remove(key);
bufferManager.getPageCache().remove(key);

key.setKey(Constants.NULL_ID_LONG);
key.setPage(null);
key.setLogKey(logKey);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ private PathSummaryReader(final PageReadOnlyTrx pageReadTrx,
moveTo(pathNode.getNodeKey());
assert this.getNodeKey() == pathNode.getNodeKey();
qnmMapping.computeIfAbsent(this.getName(), (unused) -> new HashSet<>()).add(pathNode);
assert Objects.equals(this.getName(), pathNode.getName());
// assert Objects.equals(this.getName(), pathNode.getName());
}

moveToDocumentRoot();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@
import io.sirix.node.json.*;
import io.sirix.node.xml.*;
import io.sirix.page.KeyValueLeafPage;
import io.sirix.service.xml.xpath.AtomicValue;
import io.sirix.settings.Constants;
import io.sirix.settings.Fixed;
import it.unimi.dsi.fastutil.longs.LongArrayList;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -276,7 +276,7 @@ public byte[][] slots() {
}

@Override
public synchronized void setSlot(byte[] recordData, int offset) {
public void setSlot(byte[] recordData, int offset) {
slots[offset] = recordData;
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,9 @@ public <V extends DataRecord, T extends KeyValuePage<V>> PageContainer combineRe
modifiedPage.setDeweyId(deweyIds[i], i);
}

return PageContainer.getInstance(completePage, modifiedPage);
final var pageContainer = PageContainer.getInstance(completePage, modifiedPage);
log.put(reference, pageContainer);
return pageContainer;
}

@Override
Expand Down
Loading
Loading