Skip to content

Size the translator heap from the machine so large apps build unmodified #5521

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shai-almog merged 6 commits into from
Aug 5, 2026
+1,004 −52
Merged

Size the translator heap from the machine so large apps build unmodified #5521

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
fac0ce6
Size the translator heap from the machine so large apps build unmodified
shai-almog Aug 5, 2026
5b3e06b
Add the license header the copyright gate requires on the new test
shai-almog Aug 5, 2026
baacb59
Read the cgroup limit from this process's own cgroup, and stop racing…
shai-almog Aug 5, 2026
08dbd78
Stop the timeout watcher before draining output, so a finished proces…
shai-almog Aug 5, 2026
1d32800
Honour CN1_TRANSLATOR_OPTS on every target, from one shared parser
shai-almog Aug 5, 2026
1715287
Drive the executeProcess tests with JVM helpers instead of /bin/sh
shai-almog Aug 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions maven/codenameone-maven-plugin/spotbugs-exclude.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,20 @@
</Or>
</Match>

<!--
TranslatorHeap reads the container memory limit so a build inside a cgroup
sizes the translator heap against that limit rather than the host's RAM.
The two paths it probes (/sys/fs/cgroup/memory.max for cgroup v2 and
/sys/fs/cgroup/memory/memory.limit_in_bytes for v1) are fixed locations in
the Linux kernel's pseudo-filesystem. There is nothing to make relative or
configurable, and both reads are guarded by isFile() so a non-Linux host
simply falls through to physical RAM.
-->
<Match>
<Class name="com.codename1.builders.TranslatorHeap" />
<Bug pattern="DMI_HARDCODED_ABSOLUTE_FILENAME" />
</Match>

<!-- RichPropertiesReader wraps a properties file with helper lookups. -->
<Match>
<Class name="com.codename1.util.RichPropertiesReader" />
Expand Down
22 changes: 18 additions & 4 deletions maven/codenameone-maven-plugin/src/main/java/com/codename1/builders/Executor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1955,7 +1955,7 @@ private int executeProcess(ProcessBuilder pb, final int timeout, final StringBui
final boolean[] running = new boolean[]{true};

try {
new Thread() {
Thread reader = new Thread() {
public void run() {
try {
byte[] buffer = new byte[8192];
Expand All @@ -1971,7 +1971,8 @@ public void run() {
outputMessage.append("Exception on appending to log: " + ex);
}
}
}.start();
};
reader.start();
if (timeout > -1) {
new Thread() {
public void run() {
Expand All @@ -1992,11 +1993,24 @@ public void run() {
}.start();
}
int val = p.waitFor();
if (destroyed[0]) {
// Stop the timeout watcher FIRST. The process has already exited, so
// nothing it could do from here is useful -- and the join below can
// hold us here for a while, during which a watcher still counting
// would cross the deadline and flag a completed run as timed out.
running[0] = false;
boolean timedOut = destroyed[0];
// waitFor returns as soon as the process exits, but the reader thread
// may still be draining what is left in the pipe. Callers that inspect
// outputMessage after this returns (the translator's out-of-memory
// check, for one) would otherwise race the reader and read a partial
// tail -- and the tail is exactly where a JVM prints its
// OutOfMemoryError. Bounded so a wedged reader cannot hang the build;
// the process has already exited, so the stream reaches EOF promptly.
reader.join(30000);
Comment thread
shai-almog marked this conversation as resolved.
if (timedOut) {
log("Process timed out");
return 1;
}
running[0] = false;
log("Process return code is "+val);
return val;
} finally {
Expand Down
49 changes: 27 additions & 22 deletions maven/codenameone-maven-plugin/src/main/java/com/codename1/builders/IPhoneBuilder.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3508,28 +3508,21 @@ public void usesClassMethod(String cls, String method) {
// -Xmx) from the CN1_TRANSLATOR_OPTS environment variable. The
// forked JVM does not inherit the Maven process's -D properties,
// so this is the only way to reach the translator for tuning.
String translatorOpts = System.getenv("CN1_TRANSLATOR_OPTS");
boolean heapOverridden = false;
if (translatorOpts != null && !translatorOpts.trim().isEmpty()) {
for (String opt : translatorOpts.trim().split("\\s+")) {
if (!opt.isEmpty()) {
parparCmd.add(opt);
if (opt.startsWith("-Xmx")) {
heapOverridden = true;
}
}
}
}
// Default heap; a -Xmx in CN1_TRANSLATOR_OPTS takes precedence.
// The dead-code cull builds an in-memory suffix automaton over
// all native symbols (NativeSymbolIndex, from #5236) to avoid the
// old O(N^2) substring scan that timed out on large apps -- that
// index trades time for memory, so the historical 384m cap now
// OOMs local iOS builds as the CN1 class count grows (issue
// #5344). The cloud builder already runs the same translator at
// 1024m; match it here so local and server builds behave alike.
java.util.List<String> translatorOpts = TranslatorHeap.extraJvmOptions();
parparCmd.addAll(translatorOpts);
boolean heapOverridden = TranslatorHeap.specifiesHeap(translatorOpts);
// Heap sized from the machine (see TranslatorHeap), floored at the
// 1024m the cloud builder has always used; a -Xmx in
// CN1_TRANSLATOR_OPTS still takes precedence. The dead-code cull
// builds an in-memory suffix automaton over all native symbols
// (NativeSymbolIndex, from #5236) to avoid the old O(N^2) substring
// scan that timed out on large apps -- that index trades time for
// memory, so the historical 384m cap OOMed local iOS builds as the
// CN1 class count grew (issue #5344) and even 1024m is not enough
// for a large app (issue #5511).
int heapMB = TranslatorHeap.maxHeapMB(1024);
if (!heapOverridden) {
parparCmd.add("-Xmx1024m");
parparCmd.add("-Xmx" + heapMB + "m");
}
parparCmd.add("-jar");
parparCmd.add(parparVMCompilerJar);
Expand All @@ -3543,7 +3536,19 @@ public void usesClassMethod(String cls, String method) {
parparCmd.add(buildVersion);
parparCmd.add(request.getArg("ios.project_type", "ios")); // ios, iphone, ipad
parparCmd.add(addLibs);
if (!exec(userDir, env, 420000, parparCmd.toArray(new String[0]))) {
int outputMark = message.length();
// 600s, matching the cloud builder running the identical translator
// over the identical input. Translation time grows with app size, so
// the lower local 420s cap meant a large app could translate fine on
// the build server yet be killed mid-run on the developer's own
// machine -- which is usually the slower of the two.
if (!exec(userDir, env, 600000, parparCmd.toArray(new String[0]))) {
// Name the failure rather than leaving the build to report a
// bare "translator failed" -- an out-of-memory death is
// fixable by the developer, but only if we say so.
if (!heapOverridden && TranslatorHeap.looksOutOfMemory(message.substring(outputMark))) {
Comment thread
shai-almog marked this conversation as resolved.
error(TranslatorHeap.outOfMemoryAdvice(heapMB, true), null);
}
return false;
}
} catch (Exception ex) {
Expand Down
38 changes: 20 additions & 18 deletions maven/codenameone-maven-plugin/src/main/java/com/codename1/builders/JavaScriptBuilder.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -766,19 +766,8 @@ private boolean runByteCodeTranslator(File compilerJar, File stageClasses, File
// CN1_TRANSLATOR_OPTS environment variable. The forked JVM does
// not inherit the Maven process's -D properties, so this is the
// only way to reach the translator for bisection / tuning.
String translatorOpts = System.getenv("CN1_TRANSLATOR_OPTS");
boolean heapOverridden = false;
java.util.List<String> extraOpts = new java.util.ArrayList<String>();
if (translatorOpts != null && !translatorOpts.trim().isEmpty()) {
for (String opt : translatorOpts.trim().split("\\s+")) {
if (!opt.isEmpty()) {
extraOpts.add(opt);
if (opt.startsWith("-Xmx")) {
heapOverridden = true;
}
}
}
}
java.util.List<String> extraOpts = TranslatorHeap.extraJvmOptions();
boolean heapOverridden = TranslatorHeap.specifiesHeap(extraOpts);
// Hand the translator the JavaScript-port webapp (port.js, js/, style.css,
// ...) so it bundles port.js -- the worker-side native bindings that make
// Window.current() etc. resolve. Off-repo builds can't find it via the
Expand All @@ -797,11 +786,14 @@ private boolean runByteCodeTranslator(File compilerJar, File stageClasses, File
if (jsPortWebApp != null && jsPortWebApp.isDirectory() && !webAppOverridden) {
extraOpts.add("-Dcodename1.javascriptport.webapp=" + jsPortWebApp.getAbsolutePath());
}
// Default heap; a -Xmx in CN1_TRANSLATOR_OPTS takes precedence (apps
// that disable tree-shaking, e.g. the Playground, emit a much larger
// bundle and need a bigger heap to avoid OutOfMemoryError mid-emit).
// Heap sized from the machine (see TranslatorHeap); a -Xmx in
// CN1_TRANSLATOR_OPTS still takes precedence. The old fixed 512m was
// tuned against the sample apps and made a large app fail with
// OutOfMemoryError mid-emit unless the developer found and set
// CN1_TRANSLATOR_OPTS by hand (issue #5511).
int heapMB = TranslatorHeap.maxHeapMB(512);
if (!heapOverridden) {
cmd.add("-Xmx512m");
cmd.add("-Xmx" + heapMB + "m");
}
cmd.addAll(extraOpts);
cmd.add("-cp");
Expand All @@ -816,7 +808,17 @@ private boolean runByteCodeTranslator(File compilerJar, File stageClasses, File
cmd.add(version == null ? "1.0" : version);
cmd.add("ios");
cmd.add("none");
return exec(tmpDir, env, -1, cmd.toArray(new String[0]));
int outputMark = message.length();
if (exec(tmpDir, env, -1, cmd.toArray(new String[0]))) {
return true;
}
// Name the failure rather than leaving the build to report a bare
// "translator failed" -- an out-of-memory death is fixable by the
// developer, but only if we say so.
if (!heapOverridden && TranslatorHeap.looksOutOfMemory(message.substring(outputMark))) {
Comment thread
shai-almog marked this conversation as resolved.
error(TranslatorHeap.outOfMemoryAdvice(heapMB, true), null);
}
return false;
}

private File locateDistDir(File translatorOut, String translatorAppName) {
Expand Down
23 changes: 20 additions & 3 deletions maven/codenameone-maven-plugin/src/main/java/com/codename1/builders/LinuxNativeBuilder.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -288,13 +288,23 @@ public void usesClassMethod(String cls, String method) {

List<String> parparCmd = new ArrayList<String>();
parparCmd.add("java");
// 2g: the clean target's readNativeFiles loads both .m and .c (clean's
// Extra translator JVM options from CN1_TRANSLATOR_OPTS, honoured on every
// target so the escape hatch does not work on some and silently do nothing
// on others.
List<String> translatorOpts = TranslatorHeap.extraJvmOptions();
parparCmd.addAll(translatorOpts);
// Heap sized from the machine (see TranslatorHeap), floored at 2g: the
// clean target's readNativeFiles loads both .m and .c (clean's
// extension), so the in-memory native-source set is ~2x what the iOS
// target loads, and the markDependencies/NativeSymbolIndex pass needs
// headroom on top of that. Without the JavaAPI in classesDir the
// translator never reaches that stage at scale; once JavaAPI is added
// the older 768m cap GC-thrashes.
parparCmd.add("-Xmx2g");
// the older 768m cap GC-thrashes. A -Xmx in CN1_TRANSLATOR_OPTS wins.
boolean heapOverridden = TranslatorHeap.specifiesHeap(translatorOpts);
int heapMB = TranslatorHeap.maxHeapMB(2048);
if (!heapOverridden) {
parparCmd.add("-Xmx" + heapMB + "m");
}
parparCmd.add("-jar");
parparCmd.add(parparVMCompilerJar.getAbsolutePath());
parparCmd.add("clean");
Expand All @@ -310,7 +320,14 @@ public void usesClassMethod(String cls, String method) {
parparCmd.add("linux"); // project type
parparCmd.add("none"); // additional native frameworks (none on Linux)
try {
int outputMark = message.length();
if (!exec(tmpFile, 600000, parparCmd.toArray(new String[0]))) {
// Name the failure rather than leaving the build to report a bare
// "translator failed" -- an out-of-memory death is fixable by the
// developer, but only if we say so.
if (!heapOverridden && TranslatorHeap.looksOutOfMemory(message.substring(outputMark))) {
error(TranslatorHeap.outOfMemoryAdvice(heapMB, true), null);
}
return false;
}
} catch (Exception ex) {
Expand Down
Loading
Loading