feat: add function to calculate dynamic default Actor memory #570

danpoletaev · 2025-11-14T13:02:26Z

This PR introduces the logic to calculate a dynamic default Actor memory limit based on a provided expression string.

TL;DR:

Uses a sandboxed MathJS instance to securely evaluate expressions.
Supports {{variable}} placeholders with strict validation: input.* paths are allowed, while runOptions.* keys are strictly whitelisted(only 7 variables allowed).
Injects the Actor input and run options into the evaluation scope.
Adds a custom get(obj, path, default) helper to safely access nested properties without crashing on missing paths (e.g., get(input, 'startUrls.length', 0)).
Rounds the evaluated result to the closest valid power-of-2 memory tier. Result is clamped to min/max range. (128MB-32GB)

Note: when using this function in our API, we'll use it in worker_thread. I didn't add worker_thread to this package, because I suppose it can be used from client-side as well.

More context: Dynamic default Actor run memory notion spec

danpoletaev · 2025-11-18T13:57:30Z

Asking for review 👇
@tobice and @nmanerikar - Overall logic of the calculateDefaultMemoryFromExpression and how it aligns with the whole project Dynamic default Actor run memory notion spec

@barjin & @B4nan - Best practices regarding tsconfig and our tooling. Also we agreed with @barjin to check the tests, there're few tests failing after changing tsconfig.

Thank you 🙏

…ed environments (#573) Worker-based environments (e.g. `jest` tests) do not have access to `globalThis.require()` (while calling `require()` instead works). Deeper investigation shows that `tsup` doesn't transpile this in a dangerous way, so we can use this here. Unblocks apify/apify-client-js#782 (tested) Unblocks failing tests in #570

tsconfig.build.json

B4nan · 2025-11-18T14:41:40Z

package.json

        "jest": "^29.7.0",
        "lerna": "^9.0.0",
        "lint-staged": "^16.0.0",
+        "mathjs": "^15.1.0",


you need to add this to the package.json of the utilities package, root package.json is only for dev dependencies (and this is not a dev dependency)

this should be removed

packages/utilities/src/memory_calculator.ts

tsconfig.build.json

tobice

Hey there, really sorry for the delayed review.

The general approach makes sense 👍

Three main points:

Not sure about the package name.
Not sure about using the very specific LruCache.
The decisions we make around Math.js should be better documented.

Besides that, I left my usual nits and improvement suggestions.

packages/actor-memory-expression/src/index.ts

tobice · 2025-11-20T14:20:02Z

packages/math-utils/src/memory_calculator.ts

+
+import type { LruCache } from '../../datastructures/src/lru_cache';
+
+type ActorRunOptions = {


We don't have this type already somewhere?

tobice · 2025-11-20T14:22:52Z

packages/math-utils/src/memory_calculator.ts

+    input: Record<string, unknown>;
+}
+
+export const DEFAULT_MEMORY_MBYTES_MAX_CHARS = 1000;


Suggested change

export const DEFAULT_MEMORY_MBYTES_MAX_CHARS = 1000;

export const DEFAULT_MEMORY_MBYTES_EXPRESSION_MAX_LENGTH = 1000;

I think max length is a common term and more self-explanatory.

I already got myself confused by the name of this constant. MBYTES_MAX_CHARS... I was like what the hell is this 😅 Max length only makes sense on an expression, so let's mention it.

tobice · 2025-11-20T14:26:14Z

packages/actor-memory-expression/src/memory_calculator.ts

+ * A Set of allowed keys from ActorRunOptions that can be used in
+ * the {{runOptions.variable}} syntax.
+ */
+const ALLOWED_RUN_OPTION_KEYS = new Set<keyof ActorRunOptions>([


If you define the type for ActorRunOptions the way we usually do (or just manage to reuse an existing type), you won't need this.

Copy-pasted definition for ACTOR_PERMISSION_LEVEL:

export const ACTOR_PERMISSION_LEVEL = { FULL_PERMISSIONS: 'FULL_PERMISSIONS', LIMITED_PERMISSIONS: 'LIMITED_PERMISSIONS', }; export type ACTOR_PERMISSION_LEVEL = ValueOf<typeof ACTOR_PERMISSION_LEVEL>;

You should be able to get the keys with just Object.keys(ACTOR_PERMISSION_LEVEL).

What about this one? 👋

packages/math-utils/src/memory_calculator.ts

test/memory_calculator.test.ts

tobice · 2025-11-20T15:10:23Z

packages/actor-memory-expression/src/memory_calculator.ts

+export const calculateDefaultMemoryFromExpression = (
+    defaultMemoryMbytes: string,
+    context: MemoryEvaluationContext,
+    options: { cache: LruCache<EvalFunction> } | undefined = undefined,


This means that we only support this very specific, synchronous cache. What if we want to cache it in Redis? Can we use a more generic interface?

packages/math-utils/src/memory_calculator.ts

B4nan · 2025-11-21T15:43:39Z

package.json

        "jest": "^29.7.0",
        "lerna": "^9.0.0",
        "lint-staged": "^16.0.0",
+        "mathjs": "^15.1.0",


this should be removed

B4nan · 2025-11-21T15:44:17Z

packages/actor-memory-expression/tsconfig.json

+  "include": ["src/**/*"],
+  "compilerOptions": {
+    "module": "node20",
+    "moduleResolution": "node16",


if you really need module: node20, the resolution should be also node20

i wonder if this is really needed, you only modified this config, which is used for your IDE, not for building the package (so no effect on users/production build)

tobice

Thanks for the changes 👍

I noticed some of my comments were left without a reaction and unless I missed something, they don't seem to be addressed. Pls take look.

tobice · 2025-11-24T10:39:58Z

packages/actor-memory-expression/src/memory_calculator.ts

+ * A Set of allowed keys from ActorRunOptions that can be used in
+ * the {{runOptions.variable}} syntax.
+ */
+const ALLOWED_RUN_OPTION_KEYS = new Set<keyof ActorRunOptions>([


What about this one? 👋

tobice · 2025-11-24T10:41:29Z

packages/actor-memory-expression/src/memory_calculator.ts

+    // extra (has functional impact)
+    // We disable evaluate to prevent users from calling it inside their expressions.
+    // For example: defaultMemoryMbytes = "evaluate('2 + 2')"
+    evaluate() { throw new Error('Function evaluate is disabled.'); },


compile is no longer here? Is that intentional?

tobice · 2025-11-24T10:42:21Z

packages/actor-memory-expression/src/memory_calculator.ts

+    evaluate() { throw new Error('Function evaluate is disabled.'); },
+    parse() { throw new Error('Function parse is disabled.'); },
+    simplify() { throw new Error('Function simplify is disabled.'); },
+    derivative() { throw new Error('Function derivative is disabled.'); },


I wonder: Why do we have to disable these? We have never installed them 😄 Or all these sort of "default", as in, always included?

tobice · 2025-11-24T10:45:22Z

packages/actor-memory-expression/src/memory_calculator.ts

+ *
+ * All `input.*` values are accepted, while `runOptions.*` are validated (7 variables from ALLOWED_RUN_OPTION_KEYS).
+ *
+ * Note: this approach allows developers to use a consistent double-brace


Suggested change

* Note: this approach allows developers to use a consistent double-brace

* Note: While not really needed for Math.js, this approach allows developers

* to use a consistent double-brace templating syntax `{{runOptions.timeoutSecs}}`

* across the Apify platform. We also want to avoid compiling the expression with the

* actual values as that would make caching less effective.

(opt) I added some more context.

tobice · 2025-11-24T10:47:14Z

packages/actor-memory-expression/src/memory_calculator.ts

+                return variableName;
+            }
+
+            // 3. Throw error for unrecognized variables (e.g. {{someVariable}})


As said, IMHO this particular error will never be thrown as the first check already takes care of that.

I'd remove the first check (not just the 1 number 😛 ).

tobice · 2025-11-24T10:48:47Z

packages/actor-memory-expression/src/types.ts

@@ -0,0 +1,22 @@
+import type { EvalFunction } from 'mathjs';
+
+export type ActorRunOptions = {


As asked before, don't we have this as an existing type?

tobice · 2025-11-24T10:50:05Z

packages/actor-memory-expression/src/types.ts

+    input: Record<string, unknown>;
+}
+
+export type CompilationCache = {


How about we make it async to make it more future proof? You won't be able to use e.g. Redis with the current interface.

feat: add function to calculate dynamic default Actor memory

19c8dd3

danpoletaev self-assigned this Nov 14, 2025

github-actions bot added this to the 127th sprint - Core Services team milestone Nov 14, 2025

github-actions bot added t-core-services Issues with this label are in the ownership of the core services team. tested Temporary label used only programatically for some analytics. labels Nov 14, 2025

danpoletaev added 5 commits November 18, 2025 10:50

refactor: improve preprocessing expression

5191978

refactor: clamp result to min/max range

4eef96c

refactor: clean up

95e11b5

refactor: clean up

457f3c1

Merge branch 'master' into feat/calculate-default-memory-from-expression

662a5d7

danpoletaev marked this pull request as ready for review November 18, 2025 12:16

danpoletaev added 3 commits November 18, 2025 13:21

refactor: update tsconfig.build.json

5075a6c

fix: build

bd0baa6

refactor: clean up

640167e

danpoletaev requested review from B4nan, barjin, nmanerikar and tobice November 18, 2025 13:57

barjin mentioned this pull request Nov 18, 2025

fix: improve createHmacSignatureAsync compatibility with worker-based environments #573

Merged

B4nan reviewed Nov 18, 2025

View reviewed changes

tsconfig.build.json Outdated Show resolved Hide resolved

B4nan reviewed Nov 18, 2025

View reviewed changes

barjin reviewed Nov 18, 2025

View reviewed changes

packages/utilities/src/memory_calculator.ts Show resolved Hide resolved

B4nan reviewed Nov 18, 2025

View reviewed changes

packages/utilities/src/memory_calculator.ts Outdated Show resolved Hide resolved

danpoletaev added 4 commits November 19, 2025 10:35

Merge branch 'master' into feat/calculate-default-memory-from-expression

7390d1b

refactor: move mathjs logic to a new package

9727373

fix: tsconfig

2dcee65

refactor: clean up

823b188

B4nan reviewed Nov 19, 2025

View reviewed changes

tsconfig.build.json Outdated Show resolved Hide resolved

danpoletaev added 3 commits November 19, 2025 16:40

refactor: clean up

3dbb353

refactor: change tsconfig of math-utils

9b43107

refactor: change names of functions

3b62b9f

tobice requested changes Nov 20, 2025

View reviewed changes

danpoletaev added 4 commits November 21, 2025 10:20

refactor: clean up

a23f386

refactor: clean up

543f278

refactor: make cache generic

f0d1f54

fix: build

89543ce

danpoletaev requested review from B4nan and tobice November 21, 2025 15:41

B4nan requested changes Nov 21, 2025

View reviewed changes

danpoletaev added 3 commits November 21, 2025 16:48

refactor: clean up

805365c

refactor: clean up

f6bdccc

refactor: clean up

e6c340e

danpoletaev requested a review from B4nan November 21, 2025 15:55

refactor: clean up

5ab9baf

B4nan approved these changes Nov 21, 2025

View reviewed changes

tobice requested changes Nov 24, 2025

View reviewed changes


		import type { LruCache } from '../../datastructures/src/lru_cache';

		type ActorRunOptions = {

	export const DEFAULT_MEMORY_MBYTES_MAX_CHARS = 1000;
	export const DEFAULT_MEMORY_MBYTES_EXPRESSION_MAX_LENGTH = 1000;

- * Note: this approach allows developers to use a consistent double-brace
+ * Note: While not really needed for Math.js, this approach allows developers
+ * to use a consistent double-brace templating syntax `{{runOptions.timeoutSecs}}`
+ * across the Apify platform. We also want to avoid compiling the expression with the
+ * actual values as that would make caching less effective.

		@@ -0,0 +1,22 @@
		import type { EvalFunction } from 'mathjs';

		export type ActorRunOptions = {

feat: add function to calculate dynamic default Actor memory #570

Are you sure you want to change the base?

feat: add function to calculate dynamic default Actor memory #570

Conversation

danpoletaev commented Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danpoletaev commented Nov 18, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tobice left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tobice Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tobice left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

danpoletaev commented Nov 14, 2025 •

edited

Loading

tobice Nov 20, 2025 •

edited

Loading