Allow storing scripts in multiple projects

[nielsbauman]

This commit makes the ScriptService, the ScriptCache, and the stored
script APIs project-aware. By adding the project ID to the cache key in
the ScriptCache, we avoid data leakage between projects. The method
ScriptService#compile has a lot of usages, so we introduce a temporary
method that hard-codes the default project ID. Follow-up PRs will update
the remaining usages to pass the correct project ID.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[rjernst]

ScriptService#compile has a lot of usages
Follow-up PRs will update the remaining usages to pass the correct project ID.

Compilation should normally be in the context of a user request. Shouldn't it then get the project id from the thread context?

[nielsbauman]

Compilation should normally be in the context of a user request. Shouldn't it then get the project id from the thread context?

@rjernst, thanks for chiming in! We usually try to explicitly pass the project ID around and minimize the reliance on the thread context, as it can be difficult to track down where the project ID is supposed to be set in a call chain. Since there are a lot of usages all over the codebase here, I'm open to considering using the thread context. Can you think of any usages where compilation happens outside of the context of a user request? A perhaps counterintuitive example is cluster state update tasks; those are run without the thread context of the request. Do you happen to know if we do any script compilation in cluster state update tasks, for example? I can do some investigation if you don't already happen to know.

[rjernst]

Can you think of any usages where compilation happens outside of the context of a user request?

One I'm not sure of is async search. I assume the search happens in the user context, even after being forked to the async threadpool as a task, but if not then that would be one. Otherwise, scripts are a way for users to customize functionality, so it wouldn't make sense to compile a script outside a user request.