Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Synchronize evaluateGroupCached to avoid concurrent access to cache #2980

Merged
merged 5 commits into from
Jan 21, 2024

Conversation

lefou
Copy link
Member

@lefou lefou commented Jan 18, 2024

With this change, we synchronize evaluations of the same terminal task. This isn't necessarily needed for normal Mill executions, but in a BSP context, where multiple requests where handled concurrently in the same Mill instance, evaluating the same task concurrently can happen.

We don't synchronize multiple Mill-instances (e.g. run in two shells) or multiple evaluator-instances (which should have different out-dirs anyway).

Fix: #2818

Pull request: #2980

@lefou lefou marked this pull request as ready for review January 18, 2024 12:15
@lefou lefou requested a review from lihaoyi January 18, 2024 12:16
def lock(key: K, onCollision: Option[() => Unit] = None): AutoCloseable = {
lockKeys.synchronized {
var shown = false
while (!lockKeys.add(key)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to be a while? I thought we're inside lockKeys.synchronized, there should be no need to "try again" since nobody else else going to be able to mutate lockKeys

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think yes. At least, the Javadoc of the Object.wait method suggests to do so:

As in the one argument version, interrupts and spurious wakeups are possible, and this method should always be used in a loop:

synchronized (obj) {
    while (<condition does not hold>)
        obj.wait();
    ... // Perform action appropriate to condition
}

@lihaoyi
Copy link
Member

lihaoyi commented Jan 20, 2024

Using a file lock to allow mutex between different Mill processes would definitely also be nice. e.g. It's not uncommon for IntelliJ to be running a Mill process and for my shell to be running a Mill process. But intra-process mutex is already an improvement to the status quo so we can go ahead with this first

@lefou
Copy link
Member Author

lefou commented Jan 21, 2024

Using a file lock to allow mutex between different Mill processes would definitely also be nice. e.g. It's not uncommon for IntelliJ to be running a Mill process and for my shell to be running a Mill process. But intra-process mutex is already an improvement to the status quo so we can go ahead with this first

I think providing a file-based lock strategy is the next logical step, but as it still needs to be performed on a task-granualarity, it might come with some performance implications. Maybe, we want it to be configurable. Or just use it, if we detect, that there is another Mill running.

@lefou lefou merged commit a866173 into main Jan 21, 2024
38 checks passed
@lefou lefou deleted the task-concurrency branch January 21, 2024 21:41
@lefou lefou added this to the 0.11.7 milestone Jan 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BSP: tasks seem to be executed duplicately and concurrently
2 participants