Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(Turborepo): Make package discovery async, and apply a debouncer #8058

Merged
merged 5 commits into from
Apr 30, 2024

Conversation

gsoltis
Copy link
Contributor

@gsoltis gsoltis commented Apr 29, 2024

Description

  • move the debouncer to its own module so that it is reusable
  • apply the debouncer to package discovery
  • make package discovery async so it doesn't block the file watching channel

Testing Instructions

Existing test suite.

Fixes #3455

Closes TURBO-2909

Copy link

vercel bot commented Apr 29, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
examples-nonmonorepo ✅ Ready (Inspect) Visit Preview 💬 Add feedback Apr 30, 2024 6:16pm
rust-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Apr 30, 2024 6:16pm
8 Ignored Deployments
Name Status Preview Comments Updated (UTC)
examples-basic-web ⬜️ Ignored (Inspect) Visit Preview Apr 30, 2024 6:16pm
examples-designsystem-docs ⬜️ Ignored (Inspect) Visit Preview Apr 30, 2024 6:16pm
examples-gatsby-web ⬜️ Ignored (Inspect) Visit Preview Apr 30, 2024 6:16pm
examples-kitchensink-blog ⬜️ Ignored (Inspect) Visit Preview Apr 30, 2024 6:16pm
examples-native-web ⬜️ Ignored (Inspect) Visit Preview Apr 30, 2024 6:16pm
examples-svelte-web ⬜️ Ignored (Inspect) Visit Preview Apr 30, 2024 6:16pm
examples-tailwind-web ⬜️ Ignored (Inspect) Visit Preview Apr 30, 2024 6:16pm
examples-vite-web ⬜️ Ignored (Inspect) Visit Preview Apr 30, 2024 6:16pm

Copy link
Contributor

github-actions bot commented Apr 29, 2024

🟢 Turbopack Benchmark CI successful 🟢

Thanks

Copy link
Contributor

github-actions bot commented Apr 29, 2024

🟢 CI successful 🟢

Thanks

@weyert
Copy link
Contributor

weyert commented Apr 29, 2024

Could this solve the issue I am experiencing?

2024-04-29T22:41:44.167258Z ERROR turborepo_lib::daemon::server: package changes stream closed: channel closed
pnpm exec turbo watch build --filter={packages/\*}
• Packages in scope: <snip>
• Running build in 36 packages
• Remote caching enabled
  × failed to connect to daemon
  ╰─▶ server is unavailable: channel closed
pnpm exec turbo daemon status
✓ daemon is running
log file: /Users/xxx/Development/Projects/monorepo/.turbo/daemon/f5b0831ce8ff9a15-turbo.log.2024-04-29
uptime: 2h 8m 22s 976mss
pid file: /var/folders/1m/4921_csd7gnddqdr0smhfqjh0000gn/T/turbod/f5b0831ce8ff9a15/turbod.pid
socket file: /var/folders/1m/4921_csd7gnddqdr0smhfqjh0000gn/T/turbod/f5b0831ce8ff9a15/turbod.sock

@gsoltis
Copy link
Contributor Author

gsoltis commented Apr 29, 2024 via email

Copy link
Contributor

@arlyon arlyon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few style items and a quick Q

crates/turborepo-filewatch/src/package_watcher.rs Outdated Show resolved Hide resolved
};
let initial_discovery = match discovery.discover_packages().await {
Ok(discovery) => discovery,
// If we failed the discovery, that's fine, we've reset the values, leave them as None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets just get rid of the references to None here since they aren't relevant now that we've pulled out the fn

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, done.

}
}
tracing::trace!("package watcher state: {:?}", state);
}
}

fn handle_discovery_result(&self, package_result: DiscoveryResult, state: &mut State) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What should the behaviour be if we get a DiscoveryResult with valid State? Should our state perhaps also store a version and be updated?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We won't ever get an update with a higher version. Each time we allocate a discovery request, we set the state to pending, with that version, and don't set it back to ready until get a matching version. That means if we're setting it to ready, there are no higher-versioned requests outstanding.

Co-authored-by: Alexander Lyon <arlyon@me.com>
@weyert
Copy link
Contributor

weyert commented Apr 30, 2024

Would it be possible to create a canary build that will include this PR when its merged? Love to try this change

@gsoltis gsoltis merged commit 3db7af3 into main Apr 30, 2024
56 checks passed
@gsoltis gsoltis deleted the gsoltis/package_watcher_debouncer branch April 30, 2024 20:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Daemon should have been opt in, not opt out with --no-daemon, add config opt out in turbo.json
4 participants