Migrate from pnpm to Bun Package Manager in Directus for Enhanced Performance and Efficiency

[vamsii777]

Summary

Bun is a modern, fast, and efficient package manager designed to be production-ready. It offers advantages in speed, disk usage, and memory footprint compared to pnpm, the current package manager used in Directus. The proposal aims to migrate Directus from pnpm to Bun to leverage these performance and efficiency gains, thereby enhancing the overall development and deployment experience.

Performance Metrics

Based on the Bun v1.0, Bun shows significant improvements in various performance metrics:

Note: Bun runs a "hello world" TypeScript file 5x faster than esbuild with Node.js.

By migrating to Bun, Directus can expect:

• Faster Install Times: Speed up your development and deployment cycles.
• Reduced Disk Usage: Less disk space required for your dependencies.
• Lower Memory Footprint: Reduced memory usage for running commands.

Why Migrate to Bun?

1. Speed: Bun is designed for speed, making it ideal for large-scale projects that require quick installs.
2. Efficiency: With reduced disk and memory usage, Bun is more efficient than pnpm.
3. Security: Bun has built-in security features to ensure package integrity.

Basic Example

No response

Motivation

The primary motivation for proposing the migration from pnpm to Bun is to address several constraints and challenges that are inherent with the current package management system in Directus:

Speed

• Constraint: Current install times with pnpm can be a bottleneck in development and deployment cycles.
• Use Case: Faster installs would benefit both developers and automated CI/CD pipelines, leading to quicker releases.

Note: Benchmarked by installing the dependencies for a starter Remix project.

Efficiency

• Constraint: Disk and memory usage are concerns in environments with limited resources.
• Use Case: Reduced disk and memory footprint would make Directus more suitable for low-resource environments like Docker containers or cloud-based services.

Security

• Constraint: While pnpm is secure, there's always room for improvement in package integrity and trustworthiness.
• Use Case: Enhanced security features would make Directus more appealing for enterprise-level applications where security is a high priority.

Simplified Workflow

• Constraint: The current package manager may have a steeper learning curve or may require additional steps for certain tasks.
• Use Case: A more straightforward and intuitive package manager would lower the barrier to entry for new developers and simplify the development workflow.

Expected Outcome

The expected outcome is a more efficient, secure, and developer-friendly environment for Directus, which could lead to faster development cycles, lower operational costs, and higher adoption rates.

By focusing on these constraints, alternative solutions can also be developed if migrating to Bun is not feasible. For example, optimizing the current pnpm setup or considering other package managers that address these constraints could be alternative routes.

Detailed Design

The migration from pnpm to Bun in Directus involves several key steps, each with its own set of considerations and corner-cases. Below is a detailed design plan that outlines these steps.

Step 1: Preliminary Assessment and Backup

• Specifics: Before making any changes, assess the current state of the project. Make backups of package.json, pnpm-lock.yaml, and any custom scripts that rely on pnpm.

Step 2: Install Bun and Uninstall pnpm

• Specifics: Install Bun using curl as shown in the blog. Uninstall pnpm to avoid any conflicts.

  curl -sf https://bun.sh | sh
  npm uninstall -g pnpm

• Additional Information: Bun is not just a package manager; it's a comprehensive tool that can replace various other tools and libraries in your development workflow. Here's what you can unlearn or replace when you adopt Bun:

  Tools and Libraries Made Unnecessary by Bun

  • Node.js Replacements

    • node → Bun itself
    • npx → bunx (5x faster)
    • nodemon → Bun's built-in watch mode
    • dotenv, cross-env → Bun reads .env files by default

  • Transpilers

    • tsc → Only needed for type checking
    • babel, .babelrc, @babel/preset-* → Not needed
    • ts-node, ts-node-esm → Not needed
    • tsx → Not needed

  • Bundlers

    • esbuild → Not needed
    • webpack → Not needed
    • parcel, .parcelrc → Not needed
    • rollup, rollup.config.js → Not needed

  • Package Managers

    • npm, .npmrc, package-lock.json → Not needed
    • yarn, yarn.lock → Not needed
    • pnpm, pnpm.lock, pnpm-workspace.yaml → Not needed
    • lerna → Not needed

  • Testing Libraries

    • jest, jest.config.js → Not needed
    • ts-jest, @swc/jest, babel-jest → Not needed
    • jest-extended → Not needed
    • vitest, vitest.config.ts → Not needed

Step 3: Update package.json

• Specifics: Update the scripts section in package.json to replace pnpm commands with Bun commands.

  "scripts": {
  -  "install": "pnpm install",
  +  "install": "bun install",
  }

Step 4: Migrate Lock Files

• Specifics: Bun uses its own lock file. You'll need to delete pnpm-lock.yaml and generate a new lock file using bun install.

Step 5: Update CI/CD Pipeline

• Specifics: In your GitHub Actions CI/CD pipeline, replace pnpm setup steps with Bun.

  - name: Setup Bun
    uses: oven-sh/setup-bun@v1

• Corner Case: If you have caching set up for pnpm, you'll need to update it for Bun.

Step 6: Update Documentation

• Specifics: All instances in the documentation that refer to pnpm should be updated to refer to Bun.

Step 7: Testing

• Specifics: Thoroughly test the project to ensure that all dependencies are correctly installed and that the application behaves as expected.

• Corner Case: Pay special attention to any native modules or linked packages, as these could behave differently when managed by Bun.

Step 8: Rollout

• Specifics: Once testing is complete, the changes can be merged into the main branch.

New Terminology

• Bun Lock File: The lock file generated by Bun, similar to pnpm-lock.yaml but specific to Bun.

Requirements List

To ensure the successful migration from pnpm to Bun in Directus, the following requirements have been identified:

Must-Have

1. Successful Installation of Bun: Bun must be successfully installed in the development, staging, and production environments.
2. Dependency Migration: All existing dependencies must be successfully migrated from pnpm to Bun without any conflicts.
3. CI/CD Pipeline Update: The CI/CD pipeline must be updated to use Bun for installing dependencies.
4. Testing: Comprehensive tests must be conducted to ensure that the application behaves as expected after the migration.

Should-Have

1. Documentation Update: All instances in the documentation that refer to pnpm should be updated to refer to Bun.
2. Performance Benchmarking: Conduct performance tests to validate the benefits of migrating to Bun, such as faster install times and reduced resource usage.

Could-Have

1. Migration Guide: A detailed migration guide could be provided to help existing Directus developers transition from pnpm to Bun.
2. Community Feedback: Collect feedback from the Directus community to identify any potential issues or improvements.

Won't-Have

1. Automatic Migration Scripts for External Plugins: While it would be nice to have, creating automatic migration scripts for external Directus plugins that rely on pnpm is not a priority for this migration.

By fulfilling these requirements, the migration from pnpm to Bun can be considered a success, offering benefits like speed, efficiency, and potentially better security.

Drawbacks

While the migration from pnpm to Bun offers several advantages, there are also potential drawbacks that need to be considered:

Implementation Cost

• Code Size and Complexity: The migration process may introduce new complexities, especially if custom scripts or configurations dependent on pnpm are in use. This could increase the codebase size and make maintenance more challenging.

User Space Implementation

• Limited to Core Directus: The migration would primarily affect the core Directus project. Individual users or organizations using Directus might still prefer pnpm or other package managers, making the feature less universally applicable.

Educational Impact

• Learning Curve: Existing Directus developers familiar with pnpm would need to learn how to use Bun, which could slow down development initially.

Integration with Existing and Planned Features

• Compatibility Issues: If Directus has plans for features that are tightly integrated with pnpm, migrating to Bun could complicate those plans.

Migration Costs

• Potential Breaking Changes: Depending on how tightly integrated pnpm is with existing Directus applications, the migration could introduce breaking changes, requiring users to update their projects.

• Resource Allocation: Time and resources would need to be allocated for the migration, including development, testing, and documentation updates. This could delay other planned features or improvements for Directus.

Alternatives

Stick to pnpm

One alternative to migrating to Bun is to stick with the current package manager, pnpm. Below are some considerations for this option:

Advantages of Sticking to pnpm

• No Migration Costs: There would be no need to allocate resources for migration, allowing the team to focus on other features or improvements.

• No Learning Curve: Developers already familiar with pnpm would not have to spend time learning a new package manager, thereby avoiding any slowdown in development.

• Stable and Tested: pnpm is a stable and well-tested package manager that has been used in Directus for some time. Sticking to it would mean less risk of introducing new issues.

Disadvantages of Sticking to pnpm

• Missed Performance Gains: By not migrating to Bun, Directus would miss out on the potential performance and efficiency gains that Bun offers.

• Potential Future Migration: If Bun or another package manager becomes significantly better in the future, a migration might still be necessary, and the costs could be higher the longer it is delayed.

Impact of Not Doing This

• Status Quo: Directus would continue to operate as it currently does, without benefiting from the potential advantages that Bun offers.

• Potential for Future Re-evaluation: As Bun and other package managers continue to evolve, there may be future opportunities to reconsider this migration.

Adoption Strategy

The adoption of Bun as the new package manager in Directus would require a well-planned strategy to ensure a smooth transition for existing developers and projects. Here are some key points to consider:

Communication

• Announcements: Prior to the migration, communicate the upcoming changes to the Directus community through various channels like GitHub, forums, and social media.

Documentation

• Migration Guide: Provide a comprehensive migration guide that outlines the steps to transition from pnpm to Bun. This guide should cover installation, updating package.json, and other relevant changes.

• Updated Docs: All existing documentation that references pnpm should be updated to mention Bun, including setup guides, FAQs, and tutorials.

Tooling

• Migration Scripts: If possible, provide scripts that can automate the migration process, such as converting pnpm-lock.yaml to Bun's lock file format.

Versioning

• Major Version Bump: Given that this change could affect the build and deployment process, it would likely warrant a major version bump to indicate the breaking change.

Testing

• Beta Testing: Before the official release, offer a beta version that includes Bun for users to test and provide feedback.

Community and Ecosystem

• Plugin Compatibility: Assess how this change will affect existing plugins in the Directus ecosystem. Provide guidelines for plugin developers to update their projects for compatibility with Bun.

• Community Contributions: Encourage the community to contribute to the migration process, whether it's through code, testing, or documentation.

Unresolved Questions

No response

[rijkvanzanten]

There's a couple of major questions/difficulties that come to mind:

1. We should really use Bun's native features for things like file reads to get the best performance, which in turn:
2. It would prevent Directus from running in Node, as we'd be relying on Bun APIs
3. It might be very tricky to migrate over certain parts that we rely on native binaries for, like isolated-vm / argon2 / sharp. There will surely be Bun replacements / alternatives for those things, but they tap back into point 2. I don't think it's feasible in 2023 to have a codebase for the Directus API that will work across Node and Bun simultaneously.
4. This might cause some issues with extensions, but needs more research

Now if 2 is going to be a massive issue.. depends. We currently primarily point people to our Docker image as that's the most reliable across environments, in which case Bun would almost be a drop-in replacement. For people who are used to running it from "raw" node js (like in shared host environments), this would become a big problem 🤔

Long story short; I'm personally very interested in exploring the possibilities on relying on Bun as the primary runtime, as it's proven to be so insanely performant. I am wondering if it's feasible to do so though now, or if it would make more sense to hold back and let the Bun ecosystem mature a bit before committing to it for the long term

[falmanna]

I think it would be really nice to try benchmark the minimum runnable directus instance on Bun compared to the existing Nodejs runtime. If the results are as good as in theory, then it would be feasible to investigate further.

[br41nslug]

Another big blocker for us to migrate is pino (our logger) is not compatible with bun currently.

[Natetronn]

The Bun v1.0 blog post, to add some more context: https://bun.sh/blog/bun-v1.0

[rijkvanzanten]

A counter-point; to keep this a balanced discussion:

https://dev.to/thejaredwilcurt/bun-hype-how-we-learned-nothing-from-yarn-2n3j

[cdwmhcc]

Agree with most of this

[binaryben]

For people who are used to running it from "raw" node js

Is there a solid case for continuing to support this, even unofficially? I’m inclined to making the use of Docker or Kubernetes a hard requirement anyway

[br41nslug]

I would be hesitant to making Docker a hard requirement as it does add overhead that you may not want for running low-powered/embedded devices or is not supported on shared hosting.

Luckily that would not be required if switching to Bun is viable, then "raw nodejs" would simply become "raw bun".

[Natetronn]

I don't blame you. It would cut down support for you and the community quite a bit. Aside from the Docker specific related requests, that is.

[Natetronn]

Coincidentally, after binaryben's bump of this thread, Bun 1.1 with Windows support was released today: https://bun.sh/blog/bun-v1.1

That's a long blog post (I'm still reading it lol)