Improving the developer experience by leveraging a dependency graph and abstracting away native implementation details

[pepicrft]

Introduction

At Shopify, React Native is our default technology for building mobile apps. We use the official React Native CLI in combination with some internal tooling to interact with their projects. The setup has been working fine, but the experience we’d like developers to have is not quite there yet. We think there’s some room for improvement and that’s what I’d like to discuss this issue.

Before jumping into the details, I’d like to first share what I mean when I say there’s room for improvement. In the following section, I’ll be putting myself in the shoes of our developers and share some of the inconveniences that developers stumble upon when building apps for iOS and Android using React Native and its official CLI.

Even though they are not be related to each other, the solution that I’m proposing further down will help mitigate all of them. Note that they are numerated so that I can reference them from the proposed solution.

Motivation

• 1. Migrations remain cumbersome: It’s a manual process that some developers follow in an “I-don’t-know-what-I’m-doing” mode: I’ve been told to put this line on the Podfile, add this other line on the AppDelegate, add this line in the Gradle file, and things should work. We’d like this process to be seamless so that developers feel encouraged to be in the latest version and therefore benefit from the constant improvements that are landing on the framework. Fully-automating the process is a dream, but I think we can take steps to get closer to that.
• 2. Auto-linking and CocoaPods: Many developers are not aware that auto-linking is necessary for their apps to compile. While auto-linking happens implicitly at build time, on iOS it requires an extra pod install command and a properly-configured Podfile to integrate the dependencies in the Xcode project. All it takes to go from “every works” state to “my app doesn’t compile” is forgetting about running pod install, changing something in the Xcode project, or having an invalid configuration in the Podfile. Internet is full of suggestions along the lines of “add this build setting” and it’ll work, or add this Ruby code to your Podfile and I assure you it’ll work. Moreover, there are also chances that someone uses the global Pod as opposed to the version specified in the Gemfile, leading to inconsistent results across environments. Bear in mind that I’m not blaming CocoaPods here. It does a great job, but the setup RN CLI + CocoaPods + Ruby environment will make providing a great developer experience challenging.
• 3. Binary caching across environments: More than an inconvenience, this is an improvement that I think I'd make a huge difference in regards to productivity. Assuming most of the time developers work on the Javascript layer, there’s no need to be compiling the native code when someone has done that already. That’s what Gradle and Xcode’s build system does locally by reusing past builds, but what if those artifacts can be shared with others? What if I’m a new developer in the team and I run react-native run ios and the time it takes to launch the app is the result of the time it takes to pull the binary and have it running on the simulator and pointing to my local Metro server?

Details

I think the React Native CLI could have an in-memory graph representation of the React Native packages that contain native code, and make compilation artifacts such as Gradle files or Xcode projects an implementation detail that is generated when needed and therefore doesn’t need to be part of the repository.

Projects would have a manifest file that describes the project. We can reuse the existing manifest for that or create a new one. The structure could be something along the lines of:

# Example of an app manifest

name: MyApp
targets:
  - name: MyApp
    product: app
    platform: iOS
    sources: iOS/**/*.{h.m}
    minimum_deployment_target: 13.4
    resources: Resources/iOS/**/*
    build_identifier: com.shopify.MyApp
    dependencies:
      - ReactNativeStorage
  - name: MyApp
    product: app
    platform: Android
    sources: Android/**/*.{kotlin}
    package_name: com.shopify.MyApp
      - ReactNativeStorage

# Example of a library manifest

name: ReactNativeStorage
targets:
  - name: ReactNativeStorage
    product: module # (alternatively library/package)
    platform: iOS
    sources: iOS/**/*.{h.m}
    minimum_deployment_target: 13.4
    resources: Resources/iOS/**/*
    build_identifier: com.shopify.MyApp
  - name: ReactNativeStorage
    product: module
    platform: Android
    sources: Android/**/*.{kotlin}
    package_name: com.shopify.MyApp

NPM is the source of truth of the dependency graph. However, manifests would explicitly define the native dependencies they’d like to have. Projects could have multiple targets (e.g. a target per supported platform), and the format of the targets would be different depending on the platform they are targeting. For example, while iOS identifies buildable units (targets) with what they call “build identifier”, Android uses package names.

Projects would not contain .podspec, Xcode Projects, nor Gradle files. They are generated when needed under a .react-native directory that is .gitignored. For example, if I run react-native run ios, the React Native CLI would know that we need an Xcode project for that, it’d generate it under that directory, compile it, and run it on the simulator. On iOS, the generated Xcode project would be a representation of the dependency graph, containing all the targets, linking settings, and phases. Therefore, it removes the need for having to install CocoaPods and run extra pod install steps. On Android, since the resolution of the dependency graph happens when Gradle launches the project, Gradle could proxy with the CLI to get the graph and translate that to build steps. All that logic could be implemented in Kotlin. By doing this we’d be tackling point 2. from above, Auto-linking and CocoaPods, but we’d need to do a fair amount of work to re-implement all the linking logic that CocoaPods has developed and improved over the years. Luckily, that’s something I’ve done recently in one of my side projects, Tuist, and I have a lot of contexts to translate that to the Javascript domain.

By doing this, we are also easing the migrations (point 1 from above) since developers no longer have to know what changes need to be made to their Xcode projects, Podfiles, or Build.gradle files. The React Native CLI would combine the dependency graph with the React Native version the project is using, and generate the right project for it. For instance, if the version of React Native comes with Hermes support for iOS, the generated Xcode project would contain the macros, and the right linking to enable Hermes. Developers don’t have to do anything.

Moreover, having a dependency graph allows for fingerprinting. Each target would have a fingerprint that is the result of fingerprinting the target’s settings, files, and the fingerprint of its dependencies. Why is that useful? If an app has a fingerprint, we can extend the react-native run ios to look up the binary in a remote cache using the fingerprint. For example, building upon the above example, we’d get the following fingerprint for MyApp, 1234 , so if the MyApp-1234.tar.gz exists in a remote cache, I’d take that one instead of going through the normal flow of generating, building and launching the project. Teams could set up a remote CI pipeline with the sole goal of warming the cache. This would tackle the point 3, Binary caching across environments.

Other benefits

I believe going down this path has some other benefits that are worth considering when evaluating it.
First, we’d ease the creation of libraries for React Native. A developer would only need to add a manifest file, a package.json, and the source code (JS/Typescript, Swift, Kotlin…). No Xcode projects, no Podspecs, Build.gradle files. We could add a new command, react-native edit that opens either Xcode or Android studio to edit the project.

Moreover, we could add graph validation. This is something that Gradle and CocoaPods do, but the errors that they throw, although accurate, are not actionable for React Native developers. We could make sure that the errors are clear and actionable and don’t let developers compile their projects if we know 100% that they won’t compile.

Another interesting idea that we recently explored in Tuist is the idea of focusing on projects (I believe Facebook has something similar). This is something that’d be possible if we have a graph. The idea being is that as a developer I can get a native project with a focus on one of the targets of the dependency tree. For example, let’s say that I’m having issues with one of the dependencies that I’m using, ReactNativeWhatever. I could run react-native focus ReactNativeWhatever and I’d get a project generated on-the-fly that would open on my editor and where I can build and test the native code easily.

Discussion points

• Is this a potential direction the React Native CLI could take?
• Is it worth re-implementing part of the work that CocoaPods has already done in the past?
• How does this translate to other platforms (Windows, Mac)?


[fkgozali]

Thanks for writing this up! I don't have strong opinions on the CLI mechanics itself, but I do agree that capturing information about projects/libs (however it is done) + dependencies is useful, especially related to using react-native-codegen in the new architecture. I wrote info about that in #273, especially the section "Annotation of such libraries/dependencies":

With such annotations, the infra should allow crawling dependencies that use the plugin, and have specific annotations

This is also very similar to how Buck works.

In general though, our principle has been to utilize the mechanism that's as close as possible to what the expectation of tools for the platform is. E.g. in my proposal, I discussed annotating this in build.gradle (or somewhere in Gradle), because that's a more familiar tooling for Android projects.

[pepicrft]

👋 I was wondering if I should open this issue on the cli repository instead. What do you think @grabbou ? I'd appreciate any advice on how to get more people's thoughts on this idea, decide if it aligns with the direction of the project, and therefore start taking steps towards it.

[TheSavior]

I think this is interesting, and I love that Shopify is interested in improving the developer experience!

A concern I have is that this would make things more friendly for JS engineers but more hostile to engineers with a native background. I worry they won't recognize the projects, and React Native will feel even more like a black box, making those engineers even more resistant to it. In recent history, the React Native team has been kinda pushing the other direction to make things more familiar to native engineers (but maybe at the cost of our current user base).

One of the other benefits of our current approach of leaning on the native experience has been that when issues come up, you can go look at the Xcode or Android documentation.

If we build something custom that would greatly increase the surface area of React Native (even though it wouldn't be maintained by us at FB). There would probably end up needing to be a team that owned this to make it successful and keep it well documented.

Also, what you are describing sounds very similar to Expo managed projects. What do you see as the difference between this approach and the space that Expo is supporting?

[pepicrft]

A concern I have is that this would make things more friendly for JS engineers but more hostile to engineers with a native background. I worry they won't recognize the projects, and React Native will feel even more like a black box, making those engineers even more resistant to it. In recent history, the React Native team has been kinda pushing the other direction to make things more familiar to native engineers (but maybe at the cost of our current user base).

Interesting, what do you think this would make the tool hostile for engineers with a native background? The tool would generate a standard project that you can edit as you'd edit the project that is currently part of the repository. The workflows that I was thinking about are:

• I'm a developer and want to work on the app (Javascript side): I run react-native run ios and I get my app up and running with the Metro server.
• I'm a developer that wants to edit the native code/project: I run react-native edit ios and I get the Xcode project opened and ready to interact with it. I can build it, run native tests...

One of the other benefits of our current approach of leaning on the native experience has been that when issues come up, you can go look at the Xcode or Android documentation.

Right, but in our experience, those projects quickly become a source of complexities and inconsistencies that inevitably lead to compilation issues. I think this is the result of:

• Changes in projects usually disregarded when reviewing PRs.
• Package integration steps followed without really understanding what they are for.
• Solving compilation issues by applying random responses found on Stackoverflow.

This is also solvable by adding more linting on our side (e.g. running a git hook or a pipeline on CI), or ensuring that developers have a good understanding of the native pieces before they introduce changes, but we think the developer experience will be better if we minimize the native surface developers are exposed to and expose as much internals as needed.

If we build something custom that would greatly increase the surface area of React Native (even though it wouldn't be maintained by us at FB). There would probably end up needing to be a team that owned this to make it successful and keep it well documented.

Would that surface increase if something like this was built into the React Native CLI?

Also, what you are describing sounds very similar to Expo managed projects. What do you see as the difference between this approach and the space that Expo is supporting?

Unlike Expo that has the concept of "ejecting", which takes you from we own the native project to you are the owner of the native projects, the native projects are generated when needed. Pretty much like how CocoaPods does with dependencies with native code when we need to build the native project, we need CocoaPods to generate an Xcode project that contains the build instructions for xcodebuild to be able to link the code of that dependency. Take that approach and expand it to your own project too.

[asklar]

<speaking from a React Native Windows + Windows perspective>

If I understand the proposal, this is describing a meta-project; you write the project spec and then some tool will generate the appropriate platform-specific / build system-specific projects/solutions. For example, on Windows this would generate the right vcxproj/csproj/sln files.

I think that approach would work fine for someone who is completely oblivious to the native code, someone coming from a pure JS background and just wants to use "good defaults" (it's not clear that there is a one-size-fits-all).

Adding the logic to the RN CLI would only take care of the in-tree platforms, but would leave out-of-tree platforms like Windows and macOS, to implement and maintain the logic to generate projects themselves.

On Windows we have apps that are going to be a mix of React Native and other technologies (XAML, etc.). The way this works today is when you create a react native windows app, a solution is generated (which describes a collection of projects), and a language specific project is also generated (C++ or C# depending on what the user chooses). From that point onwards, RNW is out of the picture for the most part when it comes to project management. We do perform some tasks for autolinking etc. but those are heuristic based since the projects can be manually edited/changed to best suit the developer's needs.

I'm not sure what all react-native edit ios does, does it just open the project file in an editor?

</>

[TheSavior]

The tool would generate a standard project that you can edit as you'd edit the project that is currently part of the repository.

I would assume that the project it generates wouldn't really be editable, or at least it would get blown away the next time it was generated. So you'd have to figure out how to modify the React Native specific upstream configuration files in the way you want to be able to generate the downstream native project. Is that right?

FWIW, that's how we operate at Facebook. We define everything in buck, and buck project generates local Xcode and Android studio projects that we use for debugging and working in the native editor. https://buck.build/command/project.html

If we build something custom that would greatly increase the surface area of React Native (even though it wouldn't be maintained by us at FB). There would probably end up needing to be a team that owned this to make it successful and keep it well documented.

Would that surface increase if something like this was built into the React Native CLI?

It would, wouldn't it? I would expect it would increase the surface area of the CLI and the burden on Callstack to maintain it.

Are there any roadblocks to prototyping / building this out? If Shopify is interested in building this, would it be reasonable to build it as a separate project initialization approach and if it works out and the community prefers it, then in the future we could look at switching the default?

Are there pieces you think are currently in the initialization path that make it harder for something like this to be built separately?

[ravirajn22]

@pepibumur idea is pretty cool, but my concern is can a manifest file satisfy all use cases provided by Xcode and gradle like Build Settings, Build Phases, custom gradle tasks etc. What if someone needs to do something not supported by the manifest file? I think the user then have to edit the generated xcode files and commit the same. With so much options provided by xcode and android build system managing everything under a single manifest file will be a nightmare or impossible I think.

I went through tuist, looks like it has some learning curve which will be only be familiar to someone from iOS background, for pure JS developers looks like the learning curve will be high.

[AndreiCalazans]

The current way of doing things is certainly easier to maintain for the CLI and RN Core. But, not better for the developer experience. It keeps the burden of dealing with the native configurations on the client side.

At You.i TV, they have a similar approach to what @pepibumur is proposing and what Facebook does internally with Buck. They use Cmake to generate "builds" that are essentially project outputs for each platform. They are project scaffolds that are generated at built time and do all the linking with its native code dependencies.

Besides the benefits that Pedro mentions, it gives the product more flexibility to change the internal implementation details of the scaffold project. The big drawback is for those who want to deviate too much from the generated project -- this becomes logic they maintain on the client-side, this problem already exists for React Native projects.

To support changes to the outputted project, this is where Cmake shines, they allow everything to be overwritten on the client side. You can decide which modules to link, which AppDelegate to use, or where to get the app assets from.

A few concerns come to mind though:

1. This is a lot of work to implement, technically we would need a Build Solutions Team and figure out how to manage out of tree platforms.

2. A problem we faced with the You.i TV developer experience is by using a build tool like CMake you introduce yet another thing one must learn to use your product. While a manifest file can mitigate this issue, it will unlikely cover all the possibilities that we can achieve with the build tool itself.

Despite the above points, I am very inclined towards this idea and if the community finds the resources to put into this it is likely to yield great improvements to the developer experience and longevity of the project.

One thing I desire as a React Native user external to Facebook is for the public repo to match how Facebook uses React Native internally, thus should we lean towards doing something similar to what Facebook does internally with Buck?

Are there any roadblocks to prototyping / building this out? If Shopify is interested in building this, would it be reasonable to build it as a separate project initialization approach and if it works out and the community prefers it, then in the future we could look at switching the default?

This does make sense too. As a community, this can be built outside of the RN Core to test out the idea. However, I think it is very important for the specifications of the project to have an agreement between all the RN partners for it not to be a wasted time.

[pepicrft]

Limiting the configuration of the generated projects

This is a very valid concern. However, I think we can design the project abstraction to provide defaults with the goal of minimizing the configuration surface without compromising the configuration of any internal settings. For example, most of the time you won't care about Xcode build settings, but in certain scenarios, you might need to set some compilation flags. In that case, you'll have an API to say, for my app please set this build setting. In the case of Android, we could achieve something similar by providing a Gradle plugin with some defaults.

In more complex scenarios like Microsoft's where React Native has to be integrated into existing native codebases, the CLI could generate a linkable project. I'm not very familiar with Windows and Android, but on iOS that can be achieved with an Xcode project that gets added to a workspace where other projects live. That's in essence what CocoaPods does, so we'd generate that project for you and you take the final steps of integrating it.

Prototyping & maintenace

We can build a prototype at Shopify to have something more tangible that we can play with. Once we have it, we can go through common scenarios that remain cumbersome. For example, upgrading the version of React Native. I brought this up in part because I don't want to confuse people and create fragmentation in the community. If you think it's fine doing this prototyping will most likely go ahead with it and use this issue to share updates on the progress we are making.

---

@pepibumur idea is pretty cool, but my concern is can a manifest file satisfy all use cases provided by Xcode and gradle like Build Settings, Build Phases, custom gradle tasks etc. What if someone needs to do something not supported by the manifest file? I think the user then have to edit the generated xcode files and commit the same. With so much options provided by xcode and android build system managing everything under a single manifest file will be a nightmare or impossible I think.

I touched on this in the first section, but I think as needs emerge, we can provide APIs for customizing more project components. That's what we've done with Tuist and it's worked fine. Initially, the API was very limiting but simple. Over time and thanks to the feedback that we got from users, we realized that we had to extend the API adding features that they needed for their projects. I think in this process it's key to optimize for zero-config by providing defaults. Most of the users doing React Native development don't care about many native details.

@pepibumur idea is pretty cool, but my concern is can a manifest file satisfy all use cases provided by Xcode and gradle like Build Settings, Build Phases, custom gradle tasks etc. What if someone needs to do something not supported by the manifest file? I think the user then have to edit the generated xcode files and commit the same. With so much options provided by xcode and android build system managing everything under a single manifest file will be a nightmare or impossible I think.

I went through tuist, looks like it has some learning curve which will be only be familiar to someone from iOS background, for pure JS developers looks like the learning curve will be high.

Bear in mind that Tuist was designed for developers that are familiar with Xcode and Apple's platforms. Therefore, many terms and concepts map easily to Tuist's domain. However, if we design a new tool, we'd use similar project-generation principles, but with an interface that is very friendly for both types of developers: JS and Native developers.

---

This is a lot of work to implement, technically we would need a Build Solutions Team and figure out how to manage out of tree platforms.

Fully agree with this. The amount of work is huge, but if the prototype proves that the experience improves significantly, we (Shopify) can commit to build and maintain the tool. As I mentioned in the body of the issue, despite we love the technology, we have some inconveniences due to the principles around which the tooling has been designed. We've workarounds but they look more like a patch than a proper solution.

A problem we faced with the You.i TV developer experience is by using a build tool like CMake you introduce yet another thing one must learn to use your product. While a manifest file can mitigate this issue, it will unlikely cover all the possibilities that we can achieve with the build tool itself.

Yeah, that's not ideal. It'd be the same if we introduced Buck. I think the tool should be built upon the existing build systems and tools (Gradle, xcodebuild, Xcode projects). Those elements are means to compile the app and the native code of the dependencies. Developers always have to go through that process and therefore I think needs to be frictionless. If we remove the friction from that phase React Native will shine even more than what it currently does.

One thing I desire as a React Native user external to Facebook is for the public repo to match how Facebook uses React Native internally, thus should we lean towards doing something similar to what Facebook does internally with Buck?

Facebook's setup is most likely designed for the scale needs and thus can't align with the industry in that sense. We have a similar situation at Shopify where we need to build and adjust our workflows as we grow. Despite we'd like to be aligned with the industry, but the tools and processes that work for the industry don't work for us anymore.

This does make sense too. As a community, this can be built outside of the RN Core to test out the idea. However, I think it is very important for the specifications of the project to have an agreement between all the RN partners for it not to be a wasted time.

Agree. As I mentioned above, I'll build a prototype to continue this discussion.