Skip to content

[Bug?]: yarn patch generated unparsable patch: URL #4711

Open
@lachlanhunt

Description

@lachlanhunt

Self-service

  • I'd be willing to implement a fix

Describe the bug

My company uses an internal package registry where all npm packages are fetched from. This is configured in ~/.yarnrc.yaml (company name replaced with example)

npmRegistries:
  "https://packages.example.com/artifactory/api/npm/npm-remote":
    npmAlwaysAuth: false
npmScopes:
  example:
    npmRegistryServer: "https://packages.example.com/artifactory/api/npm/example-npm"
    npmPublishRegistry: "https://packages.example.com/artifactory/api/npm/example-npm"

The result of this is that the resolution for the package in yarn.lock looks like this:

  resolution: "@example/eslint-plugin-example@npm:4.0.0::__archiveUrl=https%3A%2F%2Fpackages.example.com%2Fapi%2Fnpm%2Fexample-npm%2F%40example%2Feslint-plugin-example%2F-%2F%40example%2Feslint-plugin-example-4.0.0.tgz"

When I tried to patch this packing using:

$ yarn patch @example/eslint-plugin-example
# make edits
$ yarn patch-commit -s /private/var/folders/path/to/code/user
...

This generated a patch URL in package.json that looked like this:

```json
{
  ...
  "resolutions": {
    "@example/eslint-plugin-example@4.0.0": "patch:@example/eslint-plugin-example@npm:4.0.0::__archiveUrl=https%3A%2F%2Fpackages.example.com%2Fapi%2Fnpm%2Fexample-npm%2F%40example%2Feslint-plugin-example%2F-%2F%40example%2Feslint-plugin-example-4.0.0.tgz#.yarn/patches/@example-eslint-plugin-example-npm-4.0.0-bab1aa1587.patch"
  }

}

(Then I fixed the version range in that resolution as described in #4231 (comment) )

Finally, running yarn to apply the patch gives this error:

$ yarn
➤ YN0000: ┌ Resolution step
➤ YN0001: │ Error: Patch locators must explicitly define their source
    at xue (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:733:658)
    at dE (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:733:978)
    at iO.bindDescriptor (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:736:2747)
    at wd.bindDescriptor (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:391:1082)
    at wd.bindDescriptor (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:391:1082)
    at reduceDependency (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:390:29055)
    at tA.reduceHook (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:392:3584)
    at ee (/Users/lhunt/.node/corepack/yarn/3.2.2/yarn.js:439:6759)
➤ YN0000: └ Completed
➤ YN0000: Failed with errors in 0s 23ms

I was able to work around this issue by modifying the patch: URL like this:

{
  ...
  "resolutions": {
    "@example/eslint-plugin-example@^4.0.0": "patch:@example/eslint-plugin-example@npm:4.0.0#.yarn/patches/@example-eslint-plugin-example-npm-4.0.0-bab1aa1587.patch"
  }
}

It appears the presence of ::__archiveUrl=https... breaks the parsing of the patch URL in yarn.

To reproduce

(as explained above)

edit: see reproduction in comment below #4711 (comment)

Environment

System:
  OS: macOS 12.5
  CPU: (10) arm64 Apple M1 Pro
Binaries:
  Node: 16.16.0 - /private/var/folders/78/vv5cyf6570xcqbsy0r2nfyfw0000gn/T/xfs-99d9804c/node
  Yarn: 3.2.2 - /private/var/folders/78/vv5cyf6570xcqbsy0r2nfyfw0000gn/T/xfs-99d9804c/yarn
  npm: 8.11.0 - ~/Library/Caches/fnm_multishells/21303_1659328957074/bin/npm

Additional context

No response

Activity

xenoterracide

xenoterracide commented on Aug 5, 2022

@xenoterracide

hmm... I wonder if this is the same as the problem I just came here to report

xenoterracide

xenoterracide commented on Aug 5, 2022

@xenoterracide

I don't get an error though, mine just silently doesn't apply

xenoterracide

xenoterracide commented on Aug 5, 2022

@xenoterracide

linked my bug, seems very related to both that you mentioned, but mine seems to be both of the problems at once...

wojtekmaj

wojtekmaj commented on Aug 5, 2022

@wojtekmaj
Contributor

I was able to work around this issue by modifying the patch: URL like this:

OMG, I want to kiss you right now. I struggled with this for MONTHS, manually applying the patches.

yarnbot

yarnbot commented on Sep 4, 2022

@yarnbot
Collaborator

Hi! 👋

This issue looks stale, and doesn't feature the reproducible label - which implies that you didn't provide a working reproduction using Sherlock. As a result, it'll be closed in a few days unless a maintainer explicitly vouches for it or you edit your first post to include a formal reproduction (you can use the playground for that).

Note that we require Sherlock reproductions for long-lived issues (rather than standalone git repositories or similar) because we're a small team. Sherlock gives us the ability to check which bugs are still affecting the master branch at any given point, and decreases the amount of code we need to run on our own machines (thus leading to faster bug resolutions). It helps us help you! 😃

If you absolutely cannot reproduce a bug on Sherlock (for example because it's a Windows-only issue), a maintainer will have to manually add the upholded label. Thanks for helping us triaging our repository! 🌟

added
staleIssues that didn't get attention
on Sep 4, 2022
lachlanhunt

lachlanhunt commented on Sep 7, 2022

@lachlanhunt
Author

I couldn't figure out how exactly to reproduce this issue using Sherlock. Last time I tried to use it, the page failed to load the editor properly, so I gave up and assumed the tool was broken.

I explained the steps to reproduce in the description. I'll need a maintainer to help out.

wojtekmaj

wojtekmaj commented on Sep 7, 2022

@wojtekmaj
Contributor

This is my completely blind attempt at repro, as playground is not working, and local Sherlock does not expose (undocumented) methods like yarn or packageJsonAndInstall:

const installPromise = packageJsonAndInstall({
  dependencies: {
    [`@codertocat/hello-world-npm`]: `latest`,
  },
});

await expect(installPromise).resolves.toBeTruthy();

// Use node_modules resolver
// await yarn(`config`, `set nodeLinker node-modules`);

// Set npmRegistryServer
await yarn(`config`, `set npmRegistryServer "https://npm.pkg.github.com"`);

// Extract the package for patching
const patchOutput = await yarn(`patch`, `@codertocat/hello-world-npm`);
const patchOutputLines = patchOutput.split("\n");

// Look for "You can now edit the following folder: …"
const regex1 = /You can now edit the following folder: (.*)/;
const line1 = patchOutputLines.find((line) => line.match(regex1));
const folder = line1.match(regex1)[1];

// Make the edit in extracted package README.md file
const filePath = path.join(path.resolve(folder), `index.js`);
const file = await fs.readFile(filePath, `utf8`);
await fs.writeFile(
  filePath,
  file.replace("'Hello world!'", "'Hello world from patch!'")
);

// Look for "Once you are done run … and Yarn will store a patchfile based on your changes."
const regex2 =
  /Once you are done run (.*) and Yarn will store a patchfile based on your changes./;
const line2 = patchOutputLines.find((line) => line.match(regex2));
const command = line2.match(regex2)[1];
const [yarnText, commandText, ...args] = command.split(" ");

// Run the command
await yarn(commandText, ...args);

// Install repo again to apply patches
const install2Promise = await yarn(`install`);

/**
 * BUG #1:
 * Error: @codertocat/hello-world-npm@npm:1.0.2::__archiveUrl=https%3A%2F%2Fnpm.pkg.github.com%2Fdownload%2F%40Codertocat%2Fhello-world-npm%2F1.0.2%2F9384a68bd287b859f028683bcaacedd27803f1dd isn't supported by any available resolver
 **/
expect(install2Promise).resolves.toBeTruthy();

// See if the patch was applied
const installedModule = require("@codertocat/hello-world-npm");

const output = installedModule.helloWorld();

/**
 * BUG #2: Patch not applied?
 */
expect(output).toEqual("Hello world from patch!");
arcanis

arcanis commented on Sep 7, 2022

@arcanis
Member

Mind trying the 4.0.0-rc (yarn set version canary)? Patches have been significantly improved there, and I'm unlikely to backport those changes to 3.x as some of them are backward-incompatible, and others just difficult to safely extract.

wojtekmaj

wojtekmaj commented on Sep 8, 2022

@wojtekmaj
Contributor

@arcanis I did the tests on 2 dimensions:

  • Yarn version: latest (3.2.3), canary (4.0.0-rc.18)
  • Dependency version range: exact ("@codertocat/hello-world-npm": "1.0.2") and caret ("@codertocat/hello-world-npm": "^1.0.0")

Here are the commands I ran:

# 4.0.0 ONLY:
yarn set version canary

yarn config set nodeLinker node-modules
yarn config set npmRegistryServer "https://npm.pkg.github.com"

# CARET ONLY:
yarn add @codertocat/hello-world-npm@^1.0.0
# EXACT ONLY:
yarn add @codertocat/hello-world-npm@1.0.2

yarn patch @codertocat/hello-world-npm
# (editing the file now by changing "Hello world!" to "Hello world from patch!" & saving it)
yarn patch-commit -s # (dynamically created folder path here)

yarn

All 4 resulted in the exact same crash.

user@WS-Q6JW6PN4MP 3.2.3-caret % yarn
➤ YN0000: ┌ Resolution step
➤ YN0001: │ Error: @codertocat/hello-world-npm@npm:1.0.2::__archiveUrl=https%3A%2F%2Fnpm.pkg.github.com%2Fdownload%2F%40Codertocat%2Fhello-world-npm%2F1.0.2%2F9384a68bd287b859f028683bcaacedd27803f1dd isn't supported by any available resolver
    at Bd.getResolverByDescriptor (/Users/user/.node/corepack/yarn/3.2.3/yarn.js:391:1664)
    at Bd.getResolutionDependencies (/Users/user/.node/corepack/yarn/3.2.3/yarn.js:391:1147)
    at TR.getResolutionDependencies (/Users/user/.node/corepack/yarn/3.2.3/yarn.js:435:4101)
    at Bd.getResolutionDependencies (/Users/user/.node/corepack/yarn/3.2.3/yarn.js:391:1176)
    at ce (/Users/user/.node/corepack/yarn/3.2.3/yarn.js:439:7562)
    at /Users/user/.node/corepack/yarn/3.2.3/yarn.js:439:8135
    at async /Users/user/.node/corepack/yarn/3.2.3/yarn.js:439:7666
➤ YN0000: └ Completed
➤ YN0000: Failed with errors in 0s 19ms
user@WS-Q6JW6PN4MP 4.0.0-rc-caret % yarn
➤ YN0000: ┌ Resolution step
➤ YN0001: │ Error: @codertocat/hello-world-npm@npm:1.0.2::__archiveUrl=https%3A%2F%2Fnpm.pkg.github.com%2Fdownload%2F%40Codertocat%2Fhello-world-npm%2F1.0.2%2F9384a68bd287b859f028683bcaacedd27803f1dd isn't supported by any available resolver
    at ed.getResolverByDescriptor (/Users/user/.node/corepack/yarn/4.0.0-rc.18/yarn.js:135:1698)
    at ed.getResolutionDependencies (/Users/user/.node/corepack/yarn/4.0.0-rc.18/yarn.js:135:1177)
    at Fx.getResolutionDependencies (/Users/user/.node/corepack/yarn/4.0.0-rc.18/yarn.js:190:11001)
    at ed.getResolutionDependencies (/Users/user/.node/corepack/yarn/4.0.0-rc.18/yarn.js:135:1206)
    at Ce (/Users/user/.node/corepack/yarn/4.0.0-rc.18/yarn.js:195:7688)
    at /Users/user/.node/corepack/yarn/4.0.0-rc.18/yarn.js:195:8489
    at async /Users/user/.node/corepack/yarn/4.0.0-rc.18/yarn.js:195:7826
➤ YN0000: └ Completed
➤ YN0000: Failed with errors in 0s 19ms

Here are all 4 minirepos I created to reproduce this issue:

Yarn patch.zip

In all 4 cases, removing %3A%3A__archiveUrl=https%253A%252F%252Fnpm.pkg.github.com%252Fdownload%252F%2540Codertocat%252Fhello-world-npm%252F1.0.2%252F9384a68bd287b859f028683bcaacedd27803f1dd bit from resolution caused installation to proceed just fine & applied patch just fine.

I have a suspicion that releasing 1.0.3 version of test package would break patch on 3.2.3 as resolution created by patch-commit would no longer be applicable. But that's another story.

sargunv

sargunv commented on Feb 1, 2023

@sargunv
Contributor

I found this article quite helpful to understand the bug and how to work around it: https://charles-stover.medium.com/patching-packages-in-yarn-berry-72e4ded29a56

10 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingstaleIssues that didn't get attention

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Participants

      @xenoterracide@pniederlag@quisido@lachlanhunt@arcanis

      Issue actions

        [Bug?]: yarn patch generated unparsable patch: URL · Issue #4711 · yarnpkg/berry