This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
Add project hash information to nx print-affected output
#3292
Comments
|
This issue has been automatically marked as stale because it hasn't had any recent activity. It will be closed in 14 days if no further activity occurs. |
|
up |
|
I would love to have this to improve our CI pipeline as well - would save a lot of homegrown hacks |
|
There is a bit of a workaround for this which is to run {
"tasks": [
{
"id": "lib-name:build",
"overrides": {},
"target": {
"project": "lib-name",
"target": "build"
},
"command": "yarn nx build lib-name",
"outputs": [
"dist/libs/lib-name"
],
"hash": "3128ba521cbe508342017bff3a2429d3bfc00cb6d17269b2078209bd17992108"
},
...etcI use this in combination with other |
|
Ahh awesome, thank you so much for this - this will certainly do for now! |
|
Unfortunately this workaround was removed in 67e78dc#diff-715d57e669c659f3ed1b596f901efab8bb79d61e16149eb088768110af4eed68. @vsavkin is there a different workaround that we might be able to use instead? |
|
This is sad news indeed, I put a lot of work into rewriting our CI to use this workaround. @vsavkin are you able to shed some light on this? Is there another workaround or an official implementation being worked on? |
|
I spent some time during a hackday to see if I could get something in our case to get back the hash on a per app/per lib case. I did dug into the diff mentioned by @zakhenry, I also looked into "NX plugins and devkit", "NX plugins", "Creating Custom Executors", the Plugins and custom executors won't be able to add something like/modify the I also tried a dummy approach of using NX through a node script like this: but unless I copy the whole I'm also confused by the commit that @zakhenry mentioned. This commit is called "feat(core): add support for custom hashers" so I'd expect the ability to customize the hashing strategy from a consumer point of view but maybe that's not it. Anyway from here I've got 2 things to highlight:
Thanks for any help on this 🙏 We're now stuck to nx 12.2.0 till we solve it as our CI pipeline relies on the hashes 🙏 |
|
Any idea on this? Any guidance on this? I get 1 day per month to do what I want (hackday). As I said, I'm motivated to spend an entire day on this tomorrow if needed. If someone from NX could just help me out a little and give me a direction that'd be much appreciated 🙏 |
|
If there isn't any plans to support exposing the hash again, which would be super sad, then maybe it is enough to use the Hasher in a custom script by importing it from
e.g. something like import { Hasher } from '@nrwl/workspace/src/core/hasher/hasher';
const hasher = new Hasher(projectGraph, nxConfig, {});
const hash = await hasher.hashTaskWithDepsAndContext(task); |
|
This issue has been automatically marked as stale because it hasn't had any recent activity. It will be closed in 14 days if no further activity occurs. |
|
Would still be nice to have. So bump for the stale label :) |
|
Thanks for the infos provided by everybody. If I'm not mistaken, this should show the hash for all projects, because compared to the first commit ever, they're all affected. |
|
@Rijak0 which version of nx are you using? I think this issue slightly diverted from "It would be nice nice to have the hash for all projects" to "It'd be nice to have any hash at all" as they got removed completely from the output. This issue has been going on for a while so I may have lost track a bit and have my thoughts confused but at least I think that's the case. So I don't understand how you'd get a result with a hash in your first command? I tried it in our project and here's one object from the returned array: and there's no hash in here |
|
Howdy @maxime1992! "Works on my machine" :D prints out Interesting that it doesn't work for you. |
|
@maxime1992 |
|
Wuuuuuuuuuuut that's fantastic 🤩! It doesn't fix the original issue here but it sure helps a bit at least! And with your workaround it might be something to get started with for sure. Thanks a lot! |
|
Glad to give something back, perfect! 🎉 |
Unfortunately in v15 the hash is no longer present in this output. |
|
Still working for me with v 15.2.1:
|
|
Bumping this up. I need to know each hash to remove old caches and keep caches folder as small as possible. PS: Although the solution above works, I've found a slightly better way of getting hashes. All such solutions are based on the same trick: we need Say you have defined your named inputs like this: Now if you say your global file changed, it will think that everything is affected and provide you with all the current hashes: Output:
|
|
Seems like with nx v16 some logic to the hashing changed. The hash now looks quite different, e.g. Also looks like This is kind of a bummer, the hash really enabled some good automations and cicd options. I'll do some more digging and I'll try to find out what changed in between v15/16. |
|
This should really be part of a public and stable API IMO. It's been working on and off too often. |
|
+1… that is a core functionality, our whole CI/CD is also depending on it.
We‘re calculating affected differences with external systems like Docker,
Nuget, npm, for the build and deployment.
Also e.g. finding out which projects need to be re-tested…
…On Fri 1. Sep 2023 at 14:00, Maxime ***@***.***> wrote:
This should really be part of a public and stable API IMO. It's been
working on and off too often.
If any maintainer read this, could you please consider having the hash
somehow part of a stable API for the CLI please?
—
Reply to this email directly, view it on GitHub
<#3292 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABZMCMG5ZLEEE2V6VELIZC3XYHE55ANCNFSM4OVQDV7A>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Exactly.
We do the same. We take advantage of the fact that Gitlab is capable of having a first stage to generate a gitlab ci yaml file that is then run as the pipeline (aka dynamic pipelines on top of my head). So we have our first pipeline running as a deno script and checking the result of print-affected to analyse what's changed, and from there we generate all the pipeline (lint, test, e2e tests, docker builds, deployment etc) dynamically. While the hashes are not necessary to do that, they are for further optimisations. For example, it'd be nice to save the docker build of projects by their hash. This way we could query in our first pipeline the registry to see if a tag already exist for a given hash, if so, completely skip a build because it's been done and whatever changed since, didn't affect the files for that project. |
|
@vsavkin @AgentEnder do you have any opinions on this? Some kind of stable api or cli to consistently get the hashes would be extremely helpful. Even adding it to the graph commands would be great (when outputting to json or similar) would work. Would gladly help with implementing this, if you have a vision for what the implementation should approx. look like 🙏 |
|
A little hacky, but seems to be working: returns for a new playground nx workspace: |
For: +1 to this being provided as standard output from one of the affected/graph commands. |
|
Stuck on v17.1 for now. @vsavkin Hopefully this can get some attention soon to unblock us - and I suspect many others. |
|
@dsschneidermann I am still on v17.1 due to other issues preventing me from an upgrade, but I adapted your example today to using the NX Deamon APIs, maybe this might work more stable with v17.2 and solves the issues you are seeing? import { createProjectGraphAsync, ProjectGraph } from '@nx/devkit';
import { DaemonClient } from 'nx/src/daemon/client/client';
import { DaemonBasedTaskHasher } from 'nx/src/hasher/task-hasher';
import { createTaskGraph, mapTargetDefaultsToDependencies } from 'nx/src/tasks-runner/create-task-graph';
import { TargetDependencies, readNxJson } from 'nx/src/config/nx-json';
const uniq = <I,>(arr: I[]): I[] => [...new Set(arr))] as I[];
const resolveTargetHashes = async ({
target,
graph,
dependencies,
hasher,
}: {
target: string;
graph: ProjectGraph;
dependencies: TargetDependencies;
hasher: DaemonBasedTaskHasher;
}) => {
const projectNames = Object.values(graph.nodes)
.filter(({ data }) => !!data.targets?.[target])
.map((p) => p.name);
const taskGraph = createTaskGraph(graph, dependencies, projectNames, [target], undefined, {}, false);
const tasks = Object.values(taskGraph.tasks);
const hashes = await hasher.hashTasks(tasks, taskGraph, process.env);
return hashes.map((res) => res.value);
};
async function exec() {
const cwd = process.cwd();
const nx = readNxJson(cwd);
const graph = await createProjectGraphAsync({ exitOnError: true, resetDaemonClient: true });
const daemon = new DaemonClient(nx);
const hasher = new DaemonBasedTaskHasher(daemon, {});
const dependencies = mapTargetDefaultsToDependencies(nx.targetDefaults);
// Adapt the following line to your own likings, I automatically resolve all targets with caching enabled based on the targetDefaults.
const targets = Object.entries(nx.targetDefaults ?? {})
.filter(([target, config]) => config.cache === true)
.map(([target]) => target);
const hashes = await Promise.all(
targets.map((target) => resolveTargetHashes({ target, graph, dependencies, hasher })),
).then((results) => uniq(results.flat()));
console.log(hashes);
}
exec()
.then(() => {
process.exit(0);
})
.catch((err) => {
console.error(err);
process.exit(1);
}); |
|
Yes, functionality like this is truly needed. I hope someone on the team becomes aware of that. Ideally, we should be able to obtain the hash via a simple CLI command like:
This way, nx would be much more powerful. Especially in the context of AWS, using hashes is crucial, as there isn't any other good indicator for checking whether or not a resource needs to be updated AND having some kind of indiciator about the deployments current build. Potential time savings could be enormous. For now, we have to use simple commit hashes, sadly. From an infrastructure/deployment standpoint, this is more or less THE potential killer feature of why people would choose NX instead of Bazel or other solutions. However, as of now, NX simply doesn't provide this core functionality to fully utilize its incremental build system. I would greatly appreciate it if the team could address this issue. |
|
I've updated the code of @Inkdpixels and made a little CLI tool out of it (also works with 17.2.8), which allows you to specifically filter by project and target. Output is the requested hash value. import { createProjectGraphAsync, ProjectGraph } from "@nx/devkit"
import { DaemonClient } from "nx/src/daemon/client/client"
import { DaemonBasedTaskHasher } from "nx/src/hasher/task-hasher"
import { createTaskGraph, mapTargetDefaultsToDependencies } from "nx/src/tasks-runner/create-task-graph"
import { TargetDependencies, readNxJson } from "nx/src/config/nx-json"
import * as yargs from "yargs"
const argv = yargs
.option("project", {
alias: "p",
description: "Filter result by project.",
type: "string",
})
.option("target", {
alias: "t",
description: "Filter result by target.",
type: "string",
})
.help()
.alias("help", "h")
.parseSync()
const resolveTargetHashes = async ({
target,
graph,
dependencies,
hasher,
}: {
target: string
graph: ProjectGraph
dependencies: TargetDependencies
hasher: DaemonBasedTaskHasher
}) => {
const projectNames = Object.values(graph.nodes)
.filter(({ data }) => !!data.targets?.[target])
.map((p) => p.name)
const taskGraph = createTaskGraph(graph, dependencies, projectNames, [target], undefined, {}, false)
const tasks = Object.values(taskGraph.tasks)
const result = await Promise.all(
tasks.map((task) =>
hasher.hashTask(task, taskGraph, process.env).then((hash) => ({
id: task.id,
target: task.target,
hash: hash.value,
})),
),
)
return result
}
async function exec() {
const cwd = process.cwd()
const nx = readNxJson(cwd)
const graph = await createProjectGraphAsync({ exitOnError: true, resetDaemonClient: true })
const daemon = new DaemonClient(nx)
const hasher = new DaemonBasedTaskHasher(daemon, {})
const dependencies = mapTargetDefaultsToDependencies(nx.targetDefaults)
const targets = Object.entries(nx.targetDefaults ?? {})
.filter(([target]) => target === argv.target)
.map(([target]) => target)
if (!targets.length) {
console.error(`Project '${argv.project}' with target '${argv.target}' not found!`)
process.exit(1)
}
const targetsWithHashes = await Promise.all(targets.map((target) => resolveTargetHashes({ target, graph, dependencies, hasher }))).then((results) => results.flat())
const filteredTarget = targetsWithHashes.filter((target) => target.target.project === argv.project)
if (!filteredTarget.length) {
console.error(`Project '${argv.project}' with target '${argv.target}' not found!`)
process.exit(1)
}
if (!filteredTarget[0].hash) {
console.error(`No hash for target ${argv.target} of project ${argv.project} found`)
process.exit(1)
}
console.log(filteredTarget[0].hash)
}
exec()
.then(() => {
process.exit(0)
})
.catch((err) => {
console.error(err)
process.exit(1)
})Usage:
will result in:
|
|
Hey all, Nx cannot generate the correct hash for every task without actually running the command. The tasks are hashed right before they executed in the middle of overall task execution. This is because tasks can depend on the outputs of tasks they depend on via https://nx.dev/reference/inputs#outputs-of-dependent-tasks. The same is true for custom hashers because Nx does not know what information custom hashers are hashing.. so it could be information which changes while other tasks execute. We'll need to discuss more if this limitation is a deal breaker for this. Nx could provide the hashes of only tasks which it knows for sure should be accurate. But it wouldn't be a definitive list of hashes which Nx will run. There are cases where none of the tasks in the command can actually be hashed upfront. So I wouldn't build anything on top of it such as spawning CI agents. I think this discussion is better held at a higher level. What is the desired outcome of having the hashes? If we could discuss that, we could see if this information would suffice or if there is a better way to go about it. |
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
Description
I'd like to see a set of fields added to the print affected json output, something like
{ "projects": [...], "projectGraph": {...}, "hashes": { "my-app": { "current": "4b669d95627bf6724502d7b4ff6f41d3654c50ba9e3b1cf31e72c138370423f8", "previous": "0bdb20fbd1788957302f4cf356ff0449ee3c2d10960814f6c34fd399fa64f5d3" } } }This hash key should include all projects, not just the affected ones.
Motivation
I use nx print-affected in order to dynamically generate a CI pipeline which works amazingly well, however I have a need to determine the current hash of an unchanged project so that I can fetch the docker container
Suggested Implementaion
I assume that the affected projects is determined by some hashing already, so it must be feasible to add this into the affected output
Alternate Implementations
This could be a completely different command line tool if needed.
The text was updated successfully, but these errors were encountered: