-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs(core): enterprise custom dte (nrwl#22143)
- Loading branch information
1 parent
acf4787
commit 62e89f6
Showing
73 changed files
with
1,662 additions
and
384 deletions.
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| # Custom Distributed Task Execution on Azure Pipelines | ||
|
|
||
| Using [Nx Agents](/ci/features/distribute-task-execution) is the easiest way to distribute task execution, but it your organization may not be able to use hosted Nx Agents. With an [enterprise license](https://nx.app/enterprise), you can set up distributed task execution on your own CI provider using the recipe below. | ||
|
|
||
| ## Run Custom Agents on Azure Pipelines | ||
|
|
||
| Run agents directly on Azure Pipelines with the workflow below: | ||
|
|
||
| ```yaml {% fileName="azure-pipelines.yml" %} | ||
| trigger: | ||
| - main | ||
| pr: | ||
| - main | ||
|
|
||
| variables: | ||
| CI: 'true' | ||
| NX_CLOUD_DISTRIBUTED_EXECUTION_AGENT_COUNT: 3 # expected number of agents | ||
| ${{ if eq(variables['Build.Reason'], 'PullRequest') }}: | ||
| NX_BRANCH: $(System.PullRequest.PullRequestNumber) | ||
| TARGET_BRANCH: $[replace(variables['System.PullRequest.TargetBranch'],'refs/heads/','origin/')] | ||
| BASE_SHA: $(git merge-base $(TARGET_BRANCH) HEAD) | ||
| ${{ if ne(variables['Build.Reason'], 'PullRequest') }}: | ||
| NX_BRANCH: $(Build.SourceBranchName) | ||
| BASE_SHA: $(git rev-parse HEAD~1) | ||
| HEAD_SHA: $(git rev-parse HEAD) | ||
|
|
||
| jobs: | ||
| - job: agents | ||
| strategy: | ||
| parallel: 3 | ||
| displayName: Nx Cloud Agent | ||
| pool: | ||
| vmImage: 'ubuntu-latest' | ||
| steps: | ||
| - script: npm ci | ||
| - script: npx nx-cloud start-agent | ||
|
|
||
| - job: main | ||
| displayName: Nx Cloud Main | ||
| pool: | ||
| vmImage: 'ubuntu-latest' | ||
| steps: | ||
| # Get last successfull commit from Azure Devops CLI | ||
| - displayName: 'Get last successful commit SHA' | ||
| condition: ne(variables['Build.Reason'], 'PullRequest') | ||
| env: | ||
| AZURE_DEVOPS_EXT_PAT: $(System.AccessToken) | ||
| bash: | | ||
| LAST_SHA=$(az pipelines build list --branch $(Build.SourceBranchName) --definition-ids $(System.DefinitionId) --result succeeded --top 1 --query "[0].triggerInfo.\"ci.sourceSha\"") | ||
| if [ -z "$LAST_SHA" ] | ||
| then | ||
| echo "Last successful commit not found. Using fallback 'HEAD~1': $BASE_SHA" | ||
| else | ||
| echo "Last successful commit SHA: $LAST_SHA" | ||
| echo "##vso[task.setvariable variable=BASE_SHA]$LAST_SHA" | ||
| fi | ||
| - script: git branch --track main origin/main | ||
| - script: npm ci | ||
| - script: npx nx-cloud start-ci-run --stop-agents-after="e2e-ci" | ||
| - script: npx nx-cloud record -- nx format:check --base=$(BASE_SHA) --head=$(HEAD_SHA) | ||
| - script: npx nx affected --base=$(BASE_SHA) --head=$(HEAD_SHA) -t lint,test,build,e2e-ci --parallel=2 --configuration=ci | ||
| ``` | ||
|
|
||
| This configuration is setting up two types of jobs - a main job and three agent jobs. | ||
|
|
||
| The main job tells Nx Cloud to use DTE and then runs normal Nx commands as if this were a single pipeline set up. Once the commands are done, it notifies Nx Cloud to stop the agent jobs. | ||
|
|
||
| The agent jobs set up the repo and then wait for Nx Cloud to assign them tasks. | ||
|
|
||
| {% callout type="warning" title="Two Types of Parallelization" %} | ||
| The agent strategy of `parallel: 3` and the `nx affected --parallel=2` flag both parallelize tasks, but in different ways. The way this workflow is written, there will be 3 agents running tasks and each agent will try to run 2 tasks at once. If a particular CI run only has 2 tasks, only one agent will be used. | ||
| {% /callout %} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| # Custom Distributed Task Execution on Bitbucket Pipelines | ||
|
|
||
| Using [Nx Agents](/ci/features/distribute-task-execution) is the easiest way to distribute task execution, but it your organization may not be able to use hosted Nx Agents. With an [enterprise license](https://nx.app/enterprise), you can set up distributed task execution on your own CI provider using the recipe below. | ||
|
|
||
| ## Run Custom Agents on Bitbucket Pipelines | ||
|
|
||
| Run agents directly on Bitbucket Pipelines with the workflow below: | ||
|
|
||
| ```yaml {% fileName="bitbucket-pipelines.yml" %} | ||
| image: node:20 | ||
|
|
||
| clone: | ||
| depth: full | ||
|
|
||
| definitions: | ||
| steps: | ||
| - step: &agent | ||
| name: Agent | ||
| script: | ||
| - export NX_BRANCH=$BITBUCKET_PR_ID | ||
|
|
||
| - npm ci | ||
| - npx nx-cloud start-agent | ||
|
|
||
| pipelines: | ||
| pull-requests: | ||
| '**': | ||
| - parallel: | ||
| - step: | ||
| name: CI | ||
| script: | ||
| - export NX_BRANCH=$BITBUCKET_PR_ID | ||
| - export NX_CLOUD_DISTRIBUTED_EXECUTION_AGENT_COUNT=3 | ||
|
|
||
| - npm ci | ||
| - npx nx-cloud start-ci-run --stop-agents-after="e2e-ci" --agent-count=3 | ||
| - npx nx-cloud record -- nx format:check | ||
| - npx nx affected --target=lint,test,build,e2e-ci --parallel=2 | ||
| - step: *agent | ||
| - step: *agent | ||
| - step: *agent | ||
| ``` | ||
|
|
||
| This configuration is setting up two types of jobs - a main job and three agent jobs. | ||
|
|
||
| The main job tells Nx Cloud to use DTE and then runs normal Nx commands as if this were a single pipeline set up. Once the commands are done, it notifies Nx Cloud to stop the agent jobs. | ||
|
|
||
| The agent jobs set up the repo and then wait for Nx Cloud to assign them tasks. | ||
|
|
||
| {% callout type="warning" title="Two Types of Parallelization" %} | ||
| The agents and the `--parallel` flag both parallelize tasks, but in different ways. The way this workflow is written, there will be 3 agents running tasks and each agent will try to run 2 tasks at once. If a particular CI run only has 2 tasks, only one agent will be used. | ||
| {% /callout %} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| # Custom Distributed Task Execution on Circle CI | ||
|
|
||
| Using [Nx Agents](/ci/features/distribute-task-execution) is the easiest way to distribute task execution, but it your organization may not be able to use hosted Nx Agents. With an [enterprise license](https://nx.app/enterprise), you can set up distributed task execution on your own CI provider using the recipe below. | ||
|
|
||
| ## Run Custom Agents on Circle CI | ||
|
|
||
| Run agents directly on Circle CI with the workflow below: | ||
|
|
||
| ```yaml {% fileName=".circleci/config.yml" %} | ||
| version: 2.1 | ||
| orbs: | ||
| nx: nrwl/nx@1.5.1 | ||
| jobs: | ||
| main: | ||
| docker: | ||
| - image: cimg/node:lts-browsers | ||
| environment: | ||
| NX_CLOUD_DISTRIBUTED_EXECUTION_AGENT_COUNT: 3 # expected number of agents | ||
| steps: | ||
| - checkout | ||
| - run: npm ci | ||
| - nx/set-shas | ||
|
|
||
| # Tell Nx Cloud to use DTE and stop agents when the e2e-ci tasks are done | ||
| - run: npx nx-cloud start-ci-run --stop-agents-after=e2e-ci | ||
| # Send logs to Nx Cloud for any CLI command | ||
| - run: npx nx-cloud record -- nx format:check | ||
| # Lint, test, build and run e2e on agent jobs for everything affected by a change | ||
| - run: npx nx affected --base=$NX_BASE --head=$NX_HEAD -t lint,test,build,e2e-ci --parallel=2 --configuration=ci | ||
| agent: | ||
| docker: | ||
| - image: cimg/node:lts-browsers | ||
| parameters: | ||
| ordinal: | ||
| type: integer | ||
| steps: | ||
| - checkout | ||
| - run: npm ci | ||
| # Wait for instructions from Nx Cloud | ||
| - run: | ||
| command: npx nx-cloud start-agent | ||
| no_output_timeout: 60m | ||
| workflows: | ||
| build: | ||
| jobs: | ||
| - agent: | ||
| matrix: | ||
| parameters: | ||
| ordinal: [1, 2, 3] | ||
| - main | ||
| ``` | ||
|
|
||
| This configuration is setting up two types of jobs - a main job and three agent jobs. | ||
|
|
||
| The main job tells Nx Cloud to use DTE and then runs normal Nx commands as if this were a single pipeline set up. Once the commands are done, it notifies Nx Cloud to stop the agent jobs. | ||
|
|
||
| The agent jobs set up the repo and then wait for Nx Cloud to assign them tasks. | ||
|
|
||
| {% callout type="warning" title="Two Types of Parallelization" %} | ||
| The `ordinal: [1, 2, 3]` line and the `--parallel` flag both parallelize tasks, but in different ways. The way this workflow is written, there will be 3 agents running tasks and each agent will try to run 2 tasks at once. If a particular CI run only has 2 tasks, only one agent will be used. | ||
| {% /callout %} |
Oops, something went wrong.