Add `src actions exec` command to generate arbitrary campaign diffs #80

sqs · 2020-01-14T05:05:17Z

Usage:

Run:

go install ./cmd/src && \
docker build -t sqs1/add-lsif-to-build-pipeline-action actions/add-lsif-to-build-pipeline && \
src actions exec -j 4 -keep-logs -f actions/add-lsif-to-build-pipeline/action.json  | \
  src campaign plan create-from-patches

This will create a campaign that adds GitHub Action LSIF build steps to repositories matched in the action.json file (see and change the query to match repositories that exist on your instance). Note: You should only use this with the test sd9 and sd9org repositories, because if you click the link and press Create, it will create PRs on real live repositories!

Ping me (@sqs) for help on anything here if you pick up this PR.

TODOs:

Format of action.json files and overall model for specifying what to run, which repositories to run it on, etc.
CLI UX (it's weird you need to pipe stuff)
Clean up code and add tests
Currently only works on Linux, needs to support macOS (the diff CLI flags we use only work on GNU diff not BSD diff, and the Docker volume mounting breaks due to macOS's /var/tmp/folders/... sandboxing)
~~Document~~ (documentation & guide here: https://github.com/sourcegraph/sourcegraph/pull/8064 and also in the -help output of the commands)
The terminal library (that lets us print progress bars) does not properly handle colored text. If your terminal is too narrow, the progress bars "climb" up the screen. This is due to the terminal library thinking that a line has (for example) 80 chars but it only has (for example) 65 visible chars plus 15 ANSI escape sequence chars for colors. This appears to be a problem in all Go terminal libs. I started to make my own to fix this on a flight w/o WiFi; if it would be helpful, remind me to push it up.
The proposed src campaigns create command is not implemented
Remove sample actions in the actions/ dir. These are good candidates for examples in the docs, but they don't belong in this repo.

mrnugget · 2020-01-17T08:46:07Z

I'm picking this up in 3.13

mrnugget

Leaving a first batch of notes to myself while reviewing/working on this PR.

cmd/src/actions_cache.go

cmd/src/actions_exec.go

mrnugget · 2020-01-21T15:00:50Z

cmd/src/actions_exec.go

+	}
+	displayUserCacheDir := strings.Replace(userCacheDir, os.Getenv("HOME"), "$HOME", 1)
+	var (
+		fileFlag        = flagSet.String("f", "<stdin>", `The action file. (required)`)


Note to myself: I think it might be more idiomatic to make -f optional and read from stdin if it's not set. Or use -f - to say "read from stdin"

cmd/src/actions_exec.go

mrnugget

Left more notes to myself

cmd/src/actions_exec_backend.go

mrnugget · 2020-01-22T09:40:10Z

cmd/src/actions_exec_backend_runner.go

+
+	// Add to cache if successful.
+	if err == nil {
+		if err := x.opt.cache.set(ctx, cacheKey, status.Patch); err != nil {


Note to myself: does this also write to the cache when status.Patch == CampaignPlanPatch{}? Or do we need the len(patch) > 0 check here too?

cmd/src/actions_exec_backend_runner.go

I think this makes a lot more sense from a users perspective, especially with the upcoming changes in #80.

mrnugget · 2020-01-23T09:59:04Z

Note: I think the problem is not BSD diff vs GNU diff but rather GNU diff 2.8.1 vs GNU diff 3.7.

mrnugget · 2020-01-23T16:51:35Z

Update: I think I'm happy with the code so far and would move on to giving a last shot at fixing the moving/scrolling UI, documentation and adding campaign create.

The corresponding Sourcegraph GraphQL API mutations will be (temporarily) removed until they support full remote execution of arbitrary tools.

mrnugget · 2020-01-24T15:30:51Z

I broke the UI logic, I think, since it now also prints progress when the output is piped to another command.

mrnugget · 2020-01-24T16:42:06Z

@sqs I can't think of any UX improvements that are hard requirements right now. I actually do like the piping. What were your concerns there?

I started to work on campaigns create in this PR here #87 which is stacked on top of this one.

mrnugget · 2020-01-27T12:55:04Z

Documentation: https://github.com/sourcegraph/sourcegraph/pull/8064

mrnugget · 2020-01-27T13:40:17Z

I'm going to merge this now because the basic feature set and documentation is there. It works, but can be polished:

"interactive" mode for creating campaigns that opens the editor
fix the "scrolling" up of the progress bar
show progress when CLI is downloading repository archives
...

But the PR has gotten quite big and further additions would make it unwieldy. Once it's merged I'll open new PRs to polish things.

@sqs Let me know if you have any specific concerns besides the ones I mentioned, or whether your don't think the things I've mentioned don't have priority right now.

I think this makes a lot more sense from a users perspective, especially with the upcoming changes in #80.

) * WIP: add `src actions exec` command to generate arbitrary campaign diffs * Use `/tmp` as TempDir prefix because it works with Docker for Mac * Remove fixed TODO comment * Fall back to degraded behavior for GNU diff <3.3 * Remove example actions * Check all diff flags that we need * Various small fixes and cleanups * Check whether stdin is a pipe before attempting to read * Extract getDockerImageContentDigest * Extract Actions UI printer into separate struct * Log if skipping repo because of missing default branch * Stop leaking goroutine * Move preparation of action to separate function * Flush at the end of update not at the beginning * Add timeout flag for action execution * We don't need a trailing slash * Consistently use temp dir prefix * Rename types to get rid of redundant information * Use "-" to signify STDIN * Remove campaign plan creation via spec The corresponding Sourcegraph GraphQL API mutations will be (temporarily) removed until they support full remote execution of arbitrary tools. * Revert extraction of UI printer to fix piping * Add `campaigns create` command * Add documentation examples for `src actions exec` * Print friendly message when a campaign has been created * More documentation * Use full or relative URL of Campaign * Use ResolveReference to construct Campaign URL Co-authored-by: Thorsten Ball <mrnugget@gmail.com>

This was referenced Jan 14, 2020

campaigns Spike: Executing src actions on a server sourcegraph/sourcegraph-public-snapshot#7723

Closed

Core Services: 3.13 Tracking Issue sourcegraph/sourcegraph-public-snapshot#7719

Closed

mrnugget self-assigned this Jan 17, 2020

mrnugget mentioned this pull request Jan 17, 2020

a8n: Finish implementation of src actions exec sourcegraph/sourcegraph-public-snapshot#7850

Closed

mrnugget force-pushed the a8n/actions branch 3 times, most recently from 22465a1 to baa7686 Compare January 20, 2020 16:25

mrnugget reviewed Jan 21, 2020

View reviewed changes

mrnugget reviewed Jan 22, 2020

View reviewed changes

mrnugget added a commit that referenced this pull request Jan 22, 2020

Rename "campaigns plan preview" to "campaigns plan create"

0f94554

I think this makes a lot more sense from a users perspective, especially with the upcoming changes in #80.

mrnugget mentioned this pull request Jan 22, 2020

Rename "campaigns plan preview" to "campaigns plan create" #85

Merged

mrnugget added a commit that referenced this pull request Jan 23, 2020

Rename "campaigns plan preview" to "campaigns plan create" (#85)

8df2b07

I think this makes a lot more sense from a users perspective, especially with the upcoming changes in #80.

sqs and others added 2 commits January 23, 2020 07:46

WIP: add src actions exec command to generate arbitrary campaign diffs

45312ec

Use /tmp as TempDir prefix because it works with Docker for Mac

190ddff

mrnugget force-pushed the a8n/actions branch from 1267494 to 190ddff Compare January 23, 2020 06:47

Remove fixed TODO comment

87b426a

mrnugget added 12 commits January 23, 2020 11:25

Fall back to degraded behavior for GNU diff <3.3

8bc4a7e

Remove example actions

0ee2a33

Check all diff flags that we need

55e9d42

Various small fixes and cleanups

f708e65

Check whether stdin is a pipe before attempting to read

7b977a3

Extract getDockerImageContentDigest

e83d770

Extract Actions UI printer into separate struct

591e8d9

Log if skipping repo because of missing default branch

cf80d8a

Stop leaking goroutine

89dded7

Move preparation of action to separate function

ce9acdc

Flush at the end of update not at the beginning

1f3cbd5

Add timeout flag for action execution

301319a

mrnugget added 4 commits January 23, 2020 17:12

We don't need a trailing slash

a00c5ca

Consistently use temp dir prefix

0ee6983

Rename types to get rid of redundant information

c24666b

Use "-" to signify STDIN

1e1ec71

Remove campaign plan creation via spec

bf97036

The corresponding Sourcegraph GraphQL API mutations will be (temporarily) removed until they support full remote execution of arbitrary tools.

sqs mentioned this pull request Jan 24, 2020

a8n: support creating campaign plans from API only, not UI sourcegraph/sourcegraph-public-snapshot#8008

Merged

Revert extraction of UI printer to fix piping

784f661

Add campaigns create command

1251800

mrnugget mentioned this pull request Jan 27, 2020

WIP: a8n: Add campaigns create command #87

Closed

mrnugget added 2 commits January 27, 2020 11:18

Add documentation examples for src actions exec

7bfe60b

Print friendly message when a campaign has been created

f2f0e3f

mrnugget mentioned this pull request Jan 27, 2020

a8n: Document how to run an action, create a campaign plan and campaign with CLI sourcegraph/sourcegraph-public-snapshot#8064

Merged

mrnugget added 3 commits January 27, 2020 14:20

More documentation

c6ef2ee

Use full or relative URL of Campaign

ccc2d8d

Use ResolveReference to construct Campaign URL

8e2c894

mrnugget marked this pull request as ready for review January 27, 2020 13:36

mrnugget changed the title ~~WIP: add src actions exec command to generate arbitrary campaign diffs~~ Add src actions exec command to generate arbitrary campaign diffs Jan 27, 2020

mrnugget merged commit cbb5f08 into master Jan 27, 2020

creachadair mentioned this pull request Jan 27, 2020

Update ctxhttp module path. #88

Merged

scjohns pushed a commit that referenced this pull request Apr 24, 2023

Rename "campaigns plan preview" to "campaigns plan create" (#85)

f7819f2

I think this makes a lot more sense from a users perspective, especially with the upcoming changes in #80.

keegancsmith deleted the a8n/actions branch November 11, 2025 20:52

Add src actions exec command to generate arbitrary campaign diffs #80

Add src actions exec command to generate arbitrary campaign diffs #80

Uh oh!

Conversation

sqs commented Jan 14, 2020 • edited by mrnugget Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mrnugget commented Jan 17, 2020

Uh oh!

mrnugget left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mrnugget Jan 21, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mrnugget left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mrnugget Jan 22, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mrnugget commented Jan 23, 2020

Uh oh!

mrnugget commented Jan 23, 2020

Uh oh!

mrnugget commented Jan 24, 2020

Uh oh!

mrnugget commented Jan 24, 2020

Uh oh!

mrnugget commented Jan 27, 2020

Uh oh!

mrnugget commented Jan 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add `src actions exec` command to generate arbitrary campaign diffs #80

Add `src actions exec` command to generate arbitrary campaign diffs #80

sqs commented Jan 14, 2020 •

edited by mrnugget

Loading