Add Web ARChive example (warc) by gerhard · Pull Request #105 · dagger/cloak

gerhard · 2022-08-24T16:47:35Z

Uses Cloak's Go SDK.

Create a warc file
Save static files
Commit & push to GitHub
Wait & verify GitHub Pages update

sipsma · 2022-08-24T19:48:22Z

Thanks for giving this a shot Gerhard! It'll be extremely helpful to have more validation that the workflow implementation is moving in the right direction.

Couple of assorted pointers (after pulling my latest changes in #95)

To generate the client stubs you can run cloak generate --sdk=go --client --output-dir=. from the examples/wp-to-hugo directory (this is different than the current docs in main, need to update PR w/ that).
- If you add dependencies on extensions in other localdirs, then you can run from the root of the cloak repo cloak generate -p examples/wp-to-hugo --sdk=go --client --output-dir=examples/wp-to-hugo.
- There's also support for pulling dependencies from git now (example), in which case you don't have to switch to the root of the cloak repo. But you will need an ssh agent and SSH_AUTH_SOCK env setup.
It will be easiest to execute the workflow by running go run . from the examples/wp-to-hugo directory for now.
- If you want to execute it from somewhere else you'll have to do CLOAK_CONFIG=path/to/examples/wp-to-hugo/cloak.yaml go run path/to/examples/wp-to-hugo

I realize much (or most) of this is still very clunky if you don't follow the exact right pattern. Please let me know what else comes up and feel free to let me know what's missing, frustrating, unintuitive, etc. and we can figure out how to address it.

gerhard · 2022-08-30T16:57:56Z

@sipsma @aluzzardi @shykes Is #95 still the best branch to base my PR onto?

I am just about to start adding v2 Go workflow code and I'm wondering which is the best branch to use as my starting point.

Thanks!

sipsma · 2022-08-31T00:06:21Z

Is #95 still the best branch to base my PR onto?

Yes it still is, been merging commits from it into main (but also adding more as demov2 iteration continues). I am actively working on getting it merged though and will let you know once you can switch back to being based on main.

sipsma · 2022-09-01T03:56:42Z

@gerhard #95 has been merged finally, thank you for dealing with the hassle as it got cleaned up and ready for main!

There are some newly updated docs on creating projects with Go here: https://github.com/dagger/cloak/blob/main/docs/writing_extensions_go.md

One nice improvement is that my previous comment is now outdated: once you've updated cloak.yaml to the new format described in that doc you can just run cloak generate anywhere in the project tree and everything will get generated for you in the right place (no more insane number of --flags)

There's still plenty of rough edges of course though so let me know what you run into.

All the context is in the README.md Signed-off-by: Gerhard Lazu <gerhard@dagger.io>

Squash Signed-off-by: Gerhard Lazu <gerhard@dagger.io>

Tanguy was missing a workdir example, which this contains. Signed-off-by: Gerhard Lazu <gerhard@dagger.io>

While it doesn't fully work, it's another good step in the right direction. Signed-off-by: Gerhard Lazu <gerhard@dagger.io>

netlify · 2022-09-08T16:46:39Z

❌ Deploy Preview for cloak-docs failed.

Name	Link
🔨 Latest commit	`cc2bd75`
🔍 Latest deploy log	https://app.netlify.com/sites/cloak-docs/deploys/631b2f88ac6973000e248b18

sipsma · 2022-09-08T18:23:53Z

+			Args:    []string{"./run.sh"},
+			Workdir: "/mnt",
+			Mounts: []core.MountInput{
+				core.MountInput{Fs: workdirResp.Host.Workdir.Read.ID, Path: "/mnt/"},


I think there is a bug in cloak right now where we aren't cleaning the paths, so because you have /mnt/ here (trailing /) and /mnt (no trailing) on line 43, cloak can't find your mount.

I'll go fix (can't imagine it's more than a 1 line change) but you can also change this to be /mnt in the meantime.

sipsma · 2022-09-08T18:25:29Z

+		// Start from Alpine
+		image, err := core.Image(ctx, "alpine:3.15")
+		if err != nil {
+			return err
+		}
+
+		fs := &image.Core.Image
+
+		// Install deps & declare workdir
+		deps, err := core.Exec(ctx, fs.ID, core.ExecInput{
+			Args:    []string{"apk", "add", "-U", "--no-cache", "wget", "bash"},
+			Workdir: "/mnt",
+		})
+		if err != nil {
+			return fmt.Errorf("failed to install: %s", err)
+		}
+		fs = deps.Core.Filesystem.Exec.Fs


If you want, you can use the alpine extension to do this too. Can just add a git dep (example here)

dolanor · 2022-09-09T10:58:41Z

Found that, just leaving it here: https://github.com/richardlehane/webarchive
It has link to other more general purpose implementation, but that could be useful.

Technically, it should not git commit & push, but I'm trying to figure out if I can write it in a single flow, before I start splitting it up. Signed-off-by: Gerhard Lazu <gerhard@dagger.io>

aluzzardi · 2022-09-16T22:05:21Z

This repository has moved to https://github.com/dagger/dagger/tree/cloak

Please re-open this PR in dagger/dagger:

From the cloak repository:

Fetch dagger/dagger

git remote add dagger git@github.com:dagger/dagger.git
git fetch --all

[optional but recommended] Duplicate your branch

git checkout your-pr-branch
git checkout -b your-pr-branch-copy

Rebase your branch on top of dagger/dagger's cloak branch

NOTE: there might be some commits not authored by you. Delete them.

git rebase -i dagger/cloak

Update your local cloak repo to be downstream of your dagger fork

git remote set-url origin git@github.com:<your_username>/dagger.git
git push --set-upstream origin your-pr-branch-copy

Re-create PR in dagger/dagger

Make sure you have the GitHub CLI installed (on macOS: brew install gh)

gh pr create -R dagger/dagger --base cloak

Close PR in dagger/cloak

gerhard · 2022-09-17T11:10:26Z

It's OK to close this. All the learnings are there, will start fresh from a new baseline & just link to this one instead.

gerhard and others added 4 commits September 2, 2022 16:12

First stab at wp-to-hugo demo

283dba7

All the context is in the README.md Signed-off-by: Gerhard Lazu <gerhard@dagger.io>

cdn -> cnd fix

61d305d

Squash Signed-off-by: Gerhard Lazu <gerhard@dagger.io>

Add current state of wp-to-hugo example

6fc7c3e

Tanguy was missing a workdir example, which this contains. Signed-off-by: Gerhard Lazu <gerhard@dagger.io>

Update Web ARChive example to current version

6485d77

While it doesn't fully work, it's another good step in the right direction. Signed-off-by: Gerhard Lazu <gerhard@dagger.io>

gerhard changed the title ~~Add WordPress to Hugo example~~ Add Web ARChive example (warc) Sep 8, 2022

sipsma reviewed Sep 8, 2022

View reviewed changes

sipsma mentioned this pull request Sep 8, 2022

Clean exec mount paths. #186

Merged

Take another small step in the web archive example implementation

cc2bd75

Technically, it should not git commit & push, but I'm trying to figure out if I can write it in a single flow, before I start splitting it up. Signed-off-by: Gerhard Lazu <gerhard@dagger.io>

gerhard closed this Sep 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Web ARChive example (warc)#105

Add Web ARChive example (warc)#105
gerhard wants to merge 5 commits into
dagger:mainfrom
gerhard:wp-to-hugo

gerhard commented Aug 24, 2022 •

edited

Loading

Uh oh!

sipsma commented Aug 24, 2022

Uh oh!

gerhard commented Aug 30, 2022

Uh oh!

sipsma commented Aug 31, 2022

Uh oh!

sipsma commented Sep 1, 2022

Uh oh!

netlify Bot commented Sep 8, 2022 •

edited

Loading

Uh oh!

sipsma Sep 8, 2022

Uh oh!

sipsma Sep 8, 2022

Uh oh!

dolanor commented Sep 9, 2022

Uh oh!

aluzzardi commented Sep 16, 2022

Uh oh!

gerhard commented Sep 17, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

gerhard commented Aug 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sipsma commented Aug 24, 2022

Uh oh!

gerhard commented Aug 30, 2022

Uh oh!

sipsma commented Aug 31, 2022

Uh oh!

sipsma commented Sep 1, 2022

Uh oh!

netlify Bot commented Sep 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Deploy Preview for cloak-docs failed.

Uh oh!

sipsma Sep 8, 2022

Choose a reason for hiding this comment

Uh oh!

sipsma Sep 8, 2022

Choose a reason for hiding this comment

Uh oh!

dolanor commented Sep 9, 2022

Uh oh!

aluzzardi commented Sep 16, 2022

Uh oh!

gerhard commented Sep 17, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gerhard commented Aug 24, 2022 •

edited

Loading

netlify Bot commented Sep 8, 2022 •

edited

Loading