Europa documentation website

[shykes]

Overview

The Europa release introduces major changes, which will require new documentation. This issue is a placeholder for tracking work on these new docs.

Topics to consider include:

• New user experience: command-line syntax, getting started, operator manual
• New developer experience: APIs, programming concepts, sample code
• Migration from earlier versions
• New positioning. When we wrote the current documentation, Dagger was positioned more broadly and vaguely. Now, it is positioned as a portable devkit for CICD. The positioning emphasizes specific benefits and features. The structure of the docs should reflect that.
• Navigate the Universe. A crucial feature of Dagger is the Universe: a large and growing collection of reusable code. The experience of navigating Universe should be crucial to the design of our docs.
• Getting started tutorial
• …

Content map

Here is a map of the content to write, organized in a rough structure. Work in progress. This is a placeholder for discussion and planning.

• Getting started
• Guides
  • Learn CUE
    • Dagger for frontend development
    • Dagger for MLops
    • Dagger for backend development
    • CUE 101: the basics
    • Intermediate CUE: multi-line strings
    • Intermediate CUE: …
    • Advanced CUE: …
    • Advanced CUE: …
  • Testing your DAG
  • Managing secrets with Dagger
  • Dagger for local development
    • Using local directories
    • Connecting to local network services
    • …
  • Building Docker containers with Dagger
  • Integrating Dagger in your CI
  • Deploying review environments with Dagger
  • Buildkit administrator guide
    • Buildkit with Docker
    • Buildkit clustering
    • Multi-architecture with buildkit
    • Buildkit no AWS: the dream setup
    • Dedicated buildkit or embedded buildkit?
• Use cases
  • Developer Dagger with Dagger
  • How changelog.com improved its CICD with Dagger
  • How Particubes deploys its documentation faster with Dagger

Delivery plan

There are 3 parts to this. We will focus on 1 & 2 for 0.2.0, and continue with 3 without blocking the 0.2.0 release.

1/3. Getting Started

• Shape of the new Europa docs #1568
• Europa docs - CI/CD in your local dev #1586 + Update Europa docs - CI/CD in your local dev to the new netlify.cue plan #1732
• Capture the first CI environments that we intend to document #1678
• Europa docs: From local dev to CI environment #1693

2/3. Core Concepts

• Add dagger.#Plan core concept to Europa Docs #1683 + Europa docs plan updates after Client API & dagger do #1735
• Interacting with the client (configuring inputs) Client API #1668
• How to use secrets Client API #1668
• [Docs] Building container images #1708
• Create your own package → starting point https://docs.dagger.io/1010/dev-cue-package/
• Make your builds fast - blocked on BuildKit caching cc @TomChv https://github.com/TomChv/buildkit-cache-test

3/3. Use Cases

• Elixir/Phoenix on Kubernetes
• Docusaurus on Netlify
• Go with GoReleaser
• Europa docs: Go on Docker Swarm #1602 (this is the pre-Europa version) -> https://docs.dagger.io/1211/go-docker-swarm/
• Svelte on Vercel

There are a few other tasks which need to be done part of this & are not related to content:

• Add link to Europa docs #1679
• Add a DRAFTS section to the Europa docs sidebar #1680
• Temporarily remove Use Cases from the Europa docs sidebar #1687 (will be added back post 0.2.0)
• Default docs to Europa #1742
• Fix pre-Europa API reference post 0.2.0

[slumbering]

@Camille-Elairin : Navigate the Universe. A crucial feature of Dagger is the Universe: a large and growing collection of reusable code. The experience of navigating Universe should be crucial to the design of our docs.

[samalba]

@gerhard @talentedmrjones Please update as soon as you get a rough plan, we need to set a target date to synchronize with enabling europa by default on a dagger release.

[gerhard]

While my initial instinct was to start editing the first comment, I decided to start by sharing my context as a point-in-time reference, sleep on it, refine it and begin the PRs game that will start on "the delivery plan".

What am I talking about?

https://github.com/dagger/dagger.io/issues/38 tells a story & makes a promise. I think of it as the Why and the What. This issue delivers on that promise. I think of it as the How.

In my mind, this issue will be resolved when a new Dagger user knows:

• How to get started locally (a.k.a. dev)
• How to transition to GitHub (a.k.a. prod)
• How to switch to a different CI (a.k.a. regulated prod)

My focus is the new Dagger user - I think of her as Elise. She is an actual front-end developer that I worked with many years ago, and it helps me empathise with the end-user. My hope is that sometimes soon, she will be using Dagger to deploy her apps to Vercel (Netlify is ~10x slower, but that's a story for a different issue).

Keep that new user in mind - whoever they may be - and focus on making their experience as simple and pleasant as possible.

The higher order bit is a permanent awareness that we are helping our users achieve specific tasks. Keep it simple and focus on the task at hand. It really doesn't matter how much we know about this subject, how passionate we are about CUE or Buildkit, or how much there is to it. Help the user accomplish the reason that made them click on that docs link.

What is the value for the user?

With all the above in mind, this is the docs.dagger.io flow that I imagine taking Elise through:

• Get started with Dagger on macOS
• Transition from local dev to GitHub & GitHub Actions

When Elise is comfortable running Dagger locally & on GitHub, I would start introducing slightly more advanced topics, such as:

• Manage secrets with Dagger
• Configure inputs just-in-time
• Build container images with Dagger
• Speed up your builds

Lastly, I would give three real-world production use cases of Dagger which cover detailed and contextual specifics:

• Elixir/Phoenix on Kubernetes - changelog.com
• Svelte on Vercel - rawkode.dev
• Go on Docker Swarm - particubes.com
• Docusaurus on Netlify - docs.dagger.io
• Go with GoReleaser - dagger

While there is a lot more that we could do, I feel that this is enough to help a new user experience the Dagger promise.

What happens next?

• Tie this issue back to https://github.com/dagger/dagger.io/issues/38 🪢 https://github.com/dagger/dagger.io/issues/38#issuecomment-1027384090
• Share references that frame my docs context (video, audio & written format)
• Put the existing docs navigation next to the newly proposed one (most of the pages that I listed above)
• Submit all “must have” doc pages in a new, long-running PR Shape of the new Europa docs #1568

---

Items out of scope for Europa / 0.2.0

• Start GitHub discussion so that users can tell us which doc pages they are missing the most
• dagger cli reference Improve documentation #621
• Review issues for other doc pages that users may have asked for
• Preview environments

[gerhard]

References

• "I would like the docs for http://dagger.io to be world-class… Any recommendations or advice?" - Solomon, Nov. 2021
• "I would propose starting off with common use case and get a feedback loop possible where customers get to somewhat steer the topics they want next. Maybe via a vote system to prioritise . The community leads it all." Frankie Onuonga via Twitter, Nov. 2021
• The Documentation System +1 from @samalba
• Maybe it’s time we re-think docs - Kathy Korevec, Jun. 2021
• 🎙 Ship It #17: Docs are not optional - Kathy Korevec, Aug. 2021
• 📚 Working Backwards - Colin Bryar & Bill Carr, Feb. 2021
• 🎬 LeadDevBerlin: Writing effective documentation - Beth Aitman, Dec. 2019
• 🎬 DocOps: engineering great documentation - Adam Butler, Dec. 2017
• 🎬 PyCon: Writing great documentation - Jacob Kaplan-Moss, Sep. 2014

Noteworthy examples

• Tailscale @shykes' inspiration for unique IDs per doc
• Tailwind CSS @gerhard's latest memorable doc experience
• The open-source repo for docs.github.com
• Elixir School docs repository
• Fly.io, and also the hands-on guide which @slumbering really likes
• Node.js - Getting started +1 from @talentedmrjones

I think that we should do less of this:

• Shipa Docs - too much information with a complex structure & weak cohesion (inc. flow)

[gerhard]

This is the existing vs new docs structure that I have in mind:

EXISTING DOCS STRUCTURE                        NEW DOCS STRUCTURE
-----------------------                        ------------------

Introduction                                   Getting Started
|- What is Dagger?                             |- Local development on macOS
|- Dagger vs Other Software                    |- GitHub repository with GitHub Actions

Install Dagger

Learn Dagger                                   Core Concepts
|- What is Cue?                                |- Manage secrets
|- Get started with Dagger                     |- Configure inputs
|- Deploy to Google Cloud Run with Dagger      |- Preview environments
|- Deploy to Kubernetes with Dagger            |- Build container images
|- Provision infrastructure with Dagger        |- Dependency caching
|  and AWS CloudFormation
|- Integrate Dagger with Github Actions
|- Develop a new CUE package for Dagger
|- Manage packages using the package manager

Use Cases                                      Use Cases
|- Make your CI workflow portable              |- Elixir/Phoenix on Kubernetes - changelog.com
                                               |- Svelte on Vercel - rawkode.dev
Universe - API Reference                       |- Go on Fly.io - particubes.com
|- Index
...

Administrator Manual
|- Dagger Operator Manual

These are the principles that I have based the new docs structure on:

1. Start with what users need today
2. Write less
3. Focus on flow & experience, not documentation

For more context, please see References & Noteworthy examples above: #1327 (comment)

While I wait for input on the new docs structure, I will start on the PR that captures the docs skeleton that we absolutely must have for Europa (I proposed 10 pages above). We can refine the structure there - the one above is a quick preview of what is coming. In my mind, that PR will be the first concrete step towards the Europa docs delivery plan.

I expect things to change as we negotiate what must be there for launch. My perspective is: keep it simple, less is more & keep improving.

[slumbering]

"I would like the docs for http://dagger.io to be world-class… Any recommendations or advice?" - Solomon, Nov. 2021

Sebastien Lorber answer resume pretty much my thought: https://twitter.com/sebastienlorber/status/1460918381641621509

At start I chose Docusaurus as we needed a quick nice looking documentation and easy to maintain. The other idea was to get a versioning documentation but this has been abandoned.

The cons about Docusaurus:

• we need to overwrite (or swizzling) components to customize our UI which is discourage during Docusaurus Beta phase
• still in beta and for some release managing with dependabot we've had some breaking changes (mainly css and sometimes js behaviour).

On the other hand we could get pretty much everything we want with a documentation from scratch with Next.js but like Sebastien Lorber said it will be time consuming.

Just a thought about @gerhard Noteworthy examples and the one I particularly like (fly.io), it reminds me a conversation with @samalba about keeping a signup flow for people to get Dagger newsletter for instance but also for our future cloud service. Fly.io has a really inspire way of using terminal to make user signup during the "Get started" process (same for sign-In). I also love their Get started process with all the steps to follow (left sidebar).

This could feat very well with Dagger

Getting Started
|- Install Dagger
|- Sign Up / Sign In
|- Clone example App
|- Write Dagger Plan
|- Deploy with Dagger
|- ...

[slumbering]

Laravel voyager has also a very inspired and friendly "Get started" section: https://voyager-docs.devdojo.com/

[gerhard]

At start I chose Docusaurus as we needed a quick nice looking documentation and easy to maintain.

That is a great start. I think that we should keep it as is for now.

The other idea was to get a versioning documentation but this has been abandoned.

I don't think that we need this if every page gets a unique ID. Think /1001/install/ today, /1002/install/ tomorrow, etc.

This is how Tailscale does it for their docs, and it seems most sensible.

On the other hand we could get pretty much everything we want with a documentation from scratch with Next.js but like Sebastien Lorber said it will be time consuming.

I see us doing that for 1.0, but not for 0.2.0.

A smaller and more valuable step would be to migrate from Netlify to Vercel (the Next.js company), since the speed improvement is 10x. I am mentioning this briefly, as an idea, this PR is the wrong context to go deeper on it.

Just a thought about @gerhard Noteworthy examples and the one I particularly like (fly.io), it reminds me a conversation with @samalba about keeping a signup flow for people to get Dagger newsletter for instance but also for our future cloud service.

This definitely resonates with me. What are your thoughts @slumbering on doing a simpler version first, and then iterate towards this:

Getting Started
|- Install Dagger
|- Sign Up / Sign In
|- Clone example App
|- Write Dagger Plan
|- Deploy with Dagger
|- ...

[gerhard]

Laravel voyager has also a very inspired and friendly "Get started" section: https://voyager-docs.devdojo.com/

I don't think that fragmenting information the way they do it is a good thing.
There needs to be a flow, and clicking links every 30 seconds doesn't work well with this type of primarily text content.

I imagine stories that draw the user in and encourage them to try things out with the goal of having small wins that add up to something greater than the sum of their parts. A slideshow like experience might work, but Laravel Voyager - Get started doesn't seem to capture this well to me. This however does: https://growth.design/case-studies/amazon-purchase-ux

[gerhard]

It would be very helpful for the beta release if next week we made progress with the docs for the following use cases:

• Elixir/Phoenix on Kubernetes @jlongtine had most context
• Docusaurus on Netlify @slumbering has most context
• Go with GoReleaser depends on @samalba Port the dagger Makefile for go build, lint and test #1589
• Europa docs: Go on Docker Swarm #1602 @grouville has most context
• Svelte on Vercel @TomChv has most context via Local cache error/unused with Buildkit go client moby/buildkit#2599

When I return, I will continue with the rest of the docs.

---

@aluzzardi shares all the context for this comment.

[shykes]

After backlog review yesterday with @samalba and @aluzzardi: we are stretched too thin to work on so many use cases concurrently, especially with remaining API changes that will require follow-up changes and tests. For the sake of beta.1 release, let's focus on:

1. A "getting started" guide that gets the job done: getting people started. Doesn't have to be fancy (actually the less fancy, the better)
2. A small number of high-quality use cases, piggybacking on work that is already done, or already prioritized. Tentative list:
   • Todoapp, or a suitable replacement if @gerhard chooses one
   • Dagger CI (Port the dagger Makefile for go build, lint and test #1589)
   • Changelog CI
   • Everything else can wait at least until after beta.1
3. Basic concepts, at least enough for new users to orient themselves
4. A basic FAQ/knowledge base system that we can incrementally (and rapidly) add content to, as users ask questions. This doesn't need to be perfect or fully structured! This will become a staging area for improving and expanding the rest of the docs.

We also need to review the website landing page, from the perspective of an engineer learning about Dagger for the first time, and make sure it is suitably detailed, technically accurate, and aligned with the "getting started" guide (which it links to).

[gerhard]

Sitrep for @aluzzardi: after talking to @shykes yesterday, we are thinking to push Use Cases after 0.2.0 ships and focus on the Getting Started and Core Concepts in the next 2 days, for 0.2.0.

The goal is to get the first pass complete by Mar 3, and continue iterating on the rest at the same pace, without any change after 0.2.0 comes out. Who knows, maybe we get refined with our strategy and schedule 0.3.0 for end of March, and then 0.4.0 end of April. I love cadence, and I definitely think that we should do more in this are after the big Europa push, which is 0.2.0.

• I will update the Delivery plan at the top to reflect this

[shykes]

There is much work left to do on the Dagger 0.2 documentation, but it does exist and is in a launchable state. I hereby declare this issue closed :)