Core user flow for website and the job stories (v0.1)

[demenech]

When a user gets to the Portal.JS website, what is he going to be looking for initially? Will he be able to bootstrap a Portal.JS project in ~5 minutes or less?

❓ Questions:

What are the issues with our current flow?

What can we do to fix those issues?

What can we improve in our current flow?

💡 Ask someone who's not familiar with the Portal.JS to bootstrap a project

Plan (in progress)

graph TD

landing[Landing Page]

landing --"click get started"--> docs[Docs - getting started]

landing -.navbar.-> blog[Blog]

Loading

docs is all one page for now (can split later when needed)

[demenech]

CC: @rufuspollock @luccasmmg @anuveyatsu

[luccasmmg]

I was going to open up a separate discussion for a few thoughts that I have been having after talking with Anu on Friday and working on the github backed example, but turns out this is a nice place to put those, so apologies for the wall of text, I tried to be as concise as possible.

Obs: Sorry if there are some typos

Examples should be self sufficient

I think the current repo is not setup in the best way to allow a quick creation and deployment of a data portal by normal users. The reason being nx.

Summary if you don't want to read everything

Every example becomes a full fledged app with its own package.json and its own node_modules. If we want to share anything between them we publish a package to npm, if a user wants to have his/her own data portal they dont need to fork and clone the entire monorepo.

Pros:

• No need for users to fork and clone a gigantous repo
• Users can click on a button and deploy everything in minutes
• The getting started for users gets much much simpler
• Users don't need to know anything about nx or monorepos, they don't even need to be aware those are being used.

Cros:

• Our own developer experience gets worse slightly worse.
• If we want to share stuff between examples that stuff needs to firstly get published to npm

What we have now

If a user wants to have it's own data portal, currently he needs to fork and clone the entire repo, create an example inside the examples folder. And then run nx serve <app-name> to see the app running on localhost.

Imagine he wants to create an portal using the ckan-example as its template. He will need to have all of this store in his computer, when all he really wants is the ckan-example folder.

.
├── babel.config.json
├── CONTRIBUTING.md
├── DESIGN.md
├── dist
│   ├── cypress
│   ├── examples
│   └── packages
├── examples
│   ├── ckan-2021
│   ├── ckan-example <<<<<< ACTUALLY WHAT HE WANTS
│   ├── dataset-frictionless
│   └── simple-example
├── jest.config.ts
├── jest.preset.js
├── license
├── markdown.db
├── nx.json
├── package.json
├── package-lock.json
├── packages
│   └── portaljs-components
├── README.md
├── SECURITY.md
├── site <<<< WHY SHOULD HE CLONE THE WEBSITE JUST TO HAVE A SIMPLE PORTAL RUNNING?
│   ├── components
│   ├── config
│   ├── content
│   ├── layouts
│   ├── lib
│   ├── next-env.d.ts
│   ├── package.json
│   ├── package-lock.json
│   ├── pages
│   ├── postcss.config.js
│   ├── public
│   ├── README.md
│   ├── styles
│   ├── tailwind.config.js
│   └── tsconfig.json
├── tools
│   ├── generators
│   └── tsconfig.tools.json
└── tsconfig.base.json

But it gets worse, imagine this user wants to deploy the app to Vercel, he will not only need to make all of these overrides.

But even the framework preset(That is currently on Next.JS) will not get detected automatically, by default Vercel will detect a NX Deployment as "Other" and the deployment will give a 404 if the user tries to preview it.

What i think we should have

• User goes to some page in the docs
• There is a bright blue button saying "Deploy on Vercel"
• He clicks on it
• He gets a self suficient git repo out of it, with a full deployment working

Why this is the way it is?

The reason because of that is that we are using nx, which is a monorepo management tool, it's not a nx problem only, it would be like that if we were using any sort of monorepo management tool(eg: turbo).

On nx for example, the apps inside examples will not have a package.json describing every single dependency that is required, instead we have a global package.json file that has the job of describing every single app/library inside the repo.

This of course has a number of advantages, you get only one version of a package for every app, meaning you will have an easier time making sure that everything stays up to date, you will only need to do one npm install for the entire repo, you can cache stuff more easily, so on and so forth.

Problem is, now your app lives in some kinda of life support environment, meaning that it can't live outside, its bound by nx weird technicalities, and now if you want to do something as simple as add tailwind you need to get into a rabbit hole of documentation.

Whats my proposal

Simple, we keep everything inside the same repo, but we stop using nx to manage at least the examples, what this means? As of right now where we dont have any shared libraries, this would mostly mean that instead of running nx serve <app-name> i would need to go inside the example folder and run npm run dev and also npm installs will need to be done on a per-example page.

In the future, if we start having shared libraries, and we want to use those in the examples, firstly we will need to publish those to npm and only then we will be able to add those as dependencies to the package.json in the required examples.

We would also need to turn the next.config.js and tailwind.config.js files back to their default versions(without the stuff nx adds to them to make them work) but thats besides the point.

What is Vercel doing?

If you are not convinced by my appeals, thats fine, but take a look into the Vercel Next.JS Repo if you take a closer look at its content, this is in fact a monorepo, the only difference is that its using Turbo, but if you look at their examples folder, none of the examples are actually part of the monorepo, they are stored there, but if you want to use them you will need to go to the specific file, run npm install and then npm run dev which is exactly what im suggesting.

Some extra thoughts

Most of the stuff written here was thought out after i talked with Anu on friday and he told me how the previous experience of PortalJS as a components library was not of the best. And that now(at least on a first moment) the focus of this repo should be on having a list of comprehensible examples that can be used to quickly create a data portal.
Most libraries use monorepos because they allow the developers to quickly see the changes, without requiring them to first publish stuff to npm(something that you usually do only on a release cycle), in the future we might have a separate folder with some separate apps that fulfill this porpuse, but if the objective of the examples folder is to have some templates which some random data analyst can use to publish some data to the outside world, then i believe that using a monorepo tool like nx will only confuse and confuse that person.

[anuveyatsu]

@luccasmmg thanks for writing this 👍🏼 I think examples are similar to actual projects (eg, a government creating their own data portal) and it should be outside of monorepo so that it can be easily deployed/developed etc. We should have:

• /apps/ or similar directory where we develop actual apps. Once they are ready for release, they get exported into /examples/.
• /packages/ which we already have and can be used for re-usable parts.
• /examples/ which is exported apps and deployed to "production".

graph TD

developer --> apps
developer --> packages

apps -.Export.-> examples
apps -.- packages

Loading

cc @demenech @rufuspollock

[rufuspollock]

@luccasmmg i think we get some misunderstanding here about how things get built ...

• examples should be depending on portaljs as a library
• you wouldn't clone the portaljs full repo to install an example, you'd do npx create-next-app https://github.com/datopian/portaljs/examples/my-example or similar (this is what i requested the other day for the super simple example right ...?)
• i think this is largely in agreement with what you are saying in "Whats my proposal" (and what nextjs do)
• for right now, the way i would obsessively focus on have one simple example work in a really simple way (and for others we just do what is easiest until we have a good process for that super simple example when we can then expand that).

[rufuspollock]

I would generate job stories and personas a bit here ...

As a quick start (i can write more later)

• I'm new to this area and want to know what a data portal is
• I want to check out this framework and its features and whether worth looking further
• I want to get started immediately trying this out and building something => take me straight to a tutorial or quickstart example
• I want to see the examples that i could use so i pick one most relevant to my needs right now

[rufuspollock]

From today we converged on two core stories:

• Get started
• What and why

NB: we have one major question that informs all the docs and site

• Is our default approach about a catalog or presenting a single dataset (extending later to more) ✅2023-04-25 most natural is to start with a single dataset (it is still a portal to that data) and then evolve to a catalog

Get started

When I arrive on the site I want to know how to get started so that i can build something and find out how this works (and see if it solves my need, plus play around!)

e.g.

• Prerequisites e.g. install node
• install the default example
  • showcase a single data file and README (maybe with separate index file)
  • And then people play with it
• Then add another datasets / data file
• Create a catalog ...

What and Why

When I arrive I want to understand what this is (what i can do!) and why it is useful/good (vs other things) so that I can decide to investigate further

• What can i do?
  • Present my data (individual or catalog or just catalog?)
  • ...
• What are the top features / differentiators of this product?