Migrate to updated API reference generator

[sftim]

This is a Feature Request

What would you like to be added
k-sigs/reference-docs has a recently-added gen-resourcesdocs generator. Credit to @feloy for hard work on this.

Let's switch to using that generator [Preview] and make it the default.

This entails:

• understanding how to preserve existing incoming hyperlinks
• planning to update existing pages that link to API reference documentation
• testing the new reference generation
• revising the documentation on how to publish API reference information
• making any shortcode or other changes needed based on the planning
• switching over
• logging work (GitHub issues) to update existing pages as planned

Why is this needed

The existing API documentation (example) uses a different style from the rest of the website, is harder to navigate than it could be, and is also difficult to localize.

Whilst it has served the Kubernetes project well, with the arrival of an alternative generation option it is time to look at switching.

Comments

/kind feature

/area web-development
(sic)

also
/language en

Relevant to issue #24441 and to kubernetes/kubernetes#19680

[kbhawkey]

Is there a suggested timeline for making the proposed changes?
There are quite a few (quick search found ~828) links that refer to sections in the single page reference.
I suspect that the new shortcode could help create replacement links to the appropriate resources?

[sftim]

My sketch for part of an idea: replace the current API reference with a page that uses JavaScript to calculate a redirect to the correct URL (based on the fragment identifier and some other information).
That JavaScript could either XMLHTTPRequest the real API reference pages, or use information available at site build time.

That would allow us to update existing inbound links gradually, including across the 12 downstream localizations. It would I think take that clean up away from the critical path for migrating, too.

For a long term migration away from that we could make a whole new redirection service eg https://api.k8s.io/example.k8s.io/v1beta1/KindOfObjectGoesHere - but we don't need to do that immediately, or even at all.

[kbhawkey]

I was thinking that the root path for the API reference pages could serve as the new API reference URL (add a redirect)?
Are you suggesting to dynamically redirect every path off this base URL?

Perhaps the static assets and single HTML page can remain until every localization has updated their files.
The en content could start to use the shortcode to reference the resource pages (if every API ref link can be resolved by using the new shortcode).

[tengqm]

Should we consider the coexistence of the two API reference representations for one or two release cycles?
It is a common practice for websites to serve two flavors of some contents till a complete switch over proves to be welcomed.

[sftim]

one or two release cycles?

I really would prefer not to break these URIs because they are referenced in code and documentation all over the web. That's why I really like the idea of a long lived helper that redirects to the new home.
We could redirect everything to one place but that's not much different from a 404 page; if someone has a link to the API for ConfigMap, then what they need to see is the API reference for ConfigMap. Otherwise they may as well ask a search engine.
(which right now will probably serve https://docs.openshift.com/container-platform/3.7/rest_api/api/v1.ConfigMap.html as the top result, FWIW)

[kbhawkey]

Should we consider the coexistence of the two API reference representations for one or two release cycles?
It is a common practice for websites to serve two flavors of some contents till a complete switch over proves to be welcomed.

Yes, that seems reasonable. I have not thought about the timeline for completely switching over.
I suggest to decide on a timeline and set up a date to communicate this decision to the community (how and when these changes will finally happen). There may be issues logged. I see a short list of things to do:

1. Update the intra-site links to refer to the new API reference pages.
2. Determine how to redirect external requests for the current API reference sections to the new individual reference pages.
   There will not be a versioned or latest version of a single reference to link to.
3. Make sure the Markdown API generator tool can be rebuilt, the build instructions are clear (both repos), the tool can consistently generate new API Markdown pages, and these generated pages can be added and build with the website.

[sftim]

Another option that we could consider:

1. Replace all links (across localizations as well, if feasible) with a new shortcode. At this point the shortcode renders as a link to the old / existing API reference.
2. Draft a revised shortcode that links to the new API reference.
3. Update the “old” API generator so that the rendered API reference hyperlinks to the new one.
   • This doesn't have to be perfect.
4. Publish the new API reference and switch the the updated shortcode.
5. Update the old API reference to include links to the new API reference.

[feloy]

Should we consider the coexistence of the two API reference representations for one or two release cycles?
It is a common practice for websites to serve two flavors of some contents till a complete switch over proves to be welcomed.

Yes, that seems reasonable. I have not thought about the timeline for completely switching over.
I suggest to decide on a timeline and set up a date to communicate this decision to the community (how and when these changes will finally happen). There may be issues logged. I see a short list of things to do:

1. Update the intra-site links to refer to the new API reference pages.

I'm updating the intra-links. I've changed the links in the Concepts and Tasks, I still have to change the links in the Reference section.

The changes are visibile in the PR #23294

2. Determine how to redirect external requests for the current API reference sections to the new individual reference pages.
   There will not be a versioned or latest version of a single reference to link to.
3. Make sure the Markdown API generator tool can be rebuilt, the build instructions are clear (both repos), the tool can consistently generate new API Markdown pages, and these generated pages can be added and build with the website.

[sftim]

PR #23294 has merged. I'll approach Kubernetes SIG Architecture to co-ordinate communications and other tasks about deprecating and moving away from the existing API documentation.

SIG Docs' preference is to shift away as part of a Kubernetes release (eg v1.21). One reason for this is that each release so far has broken the old API reference URLs anyway.

[sftim]

@reylejano are you happy to have this tracked against v1.21?

[neolit123]

The existing API documentation (example uses a different style from the rest of the website, is harder to navigate than it could be, and is also difficult to localize.

one good thing about the old format is that it was easier to search the whole API reference using a browser Ctrl+F (search), since it was all on the same page.

[sftim]

Perhaps there's a way to improve the search(ability). For example, add an index to the foot of each API reference page.

[liggitt]

Thanks for the update! I especially like how much better it renders on mobile devices.

Here is some initial feedback on the organization and rendering:

1. The primary organization of the API is by API group → version → resource/kind. From https://kubernetes.io/docs/reference/kubernetes-api/, it is hard to know where documentation for types in various group/versions are, and I had to guess about which category a given type was placed into. Can we make the API group+version explicit in the index/navigational structure or overview pages?
2. The URLs include the categorizations (".../services-resources/...", ".../workloads-resources/...", etc) which are somewhat arbitrary. Making the URLs based on the API group/version/kind will produce more stable URLs (makes it easier to address item 7 later in a follow-up).
3. The representation of kinds with multiple versions available is confusing/missing:
   • For example, https://kubernetes.io/docs/reference/kubernetes-api/workloads-resources/ lists "HorizontalPodAutoscaler" and "HorizontalPodAutoscaler v2beta2". The first is implicitly in the autoscaling/v1 API group/version, but that isn't clear. We need to make it very clear what version the documentation is for (someone following the v1 HorizontalPodAutoscaler docs will not successfully construct a v2beta2 HorizontalPodAutoscaler manifest)
   • Documentation for some beta types seems to be missing altogether (Ingress v1beta1, for example).
4. Some pages mix top-level types and nested type definitions. For example, https://kubernetes.io/docs/reference/kubernetes-api/workloads-resources/ has Pod, ReplicaSet, Deployment, etc, which can be created as objects, but also has Container, which only exists nested inside top-level types. I expected only top-level types to be linked from the overview pages.
5. Grouping the "common parameters" page by operation would be helpful (parameters for get, list, watch, create, update, patch, delete, deletecollection operations, etc). Even if this duplicated some parameters (like dryRun, fieldManager, etc), it would make it easier to understand which bucket of parameters could be used in a given request).
6. It's unclear what causes a type to appear in https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/ ... these types I would have expected to have been on specific pages: DownwardAPIVolumeFile, ExecAction, HTTPGetAction, JSONSchemaProps, KeyToPath, NodeAffinity, NodeSelectorRequirement, PodAffinity, PodAntiAffinity, ResourceFieldSelector, TCPSocketAction
7. Some of the type categorizations seemed odd to me... the lines between workload (pod, deployment, etc) + node (node, runtimeclass, etc) + scheduling (priorityclass, binding) are pretty blurry. I'd suggest getting sig-apps / sig-scheduling / sig-node involved to work on that organization. If we make the URLs not depend on the categorizations (see suggestion 2 above), it will be lower impact to switch up the categorizations in a follow-up.

[reylejano]

@reylejano are you happy to have this tracked against v1.21?

@sftim Yes, I'm comfortable with this being tracked against v1.21. I want to review and capture any process changes for the release for the docs role handbook timeline.

[sftim]

We also need to add a redirect so that https://kubernetes.io/docs/reference/kubernetes-api/labels-annotations-taints/ points to the new page at https://kubernetes.io/docs/reference/labels-annotations-taints/
…as mentioned in #24661 (comment)

[kbhawkey]

Hi. As part of the migration, I'd like to understand the API categories and work on resolving some of the questions about navigation.

• Do the categories server only to group the resource pages in the left hand navigation tree; Or,
• Do the categories provide a clear pathway or hint to locate a resource?
• There is a question about missing beta types. Log a separate issue?
  Documentation for some beta types seems to be missing altogether (Ingress v1beta1, for example).

Re: Comment:

The primary organization of the API is by API group → version → resource/kind. From https://kubernetes.io/docs/reference/kubernetes-api/, it is hard to know where documentation for types in various group/versions are, and I had to guess about which category a given type was placed into. Can we make the API group+version explicit in the index/navigational structure or overview pages?

Re: More comments:

The URLs include the categorizations (".../services-resources/...", ".../workloads-resources/...", etc) which are somewhat arbitrary. Making the URLs based on the API group/version/kind will produce more stable URLs (makes it easier to address item 7 later in a follow-up).
The representation of kinds with multiple versions available is confusing/missing:

    For example, https://kubernetes.io/docs/reference/kubernetes-api/workloads-resources/ lists "HorizontalPodAutoscaler" and "HorizontalPodAutoscaler v2beta2". The first is implicitly in the autoscaling/v1 API group/version, but that isn't clear. We need to make it very clear what version the documentation is for (someone following the v1 HorizontalPodAutoscaler docs will not successfully construct a v2beta2 HorizontalPodAutoscaler manifest)
    Documentation for some beta types seems to be missing altogether (Ingress v1beta1, for example).

[sftim]

I need to talk to SIG Architecture about stability (or not) of the URLs here.

[neolit123]

The URLs include the categorizations (".../services-resources/...", ".../workloads-resources/...", etc) which are somewhat arbitrary. Making the URLs based on the API group/version/kind will produce more stable URLs (makes it easier to address item 7 later in a follow-up).

i think this is rather important.
categorizations are a nice to have and can be implement with labels / filters ALA github.

[sftim]

There is a question about missing beta types. Log a separate issue?

I agree, that sounds like a separate issue.

[kbhawkey]

Catching up on this discussion. Can you provide more details on:
We can separate out the “presentation” grouping of resources (eg workloads, policy, etc) from the URL of the page about a particular resource in the API - and we should do so that we have stable URIs for the API reference pages.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sftim]

/reopen
/lifecycle frozen

[k8s-ci-robot]

@sftim: Reopened this issue.

In response to this:

/reopen
/lifecycle frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sftim]

I'm happy to keep working on this as time allows
/assign

[sftim]

/unassign

I haven't made much progress - when I get chunks of time, I'm spending those instead on other tasks; notably, https://kubernetes.io/docs/concepts/services-networking/service/

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

[sftim]

/triage accepted

Still useful to provide a consistent reference.

[sftim]

/wg api-expression

What I'd like to see:

• we make sure that the new-style API reference uses consistent URLs (eg, put the API group in the URL path)
  Right now, the policy-resources path of https://kubernetes.io/docs/reference/kubernetes-api/policy-resources/resource-quota-v1/ is arbitrary and I'm wary of baking opinions into into the site indefinitely
• we improve https://kubernetes.io/docs/reference/kubernetes-api/ to list all the API types on one page, perhaps with some JavaScript and a way to filter APIs (this is where metadata can help; for example filtering just for APIs limited to policy mechanisms, or for APIs relevant to security)
• we make it easy to discover the equivalent reference page for other releases; see https://kubernetes.io/docs/home/supported-doc-versions/ for an example of a page that solves this already
• we make sure that all the information in the legacy-style reference is included in the new-style reference pages
  For example, https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/ lists the internal verbs that an API supports and, for each one, gives you a hint about how to make an API request using HTTP.