Skip to content

DOC-5282 RDI: restructure config/job file details #1683

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Jun 16, 2025

Conversation

andy-stark-redis
Copy link
Contributor

@andy-stark-redis andy-stark-redis commented Jun 10, 2025

DOC-5282

This is the first of a few changes I'm planning to make to improve the structure of the RDI docs. It incorporates some of the suggestions from a ChatGPT review of the pipeline config page plus some other changes that I think will be helpful.

In particular:

  • The single, long pipeline config page is broken up into shorter pages for overview, config.yaml, and job files.
  • The data pipeline overview and the job file overview are now in the landing pages for their respective folders. This avoids the boring landing pages with just a list of pages in the folder (we've found that users don't use those pages very effectively).
  • The "Transformation examples" folder is now called "Job files" (more consistent and descriptive) and the job file overview is now the index page for this folder.

There is also some restructuring within pages, but the technical content has not changed.

More changes to come, but I thought I'd get some feedback at this stage and check everyone is happy with what I've done so far. All suggestions welcome :-)

Copy link
Contributor

github-actions bot commented Jun 10, 2025

@andy-stark-redis andy-stark-redis changed the title DOC-5282 RRDI: restructure config/job file details DOC-5282 RDI: restructure config/job file details Jun 10, 2025
@andy-stark-redis andy-stark-redis self-assigned this Jun 10, 2025
@andy-stark-redis andy-stark-redis marked this pull request as ready for review June 10, 2025 12:42
Copy link
Collaborator

@dwdougherty dwdougherty left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Language LGTM. Well, that was a full day's work. See ya tomorrow. 😂

@andy-stark-redis
Copy link
Contributor Author

Thanks @dwdougherty ! Yeah, see you tomorrow ;-)

Copy link
Contributor

@ilianiliev-redis ilianiliev-redis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just a small comment for a missing config parameter.


The `processors` section configures the behavior of the pipeline. The [example](#example)
configuration above contains the following properties:

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason for omitting target_data_type both here and in the example above?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason for omitting target_data_type both here and in the example above?

@ilianiliev-redis I'm not aware of any reason - target_data_type just wasn't in the original example, but I'll add it in now.

@andy-stark-redis andy-stark-redis merged commit c3df7c3 into main Jun 16, 2025
5 checks passed
@andy-stark-redis andy-stark-redis deleted the DOC-5282-rdi-restructure-job-details branch June 16, 2025 10:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants