Generate partials instead of pages for the individual Camel bits

[ppalaga]

In #1777 I adopted a quick and easy approach to store the data necessary for the components page

The data for the entries is stored in .adoc filed under components directory

The fact that those adoc files actually get transformed to HTML pages that are accessible via HTTP could be seen as a drawback.

I wonder whether there is a better way to store the data. I'd have no problem storing it as Java properties, json, or yaml, but asciidoctor-antora-indexer AsciiDoc extension would have to able to parse that format.

Any ideas @djencks ?

[djencks]

I also wrote https://gitlab.com/djencks/asciidoctor-jsonpath which is like the indexer but gets its data from a json document rather than the content catalog, perhaps that would help.

[ppalaga]

Sounds very promising! Do you happen to have an example at hand?

[ppalaga]

Do you happen to have an example at hand?

Sorry for the noise, I see some examples in https://gitlab.com/djencks/asciidoctor-jsonpath

[djencks]

A couple of comments… - I recommend generating the json file into modules/ROOT/examples - I thought I had more examples handy, and don’t have time to construct any right now, but if you create the json file I’m happy to work on the use of the extension to generate the page.

…

On Sep 15, 2020, at 8:32 AM, Peter Palaga ***@***.***> wrote: Do you happen to have an example at hand? Sorry for the noise, I see some examples in https://gitlab.com/djencks/asciidoctor-jsonpath <https://gitlab.com/djencks/asciidoctor-jsonpath> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1795 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAELDXRLA65PWGBDXRFVUU3SF6CJZANCNFSM4RNIINJQ>.

[ppalaga]

I recommend generating the json file into modules/ROOT/examples

Hm... why examples? The data is not examples. I think catalog or similar would describe better what's inside.

I thought I had more examples handy, and don’t have time to construct any right now, but if you create the json file

Yes, I can do that, but when you speak about a json file in sigular, does that mean that there has to be a single file? I'd prefer having one JSON file per table row. That will allow us to avoid merge conflicts when people are going to work on various extensions in parallel.

[djencks]

On Sep 16, 2020, at 12:00 AM, Peter Palaga ***@***.***> wrote: I recommend generating the json file into modules/ROOT/examples Hm... why examples? The data is not examples. I think catalog or similar would describe better what's inside.

Probably I wasn’t clear enough… the best family for this sort of file is ‘example’. You can name the file anything, e.g. `modules/ROOT/examples/catalog.json`

I thought I had more examples handy, and don’t have time to construct any right now, but if you create the json file Yes, I can do that, but when you speak about a json file in sigular, does that mean that there has to be a single file? I'd prefer having one JSON file per table row. That will allow us to avoid merge conflicts when people are going to work on various extensions in parallel.

At the moment, there needs to be only one json file. I can consider querying the content catalog for multiple files, but I’m not quite convinced it’s a good idea yet, and it would certainly be a significant amount of work. Maybe there’s some adoc file from which the information can be extracted? Perhaps the component page in the main camel docs? Or if there is at most one component per extension, perhaps the info could go into the extension pages?

…

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1795 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAELDXWWKGLR4D4SNGMSS3LSGBO7XANCNFSM4RNIINJQ>.

[ppalaga]

Maybe there’s some adoc file from which the information can be extracted? Perhaps the component page in the main camel docs?

That would be ideal but I have no idea yet how it could be implemented. Camel Quarkus depends on Camel so the info cannot go to the camel git repo (we'd create a dependency cycle in that way). Perhaps there is some way how we could enhance the Camel component .adoc files in camel-website after they are cloned from the Camel repo?

Or if there is at most one component per extension, perhaps the info could go into the extension pages?

No, unfortunately, one extension can contain multiple components, languages, data formats, etc.

[djencks]

It might be possible to get the quarkus information into the main camel component pages by:

• moving the component pages generated for c-q into the c-q modules/ROOT/partials directory (so they don't end up as published pages} and just generating the attributes (no header)
• including these in the main camel components pages in the header. I think it would look something like

:last-existing-header-attribute: foo
include::{cq-version}@camel-quarkus:ROOT:partial$components/activemq.adoc[]

for e.g. the activemq component.

[djencks]

I got this idea to (mostly) work, and I think it may show up a problem with the current implementation.

My work is on partial-components branch in https://github.com/djencks/camel-quarkus.git and https://github.com/djencks/camel.git. The local antora-playbook.yml in the camel-quarkus repo is set to build if the camel-quarkus and camel repos are checked out next to one another.

The current components page is at /docs/target/site/camel-quarkus/latest/reference/components-2.html. I think the list of components is incomplete. For instance, there's only one atomix component listed whereas in my implementation there are six, which corresponds to the atomix extension.

I also altered the extension page generation to use xrefs to the base camel component pages rather than https: links. This change should be made no matter what happens with this experiment.

There's currently a problem in that there are a ton of missing include warnings for components that are not yet in camel-quarkus. This can be fixed by either:

• generating a partial in camel-quarkus for every component/databinding/language/other whether or not it's supported yet; the partials for unsupported would be blank
• waiting for https://gitlab.com/antora/antora/-/issues/678.

If this approach seems like a good idea I can presumably do the dataformats/languages/other pages in the same way.

[djencks]

I thought that the warning for missing includes was included in the page with the missing include, but it turns out that since the include is in the header the warning is not rendered. So the console output is very noisy but the site looks fine. If we're willing to live with noisy console output for a while, we could implement this now.

[ppalaga]

I like the idea to use partials instead of pages.

At the same time, I do not like that building Camel Quarkus Antora docs would require Camel source tree being checked out at a specific location, only to have a set of pages to include our partials into. That sounds unnecessarily complicated. The data stored in the partials comes from Camel Quarkus only and any pages aggregating that data should not need Camel for building.

I have looked into your proposed implementation and I came across the question whether your macros should not be taught to operate on partials? That would solve the problem in the most elegant way, IMO. I tried to do that in #1814, esp. in this commit: e599c94 and also here https://github.com/apache/camel-quarkus/pull/1814/files#diff-a5f808e798bf3612d1e44875e6bef126R13

It does what I expect it to do but I admit my code is probably not the prettiest one because I am a JavaScript noob. Anyway, what do you think about the idea?

[djencks]

The quarkus partial build wouldn't actually need a local copy of the main camel repo once this idea was accepted/committed; it could use the remote repo. The POC does need a local copy because the modified main component adoc pages don't have a remote location yet.

The indexer plugin relies on Antora's 2-pass rendering, where the first pass extracts the header attributes and attaches them as a map to the page object. My plugin then uses this map for it's queries. Since partials are never rendered separately, this map never exists for them.

I see your proposal to have the plugin load the partial, but so far I think that is architecturally inappropriate. If you do go in that direction, I'd advise caching the loaded asciidoc object on the partial, since there are likely to be a lot of passes through the set of partials. It would also be best to set the parse_header_only option in case the partial happens to have some actual content.

[ppalaga]

Thanks for the feedback. I also think that, in its essence, my proposal is a hack using partials for something they were not designed for.

The quarkus partial build wouldn't actually need a local copy of the main camel repo once this idea was accepted/committed; it could use the remote repo.

That sounds promising. No manual local setup is good for both devs and CI. I'd like to check how much time overhead will this bring before I fully commit to the idea.

[ppalaga]

I'd like to check how much time overhead will this bring

I measured the yarn build duration in the following scenarios:

1. Current Camel Quarkus master: ~12s
2. Adding the following snippet to antora-playbook.yml with Camel repo cached locally under ~/.cache/antora/content/2/:

  - url: git@github.com:apache/camel.git
    branches: master
    start_paths:
      # manual
      - docs/user-manual
      # eip
      - core/camel-core-engine/src/main/docs
      # main components doc
      - docs/components

46s

3. Same as 2. but without Camel repo cached locally under ~/.cache/antora/content/2/: 2m 54s

These numbers make me hesitant.

There is one more issue with this setup: the yarn antora --generator @antora/xref-validator antora-playbook.yml fails with this setup. ATM, it is probably caused by some missing config on the Camel Quarkus side, but generally, I think we do not want to allow the possibility that broken xrefs in Camel docs can make our CI fail. What do others think?

[djencks]

I’m not sure my suggestion is the best way forward, but I think there are more choices that use it. For one thing, you are pulling in the entire camel master site, rather than just components, which is going to increase the build time. It’s not actually necessary for the author-mode camel-quarkus build to generate the table for component-to-extension links; this would allow the build to use only the local sources. The contents of this table are not really susceptible to text changes, so it might be acceptable to build the whole site to check on the table and counts. Or, there could be two camel-quarkus author mode playbooks, one quarks-only and the other including the main components component. I don’t think it’s appropriate to fail a partial site build based on broken xrefs. This would mean that one part of the documentation can’t refer to another part. If there’s an agreed upon principle that there shouldn’t be cross-(antora-)component xrefs, it needs to be clearly stated somewhere :-) The include in the main camel component docs means that it would be easy to include a generated line, either “this component is in the camel-quarkus XXX extension” or “this component is not available in camel-quarkus”. Using one or multiple json files in camel-quarkus would not enable this. I ran some builds on my laptop and the results are inconsistent enough that I don’t really trust them, but I didn’t see a convincing increase in build time between the current whole-site build and my proposal.

…

On Sep 22, 2020, at 4:18 AM, Peter Palaga ***@***.***> wrote: I'd like to check how much time overhead will this bring I measured the yarn build duration in the following scenarios: Current Camel Quarkus master: ~12s Adding the following snippet to antora-playbook.yml with Camel repo cached locally under ~/.cache/antora/content/2/: - url: ***@***.***:apache/camel.git branches: master start_paths: # manual - docs/user-manual # eip - core/camel-core-engine/src/main/docs # main components doc - docs/components 46s Same as 2. but without Camel repo cached locally under ~/.cache/antora/content/2/: 2m 54s These numbers make me hesitant. There is one more issue with this setup: the yarn antora --generator @antora/xref-validator antora-playbook.yml fails with this setup. ATM, it is probably caused by some missing config on the Camel Quarkus side, but generally, I think we do not want to allow the possibility that broken xrefs in Camel docs can make our CI fail. What do others think? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1795 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAELDXWMAITQB3D7H7UWW5DSHCBZDANCNFSM4RNIINJQ>.

[ppalaga]

there could be two camel-quarkus author mode playbooks, one quarks-only and the other including the main components component.

I went with this option after discussing it with @jamesnetherton . I have updated #1814

I don’t think it’s appropriate to fail a partial site build based on broken xrefs. This would mean that one part of the documentation can’t refer to another part. If there’s an agreed upon principle that there shouldn’t be cross-(antora-)component xrefs, it needs to be clearly stated somewhere :-)

That's our current modus operandi Changing it would require a separate discussion.

The include in the main camel component docs means that it would be easy to include a generated line, either “this component is in the camel-quarkus XXX extension” or “this component is not available in camel-quarkus”.

Yeah, that's an important advantage of using partials (thanks again for proposing it!). I am going to file a PR against Camel once #1814 is merged.

[djencks]

BTW, the noise from 'include not found' should disappear with antora 3.0-alpha-1 which should be released any day now. I don't think there are any destabilizing changes in it.

The cq dataformats, languages, etc pages ought to be able to also use this idea, right?

[ppalaga]

The cq dataformats, languages, etc pages ought to be able to also use this idea, right?

Yes, of course, my update of #1814 does that for all.