Reference Documentation issues

[meltsufin]

• Currently 4.x refdoc generation is failing #2371
• 4.x ref documentation is not listed in the README (resolved in Reference docs version update issues #2385)
• We shouldn't publish refdocs for SNAPSHOT versions #2372
• Latest should be the latest released version of the highest major version (resolved in Reference docs version update issues #2385)
• Auto-update the links to latest refdocs for each major version in the README (resolved in Reference docs version update issues #2385)
• We should check how Spring currently generates their refdocs and see if we need to update (answered in b/294391973)

[suztomo]

Memo. This is the document https://googlecloudplatform.github.io/spring-cloud-gcp/reference/html/index.html. This is linked from this section in README.md:

[alicejli]

Pasting some investigation notes on the failing 4.x refdoc job:

1. First failure is for 4.6.0 (https://github.com/GoogleCloudPlatform/spring-cloud-gcp/actions/runs/5657280928/job/15592917305).

Suspicious logs:

+ for f in ${ROOT_FOLDER}/docs/target/generated-docs/*
+ file='*'
+ copy_docs_for_branch 1.1.x 1.2.x 2.0.0 2.0.0-RC2 2.0.1 2.0.10 2.0.11 2.0.2 2.0.3 2.0.4 2.0.5 2.0.6 2.0.7 2.0.8 2.0.9 3.0.0 3.1.0 3.2.0 3.2.1 3.3.0 3.4.0 3.4.1 3.4.2 3.4.3 3.4.4 3.4.5 3.4.6 3.4.7 3.4.8 3.4.9 3.5.0 3.5.1 3.5.2 3.5.3 3.5.4 3.5.5 3.6.0 3.6.1 3.6.2 4.0.0 4.1.0 4.1.1 4.1.2 4.1.3 4.1.4 4.2.0 4.3.0 4.3.1 4.4.0 4.5.0 4.5.1 4.6.0 Guardfile docs favicon.ico ghpages.sh index.html reference spring-cloud-gcp-autoconfigure spring-cloud-gcp-bigquery spring-cloud-gcp-cloudfoundry spring-cloud-gcp-core spring-cloud-gcp-data-datastore spring-cloud-gcp-data-firestore spring-cloud-gcp-data-spanner spring-cloud-gcp-kms spring-cloud-gcp-logging spring-cloud-gcp-pubsub spring-cloud-gcp-pubsub-stream-binder spring-cloud-gcp-samples spring-cloud-gcp-secretmanager spring-cloud-gcp-security-firebase spring-cloud-gcp-security-iap spring-cloud-gcp-starters spring-cloud-gcp-storage spring-cloud-gcp-vision spring-cloud-generator spring-cloud-previews target ./4.6.0
+ local file=1.1.x
+ local destination=1.2.x
+ git ls-files -i -o --exclude-standard --directory
+ grep -q '^1.1.x$'
+ [[ 1.1.x == \$\{\d\o\c\s\.\m\a\i\n\}\.\h\t\m\l ]]
+ [[ 1.1.x == \s\p\r\i\n\g\-\c\l\o\u\d\-\g\c\p\.\h\t\m\l ]]
+ cp -rf '/home/runner/work/spring-cloud-gcp/spring-cloud-gcp/docs/target/generated-docs/*' 1.2.x
cp: cannot stat '/home/runner/work/spring-cloud-gcp/spring-cloud-gcp/docs/target/generated-docs/*': No such file or directory

2. v4.7.0 actually fails for a different reason, during the mvn install phase before the updateDocs step:
   https://github.com/GoogleCloudPlatform/spring-cloud-gcp/actions/runs/5695458239/job/15569343444

Error:  Could not acquire lock(s)
Error:  java.lang.IllegalStateException: Could not acquire lock(s)
Error:  Could not acquire lock(s)
Error:  
Error:  To see the full stack trace of the errors, re-run Maven with the -e switch.
Error:  Re-run Maven using the -X switch to enable full debug logging.
Error: Process completed with exit code 1.

3. v4.7.2 fails for the same reason as 4.6.0, so I expect v4.7.0's failure to be a flake. Interestingly, v3.6.3 succeeds (released during same cycle as v4.7.2), but it uses a slightly different workflow file.

Key differences are in lines 24 and 32. v3.x uses Java 11 and run: echo "::set-output name=date::$(date +'%Y-%m-%d' --utc)", whereas v4.x uses Java 17 and run: echo "date=$(date +'%Y-%m-%d' --utc)" >> $GITHUB_OUTPUT.

These differences are minimal and shouldn't be the cause of failure, but just noting here for reference.

---

Something odd is happening with the 4.x workflow where it is copying all versions instead of just the specified one. Will dig into that some more.

[alicejli]

Have narrowed down the issue that the 4.x workflow where when it's running the copy_docs_for_current_version() function, it does not find anything so the file variable is set to *, suggesting that there are no files or directories in ${ROOT_FOLDER}/docs/target/generated-docs/ when that loop is executed. On a successful run, file is set to reference.

Further evidence that it's this function is that the script succeeds when run locally using the copy_docs_for_provided_version() function.

Relevant failing log lines:

+ [[ -z 4.7.2 ]]
+ copy_docs_for_provided_version
+ local FOLDER=./4.7.2
+ mkdir -p ./4.7.2
+ echo -e 'Current tag is [v4.7.2] Will copy the current docs to the [./4.7.2] folder'
Current tag is [v4.7.2] Will copy the current docs to the [./4.7.2] folder
+ for f in ${ROOT_FOLDER}/docs/target/generated-docs/*
+ file='*'
+ copy_docs_for_branch 1.1.x 1.2.x 2.0.0 2.0.0-RC2 2.0.1 2.0.10 2.0.11 2.0.2 2.0.3 2.0.4 2.0.5 2.0.6 2.0.7 2.0.8 2.0.9 3.0.0 3.1.0 3.2.0 3.2.1 3.3.0 3.4.0 3.4.1 3.4.2 3.4.3 3.4.4 3.4.5 3.4.6 3.4.7 3.4.8 3.4.9 3.5.0 3.5.1 3.5.2 3.5.3 3.5.4 3.5.5 3.6.0 3.6.1 3.6.2 3.6.3 4.0.0 4.1.0 4.1.1 4.1.2 4.1.3 4.1.4 4.2.0 4.3.0 4.3.1 4.4.0 4.5.0 4.5.1 4.6.0 4.7.0 4.7.1 4.7.2 Guardfile docs favicon.ico ghpages.sh index.html reference spring-cloud-gcp-autoconfigure spring-cloud-gcp-bigquery spring-cloud-gcp-cloudfoundry spring-cloud-gcp-core spring-cloud-gcp-data-datastore spring-cloud-gcp-data-firestore spring-cloud-gcp-data-spanner spring-cloud-gcp-kms spring-cloud-gcp-logging spring-cloud-gcp-pubsub spring-cloud-gcp-pubsub-stream-binder spring-cloud-gcp-samples spring-cloud-gcp-secretmanager spring-cloud-gcp-security-firebase spring-cloud-gcp-security-iap spring-cloud-gcp-starters spring-cloud-gcp-storage spring-cloud-gcp-vision spring-cloud-generator spring-cloud-previews spring-cloud-spanner-spring-data-r2dbc target ./4.7.2

Passing log lines:

+ [[ -z 3.6.3 ]]
+ copy_docs_for_provided_version
+ local FOLDER=./3.6.3
+ mkdir -p ./3.6.3
Current tag is [v3.6.3] Will copy the current docs to the [./3.6.3] folder
+ echo -e 'Current tag is [v3.6.3] Will copy the current docs to the [./3.6.3] folder'
+ for f in ${ROOT_FOLDER}/docs/target/generated-docs/*
+ file=reference
+ copy_docs_for_branch reference ./3.6.3
+ local file=reference
+ local destination=./3.6.3

[alicejli]

The issue appears to be within this function within ghpages.sh:

function build_docs_if_applicable() {
    if [[ "${BUILD}" == "yes" ]] ; then
        echo "Building docs in directory: $(pwd)"
        ./mvnw clean install -P docs -pl docs -DskipTests -X
    fi

where it is not building the target/generated-docs directory.

Have tentatively narrowed down the issue to the two plugins within the <docs> profile:

					<plugin>
						<groupId>org.asciidoctor</groupId>
						<artifactId>asciidoctor-maven-plugin</artifactId>
					</plugin>
					<plugin>
						<groupId>org.apache.maven.plugins</groupId>
						<artifactId>maven-antrun-plugin</artifactId>
					</plugin>

It appears that in a successful run, those two plugins are downloaded from Maven central as part of building the docs and thus generate documentation within the target/generated-docs directory.
(e.g. https://pipelines.actions.githubusercontent.com/serviceHosts/d601a7fb-4776-45f9-afde-f90cba8d1edc/_apis/pipelines/1/runs/21163/signedlogcontent/2?urlExpires=2023-08-17T13%3A37%3A46.3153857Z&urlSigningMethod=HMACV1&urlSignature=pQoFgcefU604oJojLr2HFSIV3Jh4lh%2FvgVwsfteV5VY%3D)

In a failed run, these two plugins are not downloaded. Still digging into why.

[suztomo]

@alicejli Check whether "./mvnw clean install -P docs -pl docs -DskipTests -X" succeeded or not.

[alicejli]

@alicejli Check whether "./mvnw clean install -P docs -pl docs -DskipTests -X" succeeded or not.

Yes, this is the part that only partially succeeds in the failed runs. It says it builds successfully, but it does not build the target/generated-docs directory in the failed runs.

(sorry I meant to reply but I think I updated your original comment by accident).

[alicejli]

Not making too much progress figuring out what's causing the build discrepancy, so going to pivot to looking at how other Spring projects are generating their reference docs and seeing how/if we can use their workflow.

For reference to dig into later:

https://github.com/spring-projects/spring-framework/blob/main/.github/workflows/deploy-docs.yml
https://github.com/spring-projects/spring-framework/blob/main/framework-docs/framework-docs.gradle

[JoeWang1127]

All issues are resolved.