Execute all checkstyle samples in each PR execution

[romani]

we almost missed breaking compatibility at #12392 (comment)
we need to run all samples in our CI on each PR.

we already did this for ant:

checkstyle/.semaphore/semaphore.yml

Line 54 in 8a1f198

- .ci/no-exception-test.sh no-exception-samples-ant

checkstyle/.ci/no-exception-test.sh

Line 182 in d3a5f9a

no-exception-samples-ant)

UPDATE: we leaked bad version to public due to this issue is not done, gradle build problem - #14123

[nrmancuso]

we need to run all samples in our CI on each PR.

Can we compile a list of samples?

[rnveach]

https://github.com/sevntu-checkstyle/checkstyle-samples
Every folder you see besides CI and GitHub.
CI folder can be used to duplicate it's CI into our own.

[romani]

we do not need this for maven, as we already have bunch of real maven project executions.
search for "no-error-" at violations.sh

checkstyle/.ci/validation.sh

Lines 196 to 207 in a5be3cf

	no-error-pmd)
	CS_POM_VERSION="$(getCheckstylePomVersion)"
	echo "CS_version: ${CS_POM_VERSION}"
	mvn -e --no-transfer-progress clean install -Pno-validations
	echo "Checkout target sources ..."
	checkout_from "https://github.com/pmd/build-tools.git"
	cd .ci-temp/build-tools/
	mvn -e --no-transfer-progress install
	cd ..
	git clone https://github.com/pmd/pmd.git
	cd pmd
	./mvnw verify --show-version --errors --batch-mode --no-transfer-progress \

the same was for Gradle but not pjdbc is on latest gradle+plugin, so it is now not very countable
.

[rnveach]

Those existing maven ones will have to be converted to CLI when we start to break compatibility with it.

[romani]

the same we will do with our samples, so I do not see much reason to extra.

We did have problem to find ANT project, that is true, so we are ok to keep it. But maven and gradle we should use existing real projects.

[rnveach]

I did not really see us covering out maven sample. Regressions will move off maven to continue having regression. Maven sample could be out last item to stay as maven to warn us of future breaks.

[romani]

we have xwiki on maven, as we break maven, we will need to disable xwiki for some time - it is fine.
lets find gradle project:
pjdbc (gradle 7.5.1 recently) -

checkstyle/.ci/validation.sh

Lines 608 to 612 in 95522ae

	checkout_from https://github.com/pgjdbc/pgjdbc.git
	cd .ci-temp/pgjdbc
	# pgjdbc easily damage build, we should use stable versions
	git checkout "135be5a4395033a4ba23a1dd70ad76e0bd443a8d"
	./gradlew --no-parallel --no-daemon checkstyleAll \

configurate (gradle 7.6) -

checkstyle/.ci/validation.sh

Lines 226 to 228 in 95522ae

	git clone https://github.com/SpongePowered/Configurate.git
	cd Configurate
	./gradlew -PcheckstyleVersion="${CS_POM_VERSION}" -x test check

simple-binary-encoding (gradle 7.6) -

checkstyle/.ci/validation.sh

Lines 309 to 314 in 95522ae

	checkout_from https://github.com/real-logic/simple-binary-encoding.git
	cd .ci-temp/simple-binary-encoding
	sed -i'' \
	"s/'com.puppycrawl.tools:checkstyle:.*'/'com.puppycrawl.tools:checkstyle:$CS_POM_VERSION'/" \
	build.gradle
	./gradlew build --stacktrace

[romani]

@rnveach , do you think we need to keep example on old gradle and keep it in our regression in main repo ?
even we are on gradle 7.5 - https://github.com/sevntu-checkstyle/checkstyle-samples/blob/master/gradle-project/gradle/wrapper/gradle-wrapper.properties
and we donot use any version for checkstyle plugin, so it means latest, so we do not test anything special.

So I think we need to close this issue.
Ant was the only missed item.

[rnveach]

do you think we need to keep example on old gradle and keep it in our regression in main repo ?

We should keep any example of what we are supporting. We have multiple JDKs in CI because we support them all.

Since we are delaying ANT's classpath removal because of a specific gradle version, we should ensure that version is in CI until it warns us we are breaking and we break on purpose. If we need to add newer version of gradle just for verification, then there is no harm.

You did gradle investigation, so if you say we are good, then I am good.

[romani]

we need to migrate to CIrcleCI gradle execution
https://github.com/sevntu-checkstyle/checkstyle-samples/blob/311e204e8939c2ab87796bfb73c44052ed2af023/.travis.yml#L37-L39

to be able to run gradle.

[romani]

Sample project is migrated to circleci https://app.circleci.com/pipelines/github/sevntu-checkstyle/checkstyle-samples/16/workflows/8c927582-d094-4ad7-a287-988fc63f8974/jobs/52

So we need to run task in checkstyle repo CI, to be sure that no breaking changes happening in PR update.

List of jobs to run:
Ant https://github.com/sevntu-checkstyle/checkstyle-samples/blob/1615ad3ff3aabfe6b7c5e16ce85e41763f4d4c25/.circleci/config.yml#L40
Gradle https://github.com/sevntu-checkstyle/checkstyle-samples/blob/1615ad3ff3aabfe6b7c5e16ce85e41763f4d4c25/.circleci/config.yml#L43

We can skip maven, as it is everywhere

---

We probably need new issue creation be short and exact on what needs to be done