[SPARK-26918][DOCS] All .md should have ASF license header

[srowen]

What changes were proposed in this pull request?

Add AL2 license to metadata of all .md files.
This seemed to be the tidiest way as it will get ignored by .md renderers and other tools. Attempts to write them as markdown comments revealed that there is no such standard thing.

How was this patch tested?

Doc build

[SparkQA]

Test build #104091 has finished for PR 24243 at commit b841b39.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[felixcheung]

thx!!

[attilapiros]

Question about including the license twice otherwise LGTM.

I have checked for including double licenses by executing the following search and there is only ./docs/cloud-integration.md where this problem occurs:

$ find . -name "*.md" -exec grep -c "http://www.apache.org/licenses" \{} \; -print | grep 2 -A 2
2
./docs/cloud-integration.md
1
--
./build/scala-2.12.7/doc/LICENSE.md
2
./build/scala-2.12.8/doc/LICENSE.md
1
./build/scala-2.11.7/doc/LICENSE.md
1
./build/scala-2.11.8/doc/LICENSE.md
1
./build/scala-2.11.12/doc/LICENSE.md
0
0

[attilapiros]

I run a completness check too. As I understand only the markdowns under the ./docs are targeted, in this case the PR is also complete:

$ find . -name "*.md" -exec grep -L "http://www.apache.org/licenses" \{} \;
./resource-managers/kubernetes/integration-tests/README.md
./core/target/scala-2.12/classes/org/apache/spark/deploy/security/README.md
./core/src/main/scala/org/apache/spark/deploy/security/README.md
./python/README.md
./R/WINDOWS.md
./R/README.md
./R/DOCUMENTATION.md
./R/CRAN_RELEASE.md
./README.md
./common/network-common/src/main/java/org/apache/spark/network/crypto/README.md
./common/tags/README.md
./CONTRIBUTING.md
./dev/appveyor-guide.md
./dev/README.md
./external/docker/README.md
./external/docker/spark-test/README.md
./sql/core/target/scala-2.12/classes/org/apache/spark/sql/test/README.md
./sql/core/src/test/README.md
./sql/core/src/main/scala/org/apache/spark/sql/test/README.md
./sql/README.md

[srowen]

I'll fix all that, yeah. The only annoying thing is that this metadata gets displayed in Github like ...
https://github.com/srowen/spark/blob/SPARK-26918/docs/building-spark.md
... with a big blob at the top. For README.md files clearly meant for consumption on Github, that's distracting. I wonder if we can plausibly not do this for the very top-level project one given how the license is already linked twice by Github when you're looking at that page.

[attilapiros]

Regarding github markdown comment I have found this working:

[//]: comment

But unfortunately it should be combined with the metadata and that I am afraid won't work.
I have not found any multi-line comments which kinda makes sense for markdown (to keep parsing simple).

[srowen]

I tried that and it rendered in the HTML output, so abandoned it. I couldn't find something that works with jekyll other than metadata

[attilapiros]

I have only one idea: to introduce an extra "build" step where you filter out lines starting with [//]: and pass the result to jekyll.

[SparkQA]

Test build #104106 has finished for PR 24243 at commit 226a238.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Yeah... I figure it's not worth it unless we find for some reason this solution isn't sufficient or it's too annoying. I like that it's metadata in this case. If we had to, it could convert to HTML comments but that puts the bulk of the comment in the HTML too.

[felixcheung]

I’d agree we should avoid doing a bunch of extra work for this...

[srowen]

Merged to master