Doc inspection tool 001

[kamir]

No description provided.

[justinmclean]

Hi Mirko, I appreciate what you are trying to do here but you may need to take some care with using the output of LLMs due to the possible issues around licensing and copyright. There a recent discussion on legal discuss list about that. Kind Regards, Justin

…

On 6 Jul 2023, at 1:17 am, Mirko Kämpf ***@***.***> wrote: You can view, comment on, or merge this pull request online at: #87 Commit Summary 6781e8a <6781e8a> Added module for Apache Wayang. b5029e8 <b5029e8> Clean project to prep PR. 49703a4 <49703a4> Add scripts cb66ac8 <cb66ac8> Updated Gitignore. 83d9f23 <83d9f23> Created initial version of DOCU inspector. e9c9c52 <e9c9c52> Created a simple repo-inspection tool using ChatGPT (Azure OpenAI model). 2d3d9ba <2d3d9ba> Fixed format issues. File Changes (38 files <https://github.com/apache/incubator-training/pull/87/files>) M .gitignore <https://github.com/apache/incubator-training/pull/87/files#diff-bc37d034bad564583790a46f19d807abfe519c5671395fd494d8cce506c42947> (9) A content/Apache/Wayang/pom.xml <https://github.com/apache/incubator-training/pull/87/files#diff-cfb2281123f5e3510324cfe72ae79718b3120fc73865ac9352cd325facde140d> (40) A content/Apache/Wayang/quickstart-guide/README.md <https://github.com/apache/incubator-training/pull/87/files#diff-2e569dcb4f4c0869e121306dababae32fcefbc0e03c4da15c88d76ebc9f526dd> (66) A content/Apache/Wayang/quickstart-guide/install-deps-centos.sh <https://github.com/apache/incubator-training/pull/87/files#diff-4623d090f4a6617bb0c6b45bd57a9c19c51dc7abce9155a7ad212d58cba647f7> (61) A content/Apache/Wayang/quickstart-guide/install-deps-mac.sh <https://github.com/apache/incubator-training/pull/87/files#diff-8f80a1148e0325ec7b664f6520f6a43c5e09c8e6461d23fd68fbaec786c0b415> (56) A content/Apache/Wayang/quickstart-guide/pom.xml <https://github.com/apache/incubator-training/pull/87/files#diff-6ba2ce3d5e624ce0f55dd5041014381b1c6f3046f895b0f87005ce295c3a9d3a> (40) A content/Apache/Wayang/quickstart-guide/src/.DS_Store <https://github.com/apache/incubator-training/pull/87/files#diff-6090c685c1a81693826c52c357175c15025945aa947995916fc5878be971dd94> (0) A content/Apache/Wayang/quickstart-guide/src/main/.DS_Store <https://github.com/apache/incubator-training/pull/87/files#diff-1e525b405b49c25cae0b46ffbf93f138ec2b08372293ced5412b7d79d615d372> (0) A content/Apache/Wayang/quickstart-guide/src/main/asciidoc/.DS_Store <https://github.com/apache/incubator-training/pull/87/files#diff-cef8f4db84b0c1072d3b86f1ca9138948bcbd750032d8505d7464b746143b001> (0) A content/Apache/Wayang/quickstart-guide/src/main/asciidoc/_settings.adoc <https://github.com/apache/incubator-training/pull/87/files#diff-d7edc0168c563ee1d6765dbf5d185bcee5839f6be196f06691195cb3eb89eb31> (36) A content/Apache/Wayang/quickstart-guide/src/main/asciidoc/diagrams/mermaid-flowchart-test.mmd <https://github.com/apache/incubator-training/pull/87/files#diff-5060451c7642a51492af9d0df1bd831e514db49f4432eacba8576a07a13d510f> (5) A content/Apache/Wayang/quickstart-guide/src/main/asciidoc/index.adoc <https://github.com/apache/incubator-training/pull/87/files#diff-9c6e3d75c27a2c52b92fc9ea9f5169ee2df1c0551a309acdbd1638c261ff6f5a> (621) A content/Apache/Wayang/quickstart-guide/src/main/asciidoc/subdir/_b.adoc <https://github.com/apache/incubator-training/pull/87/files#diff-e80418d360990ed0eedc5d7847c929fe51786a876e9698231421844c282db1bb> (26) A content/Apache/Wayang/quickstart-guide/src/main/asciidoc/subdir/_c.adoc <https://github.com/apache/incubator-training/pull/87/files#diff-8170392ff9a748c90a07958b7f8c6a76b85fff637ff029a70282c51e4a86ee15> (20) A content/Apache/Wayang/quickstart-guide/src/main/java/apache-wayang/StringUtils.java <https://github.com/apache/incubator-training/pull/87/files#diff-332866b530169c809b183cbf2eb2b8953487031a7ce980fa79147f40117a917d> (27) A content/Apache/Wayang/quickstart-guide/src/main/resources/images/toddy.jpg <https://github.com/apache/incubator-training/pull/87/files#diff-ea1a7e02f3dc1da9b9e5916bee027e588c50ed0bb836bdb3b8c077436eb3f888> (0) M content/Apache/pom.xml <https://github.com/apache/incubator-training/pull/87/files#diff-c3524f286499fcb9326312b86ace58392f0b45de9a931ccfec2136a3b0cef033> (1) A tools/repo-doc-analysis/.DS_Store <https://github.com/apache/incubator-training/pull/87/files#diff-4a9c9ec89529f8bb9bd77462c5d0958bc2e01c98190b165e9b2019e05fcfd28e> (0) A tools/repo-doc-analysis/bootstrap.sh <https://github.com/apache/incubator-training/pull/87/files#diff-0b8f9a7742d29cff6ec5e321856cb8e0d53b436cfe3423c9d4b0eea407134474> (8) A tools/repo-doc-analysis/data-store/incubator-wayang/index.faiss <https://github.com/apache/incubator-training/pull/87/files#diff-cb049421b6032303a84d16a4d683f2c8f6edaad4511c0fdbb5dbd88ae2ff31b0> (0) A tools/repo-doc-analysis/data-store/incubator-wayang/index.pkl <https://github.com/apache/incubator-training/pull/87/files#diff-a452de3e9b5d97ed7c61f2f66dbf9049c75943214c793d99273efc0ac5e00ed3> (0) A tools/repo-doc-analysis/data-store/kafka/index.faiss <https://github.com/apache/incubator-training/pull/87/files#diff-1a029954c8904b11b76361b70bd91fab907d3680f80a3befe5401f4c7a1ee59f> (0) A tools/repo-doc-analysis/data-store/kafka/index.pkl <https://github.com/apache/incubator-training/pull/87/files#diff-ccc406e415f0a35eeb680b251d8f0b35b91f0979464091c524b8773f6c9dafae> (0) A tools/repo-doc-analysis/docs-main/bootstrap-data.sh <https://github.com/apache/incubator-training/pull/87/files#diff-3b602c07b546e4de7eb70caaba3940ec1f61b4d96138d7dfaf944c382170a23f> (4) A tools/repo-doc-analysis/reports/incubator-wayang/answers_set_nr_2.md <https://github.com/apache/incubator-training/pull/87/files#diff-cf468b61c22fb5568aaf2cb3595ff9f5a1bb99f294f08155b5194b3208ad64b0> (102) A tools/repo-doc-analysis/reports/incubator-wayang/problems_set_nr_2.md <https://github.com/apache/incubator-training/pull/87/files#diff-1a350b00ced909376bdf4119ea4d603cf316fe2143c335de59a8de6e06ae2381> (28) A tools/repo-doc-analysis/scripts/__pycache__/question_sets.cpython-311.pyc <https://github.com/apache/incubator-training/pull/87/files#diff-886ed2e2c9ed8e9872779516cba3a9313da819936c1028932aaa9dcdf6177af5> (0) A tools/repo-doc-analysis/scripts/chatmodule.py <https://github.com/apache/incubator-training/pull/87/files#diff-46813fed3f47243b22201c16bfbaf6151fd9fceace1fe103113527fd38751309> (157) A tools/repo-doc-analysis/scripts/data-store/incubator-wayang/index.faiss <https://github.com/apache/incubator-training/pull/87/files#diff-ed0840109e0641fa57ac5aef0ea1236f928142402d49e9ebbe20a53f3493bf1d> (0) A tools/repo-doc-analysis/scripts/data-store/incubator-wayang/index.pkl <https://github.com/apache/incubator-training/pull/87/files#diff-5e43c8b20c903e5b28a0f43d9105ca70f4a02ed35bf3d833bbb38fcc66a375b5> (0) A tools/repo-doc-analysis/scripts/project_metadata/__pycache__/incubator_wayang.cpython-311.pyc <https://github.com/apache/incubator-training/pull/87/files#diff-300846a3e4c9613df03c6edd6061bde4a48c781e09916541c6e7d6ec607125b9> (0) A tools/repo-doc-analysis/scripts/project_metadata/__pycache__/kafka.cpython-311.pyc <https://github.com/apache/incubator-training/pull/87/files#diff-9f313cc81485eba93b81347fd0616f234596859115c27bf1d21f61579923a11a> (0) A tools/repo-doc-analysis/scripts/project_metadata/incubator_wayang.py <https://github.com/apache/incubator-training/pull/87/files#diff-86188c8bd055696b32671565ac2b7bd34b0c1ecd34651abccc3fa1a69ec243ed> (9) A tools/repo-doc-analysis/scripts/project_metadata/kafka.py <https://github.com/apache/incubator-training/pull/87/files#diff-031f76d8702c42ea2cea349f507667ba5e8cd38546ca6ada1517b39cb207d60f> (8) A tools/repo-doc-analysis/scripts/question_sets.py <https://github.com/apache/incubator-training/pull/87/files#diff-80b07ce2c56ac4a252de199aee1bec20857aed7de6eaab437ecd83ce5e1fb78f> (24) A tools/repo-doc-analysis/scripts/repository_based_qa_langchain.py <https://github.com/apache/incubator-training/pull/87/files#diff-4a5b210c4b0487ed49e6f2db138090a6d5d718978fdb0e7bb4d0bfdb72b7b507> (196) A tools/repo-doc-analysis/temp/data-store/index.faiss <https://github.com/apache/incubator-training/pull/87/files#diff-3a8e54aced3ca4b5e80276c50edda2574782d7344d3583dec77acf4984042325> (0) A tools/repo-doc-analysis/temp/data-store/index.pkl <https://github.com/apache/incubator-training/pull/87/files#diff-659d212ac981af6b3d2b09db484f0ccd13840fe255702ae38cd5088457c9fe43> (0) Patch Links: https://github.com/apache/incubator-training/pull/87.patch https://github.com/apache/incubator-training/pull/87.diff — Reply to this email directly, view it on GitHub <#87>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABDI6AA5LTO3BGZSL2ASTDXOWARJANCNFSM6AAAAAAZ7DPL7Y>. You are receiving this because you are subscribed to this thread.

[kamir]

Hi Justin,
currently the goal is to initiate the project with a focus on "semi-automation" of documentation tasks.
Most importantly, the prompts, the prompt templates, and the indexing logic will be developed and applied to small data sets, such as the docu-repos of several ASF projects.

Now we have two options:
(A) Creation of content, such as README or quickstart tutorials, or executive summaries etc.
(B) Automatic inspection of the content, using the "reference questions" as defined in the tool - and based on the result, a human editor can update the original documents in such a way that the questions can be answered. I

In case (B) no generated content will be published in a software repository, it is just temporary information which allows
quality assessments and improvements in a standardized semi-automatic way.

I understand that following the path towards (A) is still a risk. Hence, the goal is the creation of tool which can be configured to use multiple LLMs, starting from public offerings to personal deployments (such as OpenAI model in Azure self hosted by a user vs. the fully managed OpenAI model which is used right now.

[kamir]

Is it possible to continue with this kind of activity, as long as we do not use the generated content as "Apache Training content", but rather work on an assistance system which supports content editors (especially for software documentation and for training material) to get better docs faster released?

[justinmclean]

Please see https://www.apache.org/legal/generative-tooling.html

[kamir]

OK, thanks Justin.
I will "process" the docs and come up with a conclusion so that we know what to do with the pull request and the idea of using LLMs in general.

[chrisdutz]

Also for a PR like this ... a bit more description would be nice ... I think from having a look at the payload of the PR, I can get a rough idea of what it's about ... but I did notice quite some binary files being checked in and that's one thing we generally don't like to see in apache repos.

[kamir]

Ok, thanks for the feedback. I will provide description and a clean-up, especially the binaries.

[kamir]

Hey,

after being hit by a first wave of all the LLM Hype, I thought, it would be a nice contribution to the Apache Training project, if we would use this new opportunity. My first experiments with langchain made me confident, that this is the next great thing.

There have been several obstacles, e.g., the question: To what extend are we allowed to use LLMs in the ASF context.

I have realised that going forward on the tool-level would probably not improve the situation.
The tool I proposed would not generate content but rather would help developers and trainers to understand the code base
and the available docs faster. But since other tools with the same focus have been built in between, there is no need to bring such a tool into this project.

We can use tools like: https://github.com/peterw/Chat-with-Github-Repo for this purpose.
I suggest to not focus in building a new tool, when others have one ready to use.*

But when I started working on this topic, I did not see this coming.

With this in mind, we can close the PR.