Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

YARN-11444. Improve YARN md documentation format. #5431

Closed
wants to merge 3 commits into from

Conversation

slfan1989
Copy link
Contributor

@slfan1989 slfan1989 commented Feb 24, 2023

JIRA: YARN-11444. Improve YARN md documentation format.

  1. Improve the table format to make the readability better.

When reading the md file locally, I encountered format problems and it was not easy to read. I think we should align the columns

before modification

Not easy to read, not easy to modify

image

after modification

image

  1. Modify some typo errors.

image

  1. Modify the list number to display correctly.
    image

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 55s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+0 🆗 markdownlint 0m 1s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ trunk Compile Tests _
+1 💚 mvninstall 38m 4s trunk passed
+1 💚 mvnsite 0m 31s trunk passed
+1 💚 shadedclient 58m 19s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 15s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 20s the patch passed
+1 💚 shadedclient 19m 21s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 asflicense 0m 38s The patch does not generate ASF License warnings.
81m 17s
Subsystem Report/Notes
Docker ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5431/1/artifact/out/Dockerfile
GITHUB PR #5431
Optional Tests dupname asflicense mvnsite codespell detsecrets markdownlint
uname Linux 5e94285e9c7d 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 04cd33a
Max. process+thread count 560 (vs. ulimit of 5500)
modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5431/1/console
versions git=2.25.1 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 52s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ trunk Compile Tests _
+1 💚 mvninstall 41m 35s trunk passed
+1 💚 mvnsite 0m 26s trunk passed
+1 💚 shadedclient 64m 29s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 13s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 18s the patch passed
+1 💚 shadedclient 23m 3s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 asflicense 0m 36s The patch does not generate ASF License warnings.
91m 13s
Subsystem Report/Notes
Docker ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5431/2/artifact/out/Dockerfile
GITHUB PR #5431
Optional Tests dupname asflicense mvnsite codespell detsecrets markdownlint
uname Linux e5ba83a0e135 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 4083fca
Max. process+thread count 595 (vs. ulimit of 5500)
modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5431/2/console
versions git=2.25.1 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@slfan1989
Copy link
Contributor Author

@ayushtkn Can you help review this pr? Thank you very much!

Copy link
Member

@ayushtkn ayushtkn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't check line by line. It is huge.......

I doubt the numbering stuff, putting 1. is the md files automatically sort the numbering. If not pass me the links on every change from here, where it is unsorted.
http://apache.github.io/hadoop

Spacing changes, I couldn't figure out either, you need to attach screenshots so that folks can figure out what is the impact of having those spaces.

Hope you have built the website locally and checked nothing on those pages is getting broken either

Comment on lines 141 to +142
The above code shows the `DevicePluginScheduler` interface that you might
needed if you want to arm the plugin with a more efficient scheduler.
be needed if you want to arm the plugin with a more efficient scheduler.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this sound better?

The above code shows the you might need  `DevicePluginScheduler` interface  if you want to arm the plugin with a more efficient scheduler.

Comment on lines -62 to +78
1. It works in both secure and non secure clusters.
1. The generic history service is built on the timeline store.
1. The history can be stored in memory or in a leveldb database store;
2. It works in both secure and non secure clusters.
3. The generic history service is built on the timeline store.
4. The history can be stored in memory or in a leveldb database store;
the latter ensures the history is preserved over Timeline Server restarts.
1. The ability to install framework specific UIs in YARN is not supported.
1. Application specific information is only available via RESTful APIs using JSON type content.
1. The "Timeline Server v1" REST API has been declared one of the REST APIs
5. The ability to install framework specific UIs in YARN is not supported.
6. Application specific information is only available via RESTful APIs using JSON type content.
7. The "Timeline Server v1" REST API has been declared one of the REST APIs
whose compatibility will be maintained in future releases.
1. The single-server implementation of the Timeline Server places a limit on
8. The single-server implementation of the Timeline Server places a limit on
the scalability of the service; it also prevents the service being
High-Availability component of the YARN infrastructure.

Future Plans

1. Future releases will introduce a next generation timeline service
which is scalable and reliable, ["Timeline Service v2"](./TimelineServiceV2.html).
1. The expanded features of this service *may not* be available to
2. The expanded features of this service *may not* be available to
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this sounds correct only, the numbering is correct only
http://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/TimelineServer.html

Comment on lines -836 to +842
1. `limit` - A limit on the number of events to return for each entity. If null,
2. `limit` - A limit on the number of events to return for each entity. If null,
defaults to 100 events per entity.
1. `windowStart` - If not null, retrieves only events later than the given time
3. `windowStart` - If not null, retrieves only events later than the given time
(exclusive)
1. `windowEnd` - If not null, retrieves only events earlier than the given time
4. `windowEnd` - If not null, retrieves only events earlier than the given time
(inclusive)
1. `eventType` - Restricts the events returned to the given types. If null,
5. `eventType` - Restricts the events returned to the given types. If null,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I doubt if this is required. Pass me the link from here where it shows messed up
http://apache.github.io/hadoop

* **A queuePlacementPolicy element**: which contains a list of rule elements that tell the scheduler how to place incoming apps into queues. Rules are applied in the order that they are listed. Rules may take arguments. All rules accept the "create" argument, which indicates whether the rule can create a new queue. "Create" defaults to true; if set to false and the rule would place the app in a queue that is not configured in the allocations file, we continue on to the next rule. The last rule must be one that can never issue a continue. Valid rules are:
* **A queuePlacementPolicy element**: which contains a list of rule elements that tell the scheduler how to place incoming apps into queues. Rules are applied in the order that they are listed. Rules may take arguments. All rules accept the "create" argument, which indicates whether the rule can create a new queue. "Create" defaults to true; if set to false and the rule would place the app in a queue that is not configured in the allocations file, we continue on to the next rule. The last rule must be one that can never issue a continued. Valid rules are:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

continue to continued?Is that a command which is continued but not continue, can you pass me the link to that?

@slfan1989
Copy link
Contributor Author

slfan1989 commented Apr 25, 2023

This pr mainly modifies the format of the table in markdown, so as to better modify the content of the table in md later.

  • Take CapacityScheduler.md as an example

image

The main problem is that the tables are not aligned and we need to find the table boundaries. This is sometimes a bit difficult, if the description information is more. We need to find the boundary, when we want to modify some content in the table.

image

If we can align the table, it will be easier to find a row to modify.

image

After modification, we found that the same content is more tidy.

  • This pr modifies 27 md files, and the html files are listed in the table
md file link
CapacityScheduler.md http://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
DevelopYourOwnDevicePlugin.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/DevelopYourOwnDevicePlugin.html
DockerContainers.md http://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/DockerContainers.html
FairScheduler.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/FairScheduler.html
Federation.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/Federation.html
GracefulDecommission.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html
NodeLabel.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/NodeLabel.html
NodeManager.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/NodeManager.html
NodeManagerCgroups.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html
NodeManagerRest.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html
ResourceManagerHA.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html
ResourceManagerRest.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html
ResourceManagerRestart.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html
ResourceModel.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/ResourceModel.html
RuncContainers.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/RuncContainers.html
SharedCache.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/SharedCache.html
TimelineServer.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/TimelineServer.html
TimelineServiceV2.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html
UsingFPGA.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/UsingFPGA.html
UsingGpus.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/UsingGpus.html
UsingNuma.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/UsingNuma.html
WebApplicationProxy.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html
WebServicesIntro.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html
WritingYarnApplications.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html
YarnApplicationSecurity.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html
YarnCommands.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/YarnCommands.html
YarnUI2.md https://apache.github.io/hadoop/hadoop-yarn/hadoop-yarn-site/YarnUI2.html

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 2s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+0 🆗 markdownlint 0m 1s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ trunk Compile Tests _
+1 💚 mvninstall 41m 9s trunk passed
+1 💚 mvnsite 0m 29s trunk passed
+1 💚 shadedclient 61m 12s branch has no errors when building and testing our client artifacts.
-0 ⚠️ patch 61m 30s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 14s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 19s the patch passed
+1 💚 shadedclient 22m 46s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 asflicense 0m 39s The patch does not generate ASF License warnings.
88m 53s
Subsystem Report/Notes
Docker ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5431/3/artifact/out/Dockerfile
GITHUB PR #5431
Optional Tests dupname asflicense mvnsite codespell detsecrets markdownlint
uname Linux 40c781589c50 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / da6ccfb
Max. process+thread count 626 (vs. ulimit of 5500)
modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5431/3/console
versions git=2.25.1 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@slfan1989 slfan1989 closed this Apr 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants