Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable tiered category hierarchy/dropdowns for docs #17495

Closed
wants to merge 98 commits into from

Conversation

Zijian-Zhu
Copy link
Contributor

@Zijian-Zhu Zijian-Zhu commented May 25, 2023

I create a directory named overview-example-level-1-dir under docs/en to make an example of this tiered docs.

The following screenshot is the example of the actually docs and directory under docs/en/overview-example-level-1-dir
image
As you can see overview.md and Getting-Started.md are level 1 docs. Architecture.md and FAQ.md are level 2 docs.
JobService.md is level 3 docs and Use-Cases.md is level 4 docs.

The following screenshot is example of content in the new Yaml file.
image
All the sub-directories need to be listed in the subitems with their title which does not need to be the same as their directory name and their buttonTitle which is the title without white space since white space will bug in the HTML.
All the docs located in each directory need to be listed in the subfiles with docs title and url which is the path of the doc after jekyll bulid (this path always the same of the path of the files except the file type is .html not .md

The following screenshot is the output of the example.
WX20230525-175144@2x
This is the full expansion of the example list. The blue string is the URL of the corresponding docs. And the button is collapse type and use the buttonTitle as name

Kai-Zhang and others added 18 commits May 24, 2023 13:43
### What changes are proposed in this pull request?

mentioned in Alluxio#16629 

### Why are the changes needed?

The `Cluster.ActiveRpcReadCount` metric could get an impossible value like a negative number. The main reason is that the request handling doesn't cover all situations.

### Does this PR introduce any user facing changes?

Nope

			pr-link: Alluxio#17234
			change-id: cid-f7d28cbf57ef2212e4b8d300f8a18300b593b213
### What changes are proposed in this pull request?

Please outline the changes and how this PR fixes the issue.

### Why are the changes needed?

Please clarify why the changes are needed. For instance,
  1. If you propose a new API, clarify the use case for a new API.
  2. If you fix a bug, describe the bug.

### Does this PR introduce any user facing changes?

Please list the user-facing changes introduced by your change, including
  1. change in user-facing APIs
  2. addition or removal of property keys
  3. webui

			pr-link: Alluxio#17390
			change-id: cid-30d303cb559a315f334cdf880d4dde99ed7779c6
### What changes are proposed in this pull request?

This is the doc update from a few recent commits:
1. Alluxio@aee3c5c
2. Alluxio@9a4e154
3. Alluxio@dbac084
4. Alluxio@8da5953
5. Alluxio@141ee0e
6. Alluxio@26257e6

In short, a few improvements have been added that the admin can now see the version/revision of the running Alluxio components in the cluster. A few commands were also added to help with gracefully decommissioning/restarting workers in the cluster.

### Why are the changes needed?

This doc change helps users understand how to utilize those features.


			pr-link: Alluxio#17381
			change-id: cid-1d45c1750f84ac94d3fa79b9bbe0f0f328f7cb45
### What changes are proposed in this pull request?

Avoid the NPE exception in copy object with x-amz-tagging-directive

### Why are the changes needed?
Avoid the NPE exception

			pr-link: Alluxio#17400
			change-id: cid-db0f6ac83b1f6de7937c17f82911b153fe63155d
A workaround to fix the master health check issue, related to Alluxio#17382 which is caused by a recent change that starts grpc servers on standby masters.
Follower masters have a gRPC server always active. When the `MasterHealthCheckClient` connects to the gRPC server, it may find that some services aren't registered (NotFound) rather than unavailable. 
This manifests in a bug where `bin/alluxio-start.sh master` performs a health check after starting a master and incorrectly reports the master as unhealthy. 
			pr-link: Alluxio#17383
			change-id: cid-38f5babf01a3597099a821ec3be61f325630c098
### What changes are proposed in this pull request?

Remove s3 handler map in s3 proxy.

### Why are the changes needed?

Alluxio#17403

### Does this PR introduce any user facing changes?

Please list the user-facing changes introduced by your change, including
  1. change in user-facing APIs NO
  2. addition or removal of property keys NO
  3. webui NO

			pr-link: Alluxio#17406
			change-id: cid-ddf598601ab15f9d9ecb5a22901c191b54935d85
…nfig deprecation warning

Don't print a warning in log if the deprecated config is not set by the user explicitly.

Even if a deprecated config is not set, when a client or a worker loads cluster default config from master, the config is considered to be `CLUSTER_DEFUALT` level, and triggers a deprecation warning.

This PR changes the condition for a deprecation message to be logged to be that the level is higher than `CLUSTER_DEFAULT`, which is the same condition used by `isSetByUser`.

After this change, if a deprecated config does get set in `alluxio-site.properties`, the config source level would be `SITE_PROPERTIES` and get logged at the master's log.

No.

			pr-link: Alluxio#17333
			change-id: cid-584c3b66bbb701e7fc70d6e27d91dbd3072851f7
… and timestamp in web page

Display the config hash and config updated time

<img width="550" alt="image" src="https://user-images.githubusercontent.com/17329931/207004719-8c684f3d-24c7-4fce-bd95-32ff795b6172.png">

<img width="534" alt="image" src="https://user-images.githubusercontent.com/17329931/207027205-865183d3-c288-4c99-bbee-a53c6d4ba579.png">

Please clarify why the changes are needed. For instance,
  1. If you propose a new API, clarify the use case for a new API.
  2. If you fix a bug, describe the bug.

Please list the user-facing changes introduced by your change, including
  1. change in user-facing APIs
  2. addition or removal of property keys
  3. webui

			pr-link: Alluxio#16642
			change-id: cid-075d4f9ea1cee81f23706781dcf5cc5ca155e416
### What changes are proposed in this pull request?

Support print xattr for stat command.

### Why are the changes needed?

Needs to show the xattr for user.

### Does this PR introduce any user facing changes?

User can see the xattr leverage this PR.

			pr-link: Alluxio#17395
			change-id: cid-edcb428ee08c8d460c61ae5ce9d555cc6453f02e
…ration from mount option

Support config obs configuration from mount option

For now, we cannot specify other obs client configurations through any way.

After this PR, we can config any obs configuration.

			pr-link: Alluxio#17191
			change-id: cid-a7f9ac3929d6deceb9ee71a02a2dc5f6c44bb489
…ncurrently

Make the client poll the master concurrently.

Currently, we find the active master node by polling each node in series. If a node did not respond due to network or other issues, this may take a lot of time and affect service availability. Make the client poll the master concurrently can alleviate this problem.

			pr-link: Alluxio#17186
			change-id: cid-d1ef20b596deda02375324978adb23e6f8ce8dce
This PR change replace the `registerGaugeIfAbsent` in `DefaultBlockWorker:Metrics:registerGauges` with `registerCachedGaugeIfAbsent`, and made a container `BlockWorkerMetrics` for `BlockStoreMeta` of `DefaultBlockWorker`

Previously the MetricSystem will copy a full BlcokStoreMeta everytime it's trying to update the metric, which cost is expensive and can be avoid.

No

			pr-link: Alluxio#17357
			change-id: cid-0e05812a75e51e9b74dbb802b82abb8c56d19b27
### What changes are proposed in this pull request?

When the filesystem establishes a connection with the master, the configuration of the filesystem will be overridden by the master's configuration, resulting in changes. Therefore, we should not use the configuration of the filesystem anymore, but instead use the configuration of the local client.

### Why are the changes needed?

It is very common for the master configuration and client configuration to be different. We should avoid the newly created filesystem from being influenced by the server.

### Does this PR introduce any user facing changes?

Please list the user-facing changes introduced by your change, including
  1. change in user-facing APIs
  2. addition or removal of property keys
  3. webui

			pr-link: Alluxio#17439
			change-id: cid-1af4fd37ee8c52cd32552818989dce557256b228
the rolling upgrade feature is for limited experimental use currently with many limitations. we may bring back the docs in a future version when it is more widely applicable to various user scenarios.
			pr-link: Alluxio#17455
			change-id: cid-9a65f7f7b1fa55373594f7ca3ddcc33a979addec
Regarding Alluxio#16940 (comment). Add the substring "Xmx" that's supposed to be checked on.
			pr-link: Alluxio#17459
			change-id: cid-b4cc0afb6d84e7198a9fe245e994c9eb2100d0ac
### What changes are proposed in this pull request?

Check sleep timer interval change when update conf

### Why are the changes needed?

We currently generate these logs every time we execute the updateConf command, even if the value has not been modified.
```
2023-04-24 11:37:27,445 INFO  [config-hash-master-heartbeat-0](SleepingTimer.java:88) - update Master Cluster Metrics Updater interval supplier.
2023-04-24 11:37:27,445 INFO  [config-hash-master-heartbeat-0](SleepingTimer.java:88) - update Master Lost Worker Detection interval supplier.
2023-04-24 11:37:27,445 INFO  [config-hash-master-heartbeat-0](SleepingTimer.java:88) - update Worker register stream session cleaner interval supplier.
2023-04-24 11:37:27,445 INFO  [config-hash-master-heartbeat-0](SleepingTimer.java:88) - update Master Block Integrity Check interval supplier.
2023-04-24 11:37:27,445 INFO  [config-hash-master-heartbeat-0](SleepingTimer.java:88) - update Master TTL Check interval supplier.
2023-04-24 11:37:27,445 INFO  [config-hash-master-heartbeat-0](SleepingTimer.java:88) - update Master Lost Files Detection interval supplier.
2023-04-24 11:37:27,445 INFO  [config-hash-master-heartbeat-0](SleepingTimer.java:88) - update Master Replication Check interval supplier.
2023-04-24 11:37:27,445 INFO  [config-hash-master-heartbeat-0](SleepingTimer.java:88) - update Master Persistence Scheduler interval supplier.
2023-04-24 11:37:27,445 INFO  [config-hash-master-heartbeat-0](SleepingTimer.java:88) - update Master Persistence Checker interval supplier.
2023-04-24 11:37:27,445 INFO  [config-hash-master-heartbeat-0](SleepingTimer.java:88) - update Master Metrics Time Series interval supplier.
2023-04-24 11:37:27,445 INFO  [config-hash-master-heartbeat-0](SleepingTimer.java:88) - update Master Lost Master Detection interval supplier.
2023-04-24 11:37:27,446 INFO  [config-hash-master-heartbeat-0](SleepingTimer.java:88) - update Master Log Config Report Scheduling interval supplier.
2023-04-24 11:37:27,446 INFO  [config-hash-master-heartbeat-0](SleepingTimer.java:88) - update Master Journal Space Monitor interval supplier.
2023-04-24 11:37:27,446 INFO  [config-hash-master-heartbeat-0](SleepingTimer.java:88) - update Master Throttle interval supplier.

```

### Does this PR introduce any user facing changes?

No.
			pr-link: Alluxio#17309
			change-id: cid-4352f517ca1f1b37cab9784e1f89b104f09426d8
### What changes are proposed in this pull request?

Support changing log level at runtime for alluxio-fuse.

Now with the configuration "alluxio.fuse.web.enabled=true", the user can change the logLevel  for alluxio-fuse with the following ways

1. With `logLevel` command

```
$ ./bin/alluxio logLevel --logName=alluxio.fuse.AlluxioJniFuseFileSystem --target=localhost:49999 --level=INFO
Role inferred from port: localhost:49999[fuse]
localhost:49999[fuse]LogInfo{INFO, alluxio.fuse.AlluxioJniFuseFileSystem, Setting Level to INFO}
```

2. With `curl`

```
$ curl -X POST "http://localhost:49999/api/v1/fuse/logLevel?logName=alluxio.fuse.AlluxioJniFuseFileSystem&level=DEBUG"
{"logName":"alluxio.fuse.AlluxioJniFuseFileSystem","message":"Setting Level to DEBUG","level":"DEBUG"}
```

### Why are the changes needed?

Please refer Alluxio#17174 

### Does this PR introduce any user facing changes?

NO
			pr-link: Alluxio#17175
			change-id: cid-4130150f647217da8fd396dc542753400d6b568f
ssz1997 and others added 6 commits May 26, 2023 00:28
This reverts commit 27d7479.

There's permission issue with the Dockerfile while doing `ln`.
			pr-link: Alluxio#17500
			change-id: cid-ff42b4edacf8fbe3345f3d85182237d6da8a2a56
### What changes are proposed in this pull request?

Modify the visibility and inheritance of some interfaces

### Why are the changes needed?

Supports better scalability

			pr-link: Alluxio#17499
			change-id: cid-c53c83f377b46bb4509d9aa10a556ed3cb9bfc37
### What changes are proposed in this pull request?

Add toModeBits for extension

### Why are the changes needed?

Add toModeBits for extension

### Does this PR introduce any user facing changes?
NA

			pr-link: Alluxio#17501
			change-id: cid-196e47cd2e77cae8143b3a287b2ec96df1e96cc4
### What changes are proposed in this pull request?
- generate `libexec/version.sh` for using `--dryRun` 


### Why are the changes needed?

For using dry run flag for dev purposes

### Does this PR introduce any user facing changes?


			pr-link: Alluxio#17498
			change-id: cid-c5f26e5a2afb9db1c7d87e979bc63d84a4db7674
### What changes are proposed in this pull request?
make scheduler universally work for copyjob + doraloadjob

### Why are the changes needed?
resolve conflict of copyjob to use the new scheduler

### Does this PR introduce any user facing changes?
N/A

			pr-link: Alluxio#17368
			change-id: cid-e67f2b0a931944e56d35b06f421223e7d8eb2c4f
### What changes are proposed in this pull request?

Fix a deadlock in `LocalCacheManager.getAndLoad`.

To reproduce the deadlock: dbw9580@ff87c73

![image](https://github.com/Alluxio/alluxio/assets/6999708/d604f39b-e409-4c24-8fb1-b55893825599)


### Why are the changes needed?

Deadlock detected by jstack:

<details>
  <summary>jstack output</summary>

  ```

  Found one Java-level deadlock:
  =============================
  "pool-4-thread-1":
    waiting for ownable synchronizer 0x000000070eb3b9f0, (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync),
    which is held by "pool-4-thread-6"
  "pool-4-thread-6":
    waiting for ownable synchronizer 0x000000070eb25728, (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync),
    which is held by "pool-4-thread-3"
  "pool-4-thread-3":
    waiting for ownable synchronizer 0x000000070eb3b9f0, (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync),
    which is held by "pool-4-thread-6"
  
  Java stack information for the threads listed above:
  ===================================================
  "pool-4-thread-1":
          at jdk.internal.misc.Unsafe.park(java.base@11.0.19/Native Method)
          - parking to wait for  <0x000000070eb3b9f0> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
          at java.util.concurrent.locks.LockSupport.park(java.base@11.0.19/LockSupport.java:194)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.19/AbstractQueuedSynchronizer.java:885)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.19/AbstractQueuedSynchronizer.java:917)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.19/AbstractQueuedSynchronizer.java:1240)
          at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(java.base@11.0.19/ReentrantReadWriteLock.java:959)
          at alluxio.resource.LockResource.<init>(LockResource.java:102)
          at alluxio.resource.LockResource.<init>(LockResource.java:73)
          at alluxio.resource.LockResource.<init>(LockResource.java:57)
          at alluxio.client.file.cache.LocalCacheManager.getAndLoad(LocalCacheManager.java:525)
          at alluxio.client.file.cache.LocalCacheManagerTest.lambda$concurrentGetAndLoad$9(LocalCacheManagerTest.java:1133)
          at alluxio.client.file.cache.LocalCacheManagerTest$$Lambda$167/0x00000008001c8c40.call(Unknown Source)
          at java.util.concurrent.FutureTask.run(java.base@11.0.19/FutureTask.java:264)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.19/ThreadPoolExecutor.java:1128)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.19/ThreadPoolExecutor.java:628)
          at java.lang.Thread.run(java.base@11.0.19/Thread.java:829)
  "pool-4-thread-6":
          at jdk.internal.misc.Unsafe.park(java.base@11.0.19/Native Method)
          - parking to wait for  <0x000000070eb25728> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
          at java.util.concurrent.locks.LockSupport.park(java.base@11.0.19/LockSupport.java:194)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.19/AbstractQueuedSynchronizer.java:885)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.19/AbstractQueuedSynchronizer.java:917)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.19/AbstractQueuedSynchronizer.java:1240)
          at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(java.base@11.0.19/ReentrantReadWriteLock.java:959)
          at alluxio.resource.LockResource.<init>(LockResource.java:102)
          at alluxio.resource.LockResource.<init>(LockResource.java:73)
          at alluxio.resource.LockResource.<init>(LockResource.java:57)
          at alluxio.client.file.cache.LocalCacheManager.putAttempt(LocalCacheManager.java:374)
          at alluxio.client.file.cache.LocalCacheManager.putInternal(LocalCacheManager.java:278)
          at alluxio.client.file.cache.LocalCacheManager.put(LocalCacheManager.java:238)
          at alluxio.client.file.cache.CacheManager.put(CacheManager.java:206)
          at alluxio.client.file.cache.LocalCacheManager.getAndLoad(LocalCacheManager.java:546)
          at alluxio.client.file.cache.LocalCacheManagerTest.lambda$concurrentGetAndLoad$9(LocalCacheManagerTest.java:1133)
          at alluxio.client.file.cache.LocalCacheManagerTest$$Lambda$167/0x00000008001c8c40.call(Unknown Source)
          at java.util.concurrent.FutureTask.run(java.base@11.0.19/FutureTask.java:264)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.19/ThreadPoolExecutor.java:1128)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.19/ThreadPoolExecutor.java:628)
          at java.lang.Thread.run(java.base@11.0.19/Thread.java:829)
  "pool-4-thread-3":
          at jdk.internal.misc.Unsafe.park(java.base@11.0.19/Native Method)
          - parking to wait for  <0x000000070eb3b9f0> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
          at java.util.concurrent.locks.LockSupport.park(java.base@11.0.19/LockSupport.java:194)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.19/AbstractQueuedSynchronizer.java:885)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.19/AbstractQueuedSynchronizer.java:917)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.19/AbstractQueuedSynchronizer.java:1240)
          at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(java.base@11.0.19/ReentrantReadWriteLock.java:959)
          at alluxio.resource.LockResource.<init>(LockResource.java:102)
          at alluxio.resource.LockResource.<init>(LockResource.java:73)
          at alluxio.resource.LockResource.<init>(LockResource.java:57)
          at alluxio.client.file.cache.LocalCacheManager.putAttempt(LocalCacheManager.java:375)
          at alluxio.client.file.cache.LocalCacheManager.putInternal(LocalCacheManager.java:278)
          at alluxio.client.file.cache.LocalCacheManager.put(LocalCacheManager.java:238)
          at alluxio.client.file.cache.CacheManager.put(CacheManager.java:206)
          at alluxio.client.file.cache.LocalCacheManager.getAndLoad(LocalCacheManager.java:546)
          at alluxio.client.file.cache.LocalCacheManagerTest.lambda$concurrentGetAndLoad$9(LocalCacheManagerTest.java:1133)
          at alluxio.client.file.cache.LocalCacheManagerTest$$Lambda$167/0x00000008001c8c40.call(Unknown Source)
          at java.util.concurrent.FutureTask.run(java.base@11.0.19/FutureTask.java:264)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.19/ThreadPoolExecutor.java:1128)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.19/ThreadPoolExecutor.java:628)
          at java.lang.Thread.run(java.base@11.0.19/Thread.java:829)
  
  ```

</details>

### Does this PR introduce any user facing changes?

No.
			pr-link: Alluxio#17504
			change-id: cid-c5154832c855027011b89b7af17731eec9eddc3a
@LuQQiu
Copy link
Contributor

LuQQiu commented May 26, 2023

@Zijian-Zhu Thanks for the docs structure!
The levels looks good!

Some small suggestion about the format style:
Ideally directory and docs on the same level have the same visual effect, or slightly different visual effect
for example, level-2-dir matches the Overview and getting started style, or could have a different color indicating it's a folder instead of a doc.

huanghua78 and others added 3 commits May 28, 2023 20:07
- CreateFile from client to worker
- CompleteFile from client to worker
- OpenFileHandle in worker
- OpenFileHandles in worker

			pr-link: Alluxio#17505
			change-id: cid-8cf626c579dec84284265354ad7026430e8ecb95
### What changes are proposed in this pull request?

fixed directory deletion failure when ufs is cos

### Why are the changes needed?

1. cos bucket name is bucket-appid
2. COSUnderFileSystem.deleteObjects request is not bucket-appid case  `Failed to delete 1 paths from the under file system: /xxx (UFS delete dir failed)`
3. ObjectUnderFileSystem.submitBatch exception should print


### Does this PR introduce any user facing changes?

Please list the user-facing changes introduced by your change, including
  1. when ufs is cos can normal exec alluxio rm -r xxx

			pr-link: Alluxio#17493
			change-id: cid-1dc02fa2781254b85f0c7dc58c582469cfc19781
Cherry-pick of existing commit.
orig-pr: Alluxio#17469
orig-commit: Alluxio/alluxio@9d2f9cf
orig-commit-author: maobaolong <307499405@qq.com>

			pr-link: Alluxio#17507
			change-id: cid-1dde372d4a0aa0604b44ddbbe90762293f31c20c
### What changes are proposed in this pull request?
Modify the visibility and inheritance of some interfaces

### Why are the changes needed?
Modify the visibility and inheritance of some interfaces


			pr-link: Alluxio#17601
			change-id: cid-11e1459ed874f67c2cb17fcf31114c8d5e54786c
@Xenorith Xenorith mentioned this pull request Jun 13, 2023
jja725 and others added 7 commits June 13, 2023 13:06
### What changes are proposed in this pull request?

Make mount command work in dora

### Why are the changes needed?

na

### Does this PR introduce any user facing changes?

na

			pr-link: Alluxio#17606
			change-id: cid-e8f88e0ae0f0b14dda70d6ca7db4e8935fd4e6f9
### What changes are proposed in this pull request?

Fix move job tasks didn't picking up active workers.

### Why are the changes needed?

Fix move job bugs

### Does this PR introduce any user facing changes?
na

			pr-link: Alluxio#17604
			change-id: cid-e668e9d50017a6a6c37f15daaec3ac348ab8033d
### What changes are proposed in this pull request?
change task exceptions to catch throwable 

### Why are the changes needed?

otherwise when throwable happens it would report task success

### Does this PR introduce any user facing changes?

na

			pr-link: Alluxio#17608
			change-id: cid-018ac3d186e337f0fdf777e8065fd7fbd7900247
### What changes are proposed in this pull request?

Fixed an error in `UnderStorageReadIntegrationTest. java`.
Added some mReadNoCache cases in the test cases.

### Why are the changes needed?

Fix a test bug and add more cases.

### Does this PR introduce any user facing changes?
na

			pr-link: Alluxio#17602
			change-id: cid-97b66e220c0eb8db4f2c1e3187c43abdadd92db1
### What changes are proposed in this pull request?

Modify for GroupMappingService extension

### Why are the changes needed?

Modify for GroupMappingService extension

### Does this PR introduce any user facing changes?
NA

			pr-link: Alluxio#17613
			change-id: cid-a9bfff652b55f638250fe9472bde163032fca235
Remove dora unrelated docs
			pr-link: Alluxio#17584
			change-id: cid-e2ff76843387b6b6b5a5d6c6454865d63a2d038d
			pr-link: Alluxio#17614
			change-id: cid-73f795e1035e665071ac1db3629dc929968ee6af
@Zijian-Zhu Zijian-Zhu changed the title [WIP] Enable tiered category hierarchy/dropdowns for docs Enable tiered category hierarchy/dropdowns for docs Jun 14, 2023
@Zijian-Zhu Zijian-Zhu requested a review from Xenorith June 14, 2023 23:51
@Xenorith
Copy link
Contributor

need to fix the doc check. it used to ensure the categories on each page is an existing category listed in the config yml, but now that mechanism is removed, so we could also remove the check.

in a future PR, we should add back a similar check to ensure that config yml file list and the list of all doc md files are exactly the same.

huanghua78 and others added 10 commits June 14, 2023 23:53
### What changes are proposed in this pull request?

Add and implement functions to invalidate all cached pages by fileId.

### Why are the changes needed?

When we know that some file is to be removed, or renamed, or overwritten, we need to invalidate the cached pages in PagingStore.
Please note, at this moment, it is possible that the target file is already removed from UFS. So, getStatus() of this file will fail with exception. But we still need to invalidate all the cached pages.

### Does this PR introduce any user facing changes?

N/A

			pr-link: Alluxio#17582
			change-id: cid-0ce28e1593308a3b429162d86792249b50d3b4c9
### What changes are proposed in this pull request?

Remove mount check when starting alluxio if users do not intend to mount

### Why are the changes needed?

When the user does not want to mount and starts alluxio directly, the user can enter `./bin/alluxio-start.sh local` or `./bin/alluxio-start.sh all`. The old code will not be able to start alluxio successfully, it will refuse to start or the AlluxioWorker will fail to start. This PR fixes the issue.

![截屏2023-06-08 17 25 44](https://github.com/Alluxio/alluxio/assets/132879614/53cd6cb6-19e0-424b-8ff2-8ba49c8f0cb6)

The reason for deleting the mount check part of the code is:
We found that the mount check is only started when the user enters `./bin/alluxio-start.sh worker` or `./bin/alluxio-start.sh local`. However, if the user intends to use alluxio after mounting, the user will use `alluxio-mount.sh` for the mount operation. If the mount fails, an error will be reported, and `alluxio-start.sh` will not be executed. If alluxio is started directly without mounting, there is no need to perform mount check. Therefore, under the logic of our current `main` branch, this mount check is redundant and will cause alluxio to fail to start. This problem can be corrected after deletion.

As for `alluxio-start.sh local SudoMount` involves mount check, execute `alluxio-start.sh local SudoMount` to check whether it is mounted before. If it has been mounted before, the mount command will not be executed. If it has not been mounted before, mount it load disk. But the check mount here is redundant. This function is to force mounting before using alluxio, which does not meet the purpose of "use alluxio directly without mounting". After removing the check mount function, it does not affect the use of this command.

![截屏2023-06-14 11 57 35](https://github.com/Alluxio/alluxio/assets/132879614/4cc56dfa-8674-499f-a263-c0d672de5b21)

### Does this PR introduce any user facing changes?

No

			pr-link: Alluxio#17603
			change-id: cid-e67fe1697e4c75fbf0b565a00675f57f6136cc31
### What changes are proposed in this pull request?

Hookup permission checks in dora worker for scheduler job related endpoints

### Why are the changes needed?

To handle permission checking correctly 

### Does this PR introduce any user facing changes?

N/A
			pr-link: Alluxio#17620
			change-id: cid-7473717a206e25011d66552a09658e83f1910e67
### What changes are proposed in this pull request?

Implement http server for providing the get page RESTful API. Now we can get the page by sending a HTTP request:
`http://<HOST>:<PORT>/page?fileId=<FILE_ID>&pageIndex=<PAGE_INDEX>`

An example snapshot is shown as follows:
<img width="985" alt="image" src="https://github.com/Alluxio/alluxio/assets/6129818/b079c763-cc86-4df7-b49d-5d6cd822a85b">

### Why are the changes needed?

It can provide a more convenient API for AI scenarios and improve reading performance.

### Does this PR introduce any user facing changes?

No, it doesn't.

			pr-link: Alluxio#17579
			change-id: cid-5d0cbd8311081444fc46e1ad2799e9de04d16108
### What changes are proposed in this pull request?

Add getOpenFileHandleContainer method

### Why are the changes needed?

Add getOpenFileHandleContainer method


			pr-link: Alluxio#17618
			change-id: cid-2d29ea8c51f0c219c21b308296bfdde6410c632c
### What changes are proposed in this pull request?

Add a maven export action in `pom.xml` of `dora/tests` to export all the test classes for other classes, because the tests module provides utilities for local integration testing.

### Why are the changes needed?

For other modules to reuse the code

### Does this PR introduce any user facing changes?

No

			pr-link: Alluxio#17619
			change-id: cid-e6147b62f62464b3211134b6286f0d3c1fbb1e3e
remove the doc check in pr build since the config.yml file changes its structure.

			pr-link: Alluxio#17624
			change-id: cid-ed9092e74b1cf8677930523fc795174e44274f51
### What changes are proposed in this pull request?

Add get mount id to worker master rpc

### Why are the changes needed?

support get mount id from worker

### Does this PR introduce any user facing changes?

na

			pr-link: Alluxio#17625
			change-id: cid-b9e179fad02234bc345532bd70fdd3ec1e61dc6d
### What changes are proposed in this pull request?

Expose the getJobState API to scheduler.

### Why are the changes needed?

This is needed for EE updates policy status.
### Does this PR introduce any user facing changes?
na

			pr-link: Alluxio#17626
			change-id: cid-1c5b8bab24b76ed1a524b79b1105ae20d0aa838c
### What changes are proposed in this pull request?
- delete recognized page file, including TEMP pages
- handle empty result from getPageInfo(), which returns Optional.empty() if page is not recognized

### Why are the changes needed?

When Worker (re-)starts, worker scans the StoreDir to establish the metadata for PagingStore.
If some page files are not recognized, the original solution is to delete all metadata and all existing page files.
One special case is page files in TEMP dir. These TEMP page files are not recognized as normal page files.
And also, we don't know if these TEMP page files are valid or complete. So, we have to delete those unrecognized page files (including TEMP page files).

When this happens, we also need to handle the Optional.empty result from getPageInfo().

### Does this PR introduce any user facing changes?
N/A

			pr-link: Alluxio#17593
			change-id: cid-8889577f492e46470411eedeed80dfaa613a28aa
update with a clear example

update the style

updated based on comments

small fix

remove not used directory

update for better style and url

change the margin of url

refactor style relate code to golluxio

change the config file to match the docs files change
@Zijian-Zhu Zijian-Zhu closed this Jun 16, 2023
alluxio-bot pushed a commit that referenced this pull request Jun 16, 2023
This pr is recreated pr for #17495
			pr-link: #17630
			change-id: cid-7d98b578d2ac87d48fbcb9e0d75b41709b2de91f
@Zijian-Zhu Zijian-Zhu deleted the newdocs branch July 26, 2023 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.