arrange docs after some cli parameters are deleted

Signed-off-by: Changwei Ge <gechangwei@bytedance.com>

README.md

@@ -62,8 +62,7 @@ will fork a nydusd process when necessary.
 
 ## Configure Nydus
 
-Nydus is configured by a json file which is required now. Since Nydus container images are likely stored in a registry, where auth has to be provided.
-Please follow instructions to [configure nydus](./docs/configure_nydus.md) configure Nydus in order to make it work properly in your environment.
+Please follow instructions to [configure nydus](./docs/configure_nydus.md) in order to make it work properly in your environment.
 
 ## Start Nydus Snapshotter
 
@@ -80,22 +79,16 @@ systemctl restart containerd
 Or you can start nydus-snapshotter manually.
 
 ```bash
-# `nydusd-path` is the path to nydusd binary. If `nydusd` and `nydus-image` are installed, `--nydusd-path` and `--nydusimage-path`can be omitted.
+# `nydusd-path` is the path to nydusd binary. If `nydusd` and `nydus-image` are installed and can be searched from `PATH`, `--nydusd-path` and `--nydusimage-path`can be omitted.
 # Otherwise, provide them in below command line.
 # `address` is the domain socket that you configured in containerd configuration file
-# `config-path` is the path to Nydus configuration file
+# `--nydusd-config` is the path to `nydusd` configuration file
 # The default nydus-snapshotter work directory is located at `/var/lib/containerd-nydus`
 
-$ ./containerd-nydus-grpc \
-    --config-path /etc/nydus/nydusd-config.json \
-    --address /run/containerd-nydus/containerd-nydus-grpc.sock \
-    --nydusd-path /usr/local/bin/nydusd \
-    --nydusimg-path /usr/local/bin/nydus-image \
-    --log-to-stdout
+$ sudo ./containerd-nydus-grpc --config /etc/nydus/config.toml --nydusd-config /etc/nydus/nydusd-config.json \
+--address /run/containerd-nydus/containerd-nydus-grpc.sock --log-to-stdout \
 ```
 
-You could append `--enable-stargz` to the command line above in order to enable (e)Stargz support.
-
 ### Validate Nydus-snapshotter Setup
 
 Utilize containerd's `ctr` CLI command to validate if nydus-snapshotter is set up successfully.
@@ -108,33 +101,16 @@ io.containerd.snapshotter.v1    nydus                    -              ok
 
 ## Quickly Start Container with Lazy Pulling
 
-### Start Container on Node
-
-Containerd can start container with specified snapshotter, so `nerdctl` or `ctr` needs to specify the Nydus snapshotter when start container.
+### Start Container on single Node
 
-A CLI tool [ctr-remote](https://github.com/dragonflyoss/image-service/tree/master/contrib/ctr-remote) is alongside. Use Nydus `ctr-remote` to pull Nydus image or start container based on nydus image.
+Start container using `nydus-snapshotter` using `nerdctl` which has native nydus support.
 
 ```bash
-$ sudo ctr-remote image rpull ghcr.io/dragonflyoss/image-service/nginx:nydus-latest
-fetching sha256:75002dfe... application/vnd.oci.image.manifest.v1+json
-fetching sha256:5a42e21c... application/vnd.oci.image.config.v1+json
-fetching sha256:eb1af2e1... application/vnd.oci.image.layer.v1.tar+gzip
-
-# Start container by `ctr-remote`
-$ sudo ctr-remote run --snapshotter nydus ghcr.io/dragonflyoss/image-service/nginx:nydus-latest awesome-nydus
-
 # Start container by `nerdctl`
 nerdctl --snapshotter nydus run ghcr.io/dragonflyoss/image-service/nginx:nydus-latest
 ```
 
-In addition that, `nerdctl` can now directly pull Nydus or (e)Stargz images with Nydus snapshotter without `ctr-remote` involved:
-
-```bash
-# Start an eStargz container with Nydus snapshotter by `nerdctl`
-nerdctl --snapshotter nydus run -it --rm ghcr.io/stargz-containers/fedora:35-esgz
-```
-
-### Start Container in Kubernetes
+### Start Container in Kubernetes Cluster
 
 **NOTE:** A potential drawback using CRI is that we can hardly specify snapshotter to `nydus-snapshotter`. So we have to change containerd's default snapshotter in its configuration file and enable snapshot annotations like below:
 

docs/configure_nydus.md

@@ -1,113 +1,45 @@
-# How to Configure Nydus
-
-Nydus-snapshotter can receive a toml file as Nudus-snapshotter configuration to startup Nydus-snapshotter
-through CLI option `--config-path`.The `--config-path` can also be set to a json file path as nydusd configuration.
-An example configuration looks like below.
-You can refer to the following template for configuration.
-
-```toml
-# Nydus-snapshotter startup parameter
-[snapshotter]
-address = "/run/containerd-nydus/containerd-nydus-grpc.sock"
-config_path = "/etc/nydusd-config.json" 
-root_dir  = "/var/lib/containerd-nydus"
-cache_dir  = "/var/lib/nydus/cache"
-nydusd_binary_path  = "/usr/local/bin/nydusd"
-nydus_image_binary_path = "/usr/local/bin/nydus-image"
-log_to_stdout = true
-log_level = "info"
-log_dir = "logs"
-gc_period = "24h"
-validate_signature = true
-public_key_file = "/signing/nydus-image-signing-public.key"
-convert_vpc_registry = true
-daemon_mode = "multiple"
-fs_driver = "fusedev"
-sync_remove = true
-enable_metrics = true
-metrics_file ="metrics"
-enable_stargz = true
-disable_cache_manager = false
-enable_nydus_overlay_fs = true
-nydusd_thread_num = 1
-cleanup_on_close = false
-kubeconfig_path = "$HOME/.kube/config"
-enable_kubeconfig_keychain = true
-recover_policy = "restart"
-enable_cri_keychain = false
-image_service_address = ""
-```
-
-```json
-{
-  "device": {
-    "backend": {
-      "type": "registry",
-      "config": {
-        "scheme": "https",
-        "auth": "<registry auth token>",
-        "timeout": 5,
-        "connect_timeout": 5,
-        "retry_limit": 0
-      }
-    },
-    "cache": {
-      "type": "blobcache",
-      "config": {
-        "work_dir": "/tmp/cache"
-      }
-    }
-  },
-  "mode": "direct",
-  "digest_validate": false,
-  "iostats_files": false,
-  "enable_xattr": true,
-  "fs_prefetch": {
-    "enable": true,
-    "threads_count": 6,
-    "merging_size": 131072
-  }
-}
-```
+# Configure Nydus-snapshotter
 
+Nydus-snapshotter can receive a toml file as its configurations to start providing image service through CLI parameter `--config`. A example configuration file can be found [here](./misc/snapshotter/config.toml). Besides nydus-snapshotter's configuration, `nydusd`'s configuration has to be provided to nydus-snapshotter too. Nydusd is started by nydus-snapshotter and it is configured by the provided json configuration file. A minimal configuration file can be found [here](./misc/snapshotter/nydusd-config.fusedev.json)
 
 ## Authentication
-As [contianerd#3731](https://github.com/containerd/containerd/issues/3731) discussed, containerd doesn't share credentials with third snapshotters now. Like [stargz snapshotter](https://github.com/containerd/stargz-snapshotter/blob/main/docs/overview.md#authentication), nydus-snapshotter supports 3 main ways to access registries with custom configurations. You can use config file or command line flags to enable them.
+
+As [contianerd#3731](https://github.com/containerd/containerd/issues/3731) discussed, containerd doesn't share credentials with third snapshotters now. Like [stargz snapshotter](https://github.com/containerd/stargz-snapshotter/blob/main/docs/overview.md#authentication), nydus-snapshotter supports 3 main ways to access registries with custom configurations. You can use configuration file to enable them.
 
 The snapshotter will try to get image pull keychain in the following order if such way is enabled:
 
-1. cri request
+1. intercept CRI request and extract private registry auth
 2. docker config (default enabled)
 3. k8s docker config secret
 
 ### dockerconfig-based authentication
 
-By default, the snapshotter tries to get creds from `$DOCKER_CONFIG` or `~/.docker/config.json`.
-Following example enables nydus-snapshotter to access to private registries using `docker login` command. 
+By default, the snapshotter tries to get credentials from `$DOCKER_CONFIG` or `~/.docker/config.json`.
+Following example enables nydus-snapshotter to access to private registries using `docker login` command.
 
 ```console
 # docker login
 (Enter username and password)
 # crictl pull --creds USERNAME[:PASSWORD] docker.io/<your-repository>/ubuntu:22.04
-(Here the creds is only used by containerd)
+(Here the credential is only used by containerd)
 ```
 
 ### CRI-based authentication
 
 Following configuration enables nydus-snapshotter to pull private images via CRI requests.
 
 ```toml
+
+[remote.auth]
+# Fetch the private registry auth as CRI image service proxy
 enable_cri_keychain = true
 image_service_address = "/run/containerd/containerd.sock"
 ```
 
-The equivalent command line flags are `--enable-image-proxy-keychain --image-service-address "/run/containerd/containerd.sock"`
-
 The snapshotter works as a proxy of CRI Image Service and exposes CRI Image Service API on the snapshotter's unix socket (i.e. `/run/containerd/containerd-nydus-grpc.sock`). The snapshotter acquires registry creds by scanning requests.
 The `image_service_address` is the original image service socket. It can be omitted and the default value will be the containerd's default socket path `/run/containerd/containerd.sock`.
 
-
-You **must** specify `--image-service-endpoint=unix:///run/containerd/containerd-nydus-grpc.sock` option to kubelet when using Kubernetes. Or specify `image-endpoint: "unix:////run/containerd/containerd-nydus-grpc.sock"` in `crictl.yaml` when using crictl.
+You **must** specify `--image-service-endpoint=unix:///run/containerd-nydus/containerd-nydus-grpc.sock` option to kubelet when using Kubernetes. Or specify `image-endpoint: "unix:////run/containerd-nydus/containerd-nydus-grpc.sock"` in `crictl.yaml` when using `crictl`.
 
 ### kubeconfig-based authentication
 
@@ -118,15 +50,15 @@ This is another way to enable lazy pulling of private images on Kubernetes, Nydu
 Following configuration enables nydus-snapshotter to access to private registries using kubernetes secrets in the cluster using kubeconfig files.
 
 ```toml
+[remote.auth]
 enable_kubeconfig_keychain = true
-kubeconfig_path = "$HOME/.kube/config"
+kubeconfig_path = "/path/to/.kubeconfig"
 ```
-The equivalent command line flags are `--enable-kubeconfig-keychain --kubeconfig-path "$HOME/.kube/config"`
 
 If no `kubeconfig_path` is specified, snapshotter searches kubeconfig files from `$KUBECONFIG` or `~/.kube/config`.
 
 Please note that kubeconfig-based authentication requires additional privilege (i.e. kubeconfig to list/watch secrets) to the node.
-And this doesn't work if kubelet retrieve creds from somewhere not API server (e.g. [credential provider](https://kubernetes.io/docs/tasks/kubelet-credential-provider/kubelet-credential-provider/)).
+And this doesn't work if `kubelet` retrieve credentials from somewhere not API server (e.g. [credential provider](https://kubernetes.io/docs/tasks/kubelet-credential-provider/kubelet-credential-provider/)).
 
 #### use ServiceAccount
 
@@ -184,7 +116,7 @@ spec:
 
 #### create secrets
 
-If you have logined into a private registry, you can create a secret from the config file:
+If you have logged into a private registry, you can create a secret from the config file:
 
 ```bash
 $ kubectl create --namespace nydus-system secret generic regcred \

misc/snapshotter/config.toml

@@ -46,11 +46,11 @@ address = ":9110"
 convert_vpc_registry = false
 
 [remote.auth]
-# Fetch the pricate registry auth by listening to K8s API server
+# Fetch the private registry auth by listening to K8s API server
 enable_kubeconfig_keychain = false
 # synchronize `kubernetes.io/dockerconfigjson` secret from kubernetes API server with specified kubeconfig (default `$KUBECONFIG` or `~/.kube/config`)
 kubeconfig_path = ""
-# Fetch the private regsitry auth as CRI image service proxy
+# Fetch the private registry auth as CRI image service proxy
 enable_cri_keychain = false
 # the target image service when using image proxy
 image_service_address = ""