after tiup destroy, some node still have tikv-xxx directory. reuse tiup deploy success. but start cluster error. #1047
Comments
|
Could you please post the destroy step and logs here? |
|
[tidb6wu@ip-172-16-5-192 ~]$ tiup cluster destroy tidb-01
|
|
[tidb6wu@ip-172-16-5-192 audit]$ tiup cluster audit fwVwHt8XCqn 2021-01-11T20:44:46+08:00 /home/tidb6wu/.tiup/components/cluster/v1.3.1/tiup-cluster destroy tidb-01 |
|
/home/tidb6wu/.tiup/components/cluster/v1.3.1/tiup-cluster destroy tidb-01 |
|
need more logs? thanks. |
From the log you provided, seems the data-dir of TiKV and other components were not deleted. So Is this a 100 percent recurring problem? If it is, could you please provide your topology.yaml file? And before the |
|
[tidb6wu@ip-172-16-5-192 ~]$ tiup cluster display tidb-01 xxx.xxx.5.121:29093 alertmanager xxx.xxx.5.121 29093/29094 linux/x86_64 Up /home/tidb6wu/tidb1-alertmanager-29093 /home/tidb6wu/tidb1-deploy/alertmanager-29093 |
|
This is this a 100 percent recurring problem. |
|
You can recrrring this problem again on yourself architecture. |
|
From the log you provided, I can probably find out the cause of the problem, and I can reproduce your problem locally with the topology as below: global:
user: tidb6wu
group: pingcap
deploy_dir: tidb1-deploy
data_dir: tidb1-data
server_configs:
tidb:
binlog.enable: true
binlog.ignore-error: false
tikv:
storage.reserve-space: 1K
pump:
storage.stop-write-at-available-space: 1 mib
pd_servers:
- host: n3
- host: n4
- host: n5
tikv_servers:
- host: n1
port: 32160
status_port: 32180
log_dir: '/home/tidb6wu/tidb1-data/tikv-32160/log'
data_dir: '/home/tidb6wu/tidb1-data/tikv-32160'
- host: n2
- host: n3Does this topology.yaml similar to yours? After deploying the cluster, the disk layout in # cd /home/tidb6wu
# tree .
.
|-- tidb1-data
| `-- tikv-32160
| |-- LOCK
| |-- db
| | |-- 000003.log
| | |-- CURRENT
| | |-- IDENTITY
| | |-- LOCK
| | |-- LOG
| | |-- MANIFEST-000001
| | |-- MANIFEST-000006
| | |-- OPTIONS-000010
| | `-- OPTIONS-000012
| |-- import
| |-- last_tikv.toml
| |-- log
| | |-- tikv.log
| | `-- tikv_stderr.log
| |-- raft
| | |-- 000003.log
| | |-- CURRENT
| | |-- IDENTITY
| | |-- LOCK
| | |-- LOG
| | |-- MANIFEST-000001
| | `-- OPTIONS-000005
| |-- snap
| `-- space_placeholder_file
`-- tidb1-deploy
|-- monitor-9100
| |-- bin
| | |-- blackbox_exporter
| | | |-- LICENSE
| | | |-- NOTICE
| | | |-- blackbox.yml
| | | `-- blackbox_exporter
| | `-- node_exporter
| | |-- LICENSE
| | |-- NOTICE
| | `-- node_exporter
| |-- conf
| | `-- blackbox.yml
| |-- scripts
| | |-- run_blackbox_exporter.sh
| | `-- run_node_exporter.sh
| |-- tidb1-data
| | `-- monitor-9100
| `-- tidb1-deploy
| `-- monitor-9100
| `-- log
| |-- blackbox_exporter.log
| `-- node_exporter.log
`-- tikv-32160
|-- bin
| `-- tikv-server
|-- conf
| `-- tikv.toml
`-- scripts
`-- run_tikv.shthe log directory is embedded in the data directory: Let's look back at our topology.yaml: tikv_servers:
- host: n1
port: 32160
status_port: 32180
log_dir: '/home/tidb6wu/tidb1-data/tikv-32160/log'
data_dir: '/home/tidb6wu/tidb1-data/tikv-32160'You SHOULD NOT embed the log directory inside data, and it's not easy to maintain in prod env. tikv_servers:
- host: n1
port: 32160
status_port: 32180
log_dir: '/home/tidb6wu/tidb1-data/tikv-32160-log'
data_dir: '/home/tidb6wu/tidb1-data/tikv-32160'or leave the eg, the node
|
|
yes, you are right. thanks. |
|
Thanks for your report, and for this usage scenario, I'll add some checking steps. |
|
@mengling2012 @9547 Thanks for investigating. Maybe we should forbid deploying cluster with log dir under data dir... It's very risky... We support a |
|
I'll file a pr to check this condition, it's wip
|
Thank you |
Bug Report
Please answer these questions before submitting your issue. Thanks!
tiup destroy cluster , then tiup deploy same cluster. tiup deploy success,but start cluster failed.
tiup start ok.
What did you see instead?
tiup start failed.
What version of TiUP are you using (
tiup --version)?tiup --version
v1.3.1 tiup
Go Version: go1.13
Git Branch: release-1.3
GitHash: d51bd0c
The text was updated successfully, but these errors were encountered: