Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NIC not connected after cloning vm - Ubuntu 2020 Beta #421

Closed
icopanw opened this issue Apr 22, 2020 · 39 comments
Closed

NIC not connected after cloning vm - Ubuntu 2020 Beta #421

icopanw opened this issue Apr 22, 2020 · 39 comments

Comments

@icopanw
Copy link

icopanw commented Apr 22, 2020

After cloning Ubuntu 2020 vm, the NIC is not connected in esxi,
in vm settings under "Network Adapter" the checkbox "connect at power on" is checked off,
when checking the the box error message presented:
Task name
Reconfigure virtual machine
Status
Invalid configuration for device '0'

vSphere Client version 6.7.0.42000
VM Compatibility:ESXi 6.7 Update 2 and later (VM version 15)
Guest OS: Linux
Version: Ubuntu Linux 64 bit
Adapter Type: VMXNET 3
VM tools - 2 version tested, open-vm-tools and other is Esxi VMware tools
VMware tools 10.3.21 build14772444
open-vm-tools 11269 (11.0.5)

@icopanw icopanw changed the title NIC not connected after in vm clone - Ubuntu 2020 Beta NIC not connected after vm clone - Ubuntu 2020 Beta Apr 22, 2020
@icopanw icopanw changed the title NIC not connected after vm clone - Ubuntu 2020 Beta NIC not connected after cloning vm - Ubuntu 2020 Beta Apr 22, 2020
@vmwzhangv
Copy link

If you see the vm settings has the "connect at power on" checked off, it is a clone issue not an open vm tools issue.

How did you clone the VM ? Was the source VM NIC connected ?

@rolffujino
Copy link

rolffujino commented Apr 24, 2020

I have been having the similar issue with the fully released version of Ubuntu 20.04 LTS.
Going through the /var/log/vmware-imc/toolsDeployPkg.log, I have narrowed it down to being an issue with OS customization trying to use the wrong network tool:

DEBUG: Command: 'modprobe pcnet32 2> /dev/null'                                                                           
DEBUG: Result:  
DEBUG: Exit Code: 0 
DEBUG: Command: '/sbin/ifconfig eth0 2> /dev/null' 
DEBUG: Result:  
DEBUG: Exit Code: 32512 
INFO: Customizing NICS. { NIC1 }                                                                                          
INFO: Customizing NIC NIC1                                                                                                
DEBUG: Command: '/sbin/ifconfig -a'                                                                                       
DEBUG: Result:  
DEBUG: Exit Code: -1 
ERROR: Fatal error occoured during customization !! Customization halted.                                                 
ERROR: Error : Error finding the specified NIC for MAC address = 00:50:56:99:0a:e8 at /var/run/.vmware-imgcust-dHO6i3N/scripts/DebianCustomization.pm line 154.
    
INFO: Return code is 253.

/sbin/ifconfig doesn't exist in Ubuntu 20.04, funnily enough the issue doesn't exist in the 10.3.5.7752 version of open-vm-tools because it is using the ip command when I deploy an Ubuntu 18.04 VM.

@rolffujino
Copy link

To provide some logs/info, here are some logs from a successful run on Ubuntu 18.04 with open-vm-tools 10.3.5.7752:

INFO: Detected distribution flavour: Ubuntu 18.04
INFO: Checking if can run 'Customize' workflow
INFO: Calling 'Customize' method
DEBUG: Command: 'hostname 2>/dev/null'
DEBUG: Result: citd-ubuntu1804-template

DEBUG: Exit Code: 0
DEBUG: TimedCommand: 'hostname -f 2>/dev/null' with timeout of 5 sec
DEBUG: Fetching result from /tmp/timed_out_tmp_file_1252
DEBUG: TimedResult: citd-ubuntu1804-template
 
DEBUG: opening file /etc/hostname.                                                                                        
DEBUG: Match found   : Line = citd-ubuntu1804-template
                                                                                                                          
DEBUG: Actual String : citd-ubuntu1804-template 
INFO: OLD HOST NAME = citd-ubuntu1804-template                                                                            
INFO: Marker file exists or is undefined, pre-customization is not needed                                                 
INFO: Customizing Network settings ...  
INFO: Erasing DHCP leases                                                                                                 
DEBUG: Command: 'pkill dhclient' 
DEBUG: Result:  
DEBUG: Exit Code: 256                                                                                                     
DEBUG: Command: 'rm -f /var/lib/dhcp/*' 
DEBUG: Result:  
DEBUG: Exit Code: 0                                                                                                       
DEBUG: Set the host name to base001 via [hostnamectl set-hostname]                                                        
DEBUG: Command: 'hostnamectl set-hostname base001'                                                                        
DEBUG: Result:  
DEBUG: Exit Code: 0                                                                                                       
DEBUG: Host name is base001     
DEBUG: opening file for writing (/etc/hostname).                                                                          
DEBUG: Command: 'chmod 644 /etc/hostname' 
DEBUG: Result:                                                                                                            
DEBUG: Exit Code: 0 
INFO: Customizing NICS ...                                                                                                
INFO: Customizing NICS. { NIC1 } 
INFO: Customizing NIC NIC1 
DEBUG: Get interface name for MAC 00:50:56:99:8c:81, via [ip addr show] 
DEBUG: Command: 'whereis ip'    
DEBUG: Result: ip: /bin/ip /sbin/ip /usr/share/man/man7/ip.7.gz /usr/share/man/man8/ip.8.gz                               
 
DEBUG: Exit Code: 0                                                                                                       
DEBUG: Command: '/bin/ip addr show' 
DEBUG: Result: 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000                
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo                                                                                        
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host                                                                                              
       valid_lft forever preferred_lft forever
2: ens160: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000

@rolffujino
Copy link

Her are the equivalent logs from a failed Ubuntu 20.04 with the latest version of open-vm-tools from the Ubuntu repos:

DEBUG: Exit Code: 0
INFO: Detected distribution: Ubuntu Linux Distribution
DEBUG: Reading issue file ...
DEBUG: Command: 'cat /etc/issue'
DEBUG: Result: Ubuntu 20.04 LTS \n \l


DEBUG: Exit Code: 0
DEBUG: Ubuntu 20.04 LTS \n \l


INFO: Detected distribution flavour: Ubuntu
INFO: Checking if can run 'Customize' workflow
INFO: Calling 'Customize' method
DEBUG: Command: 'hostname 2>/dev/null'
DEBUG: Result: citd-ubuntu2004-template

DEBUG: Exit Code: 0
DEBUG: TimedCommand: 'hostname -f 2>/dev/null' with timeout of 5 sec
DEBUG: Fetching result from /tmp/timed_out_tmp_file_918
DEBUG: TimedResult: citd-ubuntu2004-template

DEBUG: opening file /etc/hostname.
DEBUG: Match found   : Line = citd-ubuntu2004-template

DEBUG: Actual String : citd-ubuntu2004-template
INFO: OLD HOST NAME = citd-ubuntu2004-template
INFO: Marker file exists or is undefined, pre-customization is not needed
INFO: Customizing Network settings ...
INFO: Erasing DHCP leases
DEBUG: Command: 'pkill dhclient'
DEBUG: Result:
DEBUG: Exit Code: 256
DEBUG: Command: 'rm -f /var/lib/dhcp/*'
DEBUG: Result:
DEBUG: Exit Code: 0
DEBUG: Host name is base001
DEBUG: opening file for writing (/etc/hostname).
DEBUG: Command: 'chmod 644 /etc/hostname'
DEBUG: Result:
DEBUG: Exit Code: 0
INFO: Customizing NICS ...
DEBUG: Command: 'modprobe pcnet32 2> /dev/null'
DEBUG: Result:
DEBUG: Exit Code: 0
DEBUG: Command: '/sbin/ifconfig eth0 2> /dev/null'
DEBUG: Result:
DEBUG: Exit Code: 32512
INFO: Customizing NICS. { NIC1 }
INFO: Customizing NIC NIC1
DEBUG: Command: '/sbin/ifconfig -a'
DEBUG: Result:
DEBUG: Exit Code: -1
ERROR: Fatal error occoured during customization !! Customization halted.
ERROR: Error : Error finding the specified NIC for MAC address = 00:50:56:99:0a:e8 at /var/run/.vmware-imgcust-dHO6i3N/scripts/DebianCustomization.pm line 154.

INFO: Return code is 253.
'.

[2020-04-24T05:53:29.393Z] [   error] Customization command failed with exitcode: 253, stderr: ''.

@rolffujino
Copy link

Basically, open-vm-tools or deployPkg can't recognise version 20.04 and deploys a 'default' Ubuntu customization. Unfortunately the 'default' Ubuntu solution is to use the legacy /sbin/ifconfig command which of course (it also doesn't use hostnamectl unlike the 18.04 run) breaks the customization.

Not sure if this is a open-vm-tools problem or a vCenter problem @vmwzhangv

@kradalby
Copy link

I am seeing the same on the released version.

@vmwzhangv
Copy link

Thanks, @rolffujino! This clarifies the original issue. Yes, if the guest customization is involved in a clone workflow, the VM NICs were disabled, and only get reenabled if the guest customization runs successfully. Otherwise, the cloned VM will still use the parent VM IP address, causing network issue.

Thanks for the Guest Customization logs. We will file a bug to address this issue.

@rolffujino
Copy link

rolffujino commented Apr 24, 2020

@vmwzhangv I looked through the code in this repository and it looks like the customization scripts are fed into VMware tools and not a part of it. Therefore am I correct in assuming this is an issue that would need to be resolved with a patch to vCenter? (If this is a vCenter issue you probably want to close this issue)

@vmwzhangv
Copy link

Yes, I already filed a bug today. The fix is going to be done at vCenter.

I am closing this issue. Thanks.

@rolffujino
Copy link

rolffujino commented Apr 24, 2020

Being a bit stubborn, the workaround to this problem is to replace the /etc/issue with one that looks like Ubuntu 18.04 since the customization script only looks at that file to identify the OS and what commands to run. After testing this in my environment the VM deployed successfully, so @vmwzhangv please just request them to add 'Ubuntu 20.04' as an OR to the condition for 18.04.

For those who are interested just do this in your template:

cp -vp /etc/issue{,.original}
sed -i 's/18/20/g' /etc/issue'

After you have created your VMs you should add something in Terraform or cron or where ever to revert back to 20.04.

@vmwzhangv
Copy link

This for letting us know this @rolffujino

@kradalby
Copy link

@vmwzhangv is there anywhere we can track the vCenter issue?

@PengpengSun
Copy link

Hi @icopanw , @rolffujino , @kradalby ,

Ubuntu 20.04 customization will be supported from next vSphere 6.7 patch release, and it has been supported in vSphere 7.0 release.
Ubuntu 20.04 customization is NOT supported at vSphere 6.5 releases.

@kradalby
Copy link

I use 6.5 because of my CPU’s, so I guess I have to stick to the workaround. Which works fine.

I thought this was a vCenter thing?

Or is vSphere the same as vCenter in this setting and not ESXi

@PengpengSun
Copy link

Yes, it's a vCenter change, not on server side.

@icopanw
Copy link
Author

icopanw commented Apr 26, 2020

@PengpengSun thank you for the clarification,
is there track number for the issue? the patch is scheduled for specific date?

@PengpengSun
Copy link

@icopanw , the change has been checked-in to the patch release, I will update this thread once the patch is released.

@PengpengSun
Copy link

PengpengSun commented Apr 29, 2020

Hi @icopanw , VMware vCenter Server 6.7U3g has been released today, Ubuntu 20.04 customization is supported from this release. Please find 6.7U3g release at https://my.vmware.com/web/vmware/info/slug/datacenter_cloud_infrastructure/vmware_vsphere/6_7

@kradalby
Copy link

Works for me, thanks :)

@icopanw
Copy link
Author

icopanw commented May 3, 2020

Solved After vCenter upgrade.
@PengpengSun @vmwzhangv thank you for the quick response,
@rolffujino @kradalby thank you for your analysis.
Closing issue

@adam-kosseck
Copy link

Being a bit stubborn, the workaround to this problem is to replace the /etc/issue with one that looks like Ubuntu 18.04 since the customization script only looks at that file to identify the OS and what commands to run.

@vmwzhangv If /etc/issue is really used to confirm OS, can we please get ESXi updated to use /etc/os-release instead?

In business environments /etc/issue is often replaced, and this is definitely the case with our templates, so checking this file will not work.

@vmwzhangv
Copy link

@adam-kosseck I forwarded your request to our guest customization team for an evaluation. Please wait for our response. Thanks.

@PengpengSun
Copy link

@adam-kosseck , For Ubuntu, /etc/issue is used to confirm OS version, if your env overwrites this file, customization script should get Ubuntu version from /etc/lsb-release file content.

Please share with me the /var/log/vmware-imc/toolsDeployPkg.log file from the VM you met issue.

@adam-kosseck
Copy link

If it's checking /etc/issue and falling back to /etc/lsb-release this is fine for Ubuntu machines, but won't work on Red Hat.

@PengpengSun
Copy link

@adam-kosseck For RedHat, customization script is checking /etc/issue and falling back to /etc/redhat-release.

@an-t-oine
Copy link

an-t-oine commented Jul 7, 2020

vCenter Server 6.7U3g

i don't see any reference of ubuntu 20.04 guest os support in this release

ubuntu versions supported are:
Ubuntu LTS 18.04.2 Ubuntu non-LTS 18.10 Ubuntu non LTS 15.04 Ubuntu LTS 18.04.1 Ubuntu LTS 18.04 Ubuntu 17.x Ubuntu LTS 16.04.5 Ubuntu LTS 16.04.4 Ubuntu 18.04 Desktop Ubuntu 16.04 Desktop Ubuntu 16.04 Ubuntu 14.04 Ubuntu 13.10 Ubuntu 11.x Ubuntu 10.x Ubuntu 9.x Ubuntu 8.x Ubuntu 7.x

source: https://docs.vmware.com/en/VMware-vSphere/6.7/rn/vsphere-update-manager-67u3g-release-notes.html#Tools&HW

@PengpengSun
Copy link

vCenter Server 6.7U3g

i don't see any reference of ubuntu 20.04 guest os support in this release

ubuntu versions supported are:
Ubuntu LTS 18.04.2 Ubuntu non-LTS 18.10 Ubuntu non LTS 15.04 Ubuntu LTS 18.04.1 Ubuntu LTS 18.04 Ubuntu 17.x Ubuntu LTS 16.04.5 Ubuntu LTS 16.04.4 Ubuntu 18.04 Desktop Ubuntu 16.04 Desktop Ubuntu 16.04 Ubuntu 14.04 Ubuntu 13.10 Ubuntu 11.x Ubuntu 10.x Ubuntu 9.x Ubuntu 8.x Ubuntu 7.x

source: https://docs.vmware.com/en/VMware-vSphere/6.7/rn/vsphere-update-manager-67u3g-release-notes.html#Tools&HW

@an-t-oine I believe the Guest OSes in above link is for "Supported Operating Systems for Upgrade of VMware Tools and Virtual Hardware".
Please find Guest OS Customization Support Matrix PDF at Guest OS Customization Support Matrix
Row "Ubuntu 20.04 LTS " in 6.7 column is Yes3g, it means Ubuntu 20.04 LTS customization supported from vCenter Server U3g.

@an-t-oine
Copy link

thank you for the accuracies.

@gmitch64
Copy link

Hi @icopanw , @rolffujino , @kradalby ,

Ubuntu 20.04 customization will be supported from next vSphere 6.7 patch release, and it has been supported in vSphere 7.0 release.

This IS an issue with vCenter 7.0.0.10400, though possibly slightly different - I've just checked for updates, and this is the latest version.

From the log file...

020-07-19T19:00:46 INFO: Detected distribution flavour: Ubuntu 20.04
2020-07-19T19:00:46 INFO: Customization instance Ubuntu1910Customization loaded.
2020-07-19T19:00:46 INFO: Customization started

Then we get upset.

2020-07-19T19:00:46 DEBUG: opening file /etc/hostname.
2020-07-19T19:00:46 DEBUG: Match found : Line = ubuntu-20-04-template

2020-07-19T19:00:46 DEBUG: Actual String : ubuntu-20-04-template
2020-07-19T19:00:46 INFO: OLD HOST NAME = ubuntu-20-04-template
2020-07-19T19:00:46 INFO: Reads value of the deployPkg enable-custom-scripts
2020-07-19T19:00:46 DEBUG: Command: 'vmware-toolbox-cmd config get "deployPkg" "enable-custom-scripts" 2>/tmp/guest.customization.stderr'
2020-07-19T19:00:46 DEBUG: Exit Code: 0
2020-07-19T19:00:46 DEBUG: Result: [deployPkg] enable-custom-scripts UNSET

2020-07-19T19:00:46 DEBUG: Stderr:
2020-07-19T19:00:46 DEBUG: Couldn't get value, return default value: false
2020-07-19T19:00:46 INFO: Marker file exists or is undefined, pre-customization is not needed
2020-07-19T19:00:46 INFO: Customizing Network settings ...
2020-07-19T19:00:46 INFO: Erasing DHCP leases
2020-07-19T19:00:46 DEBUG: Command: 'pkill dhclient'
2020-07-19T19:00:47 DEBUG: Exit Code: 1
2020-07-19T19:00:47 DEBUG: Result:
2020-07-19T19:00:47 DEBUG: Command: 'rm -f /var/lib/dhcp/*'
2020-07-19T19:00:47 DEBUG: Exit Code: 0
2020-07-19T19:00:47 DEBUG: Result:

And more upset

2020-07-19T19:00:48 INFO: Check if hostnamectl is available
2020-07-19T19:00:48 DEBUG: Command: 'hostnamectl status 2>/tmp/guest.customization.stderr'
2020-07-19T19:00:48 DEBUG: Exit Code: 1
2020-07-19T19:00:48 DEBUG: Result:
2020-07-19T19:00:48 DEBUG: Stderr: ^[[0;1;31mFailed to create bus connection: No such file or directory^[[0m

2020-07-19T19:00:49 INFO: Check if hostnamectl is available
2020-07-19T19:00:49 DEBUG: Command: 'hostnamectl status 2>/tmp/guest.customization.stderr'
2020-07-19T19:00:49 DEBUG: Exit Code: 1
2020-07-19T19:00:49 DEBUG: Result:
2020-07-19T19:00:49 DEBUG: Stderr: ^[[0;1;31mFailed to create bus connection: No such file or directory^[[0m

2020-07-19T19:00:50 INFO: Check if hostnamectl is available
2020-07-19T19:00:50 DEBUG: Command: 'hostnamectl status 2>/tmp/guest.customization.stderr'
2020-07-19T19:00:50 DEBUG: Exit Code: 1
2020-07-19T19:00:50 DEBUG: Result:
2020-07-19T19:00:50 DEBUG: Stderr: ^[[0;1;31mFailed to create bus connection: No such file or directory^[[0m

2020-07-19T19:00:51 INFO: Check if hostnamectl is available
2020-07-19T19:00:51 DEBUG: Command: 'hostnamectl status 2>/tmp/guest.customization.stderr'
2020-07-19T19:00:51 DEBUG: Exit Code: 1
2020-07-19T19:00:51 DEBUG: Result:
2020-07-19T19:00:51 DEBUG: Stderr: ^[[0;1;31mFailed to create bus connection: No such file or directory^[[0m

Looks like it is using the ip command ok...

2020-07-19T19:00:58 DEBUG: Command: '/usr/bin/ip addr show'
2020-07-19T19:00:58 DEBUG: Exit Code: 0
2020-07-19T19:00:58 DEBUG: Result: 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens192: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 00:50:56:b4:3e:da brd ff:ff:ff:ff:ff:ff

And then it fails out...

2020-07-19T19:01:00 DEBUG: Removing lock file /var/lock/vmware/gosc.
'.

[2020-07-19T19:01:00.574Z] [ error] Customization command failed with stderr: '^[[0;1;31mFailed to create bus connection: No such file or directory^[[0m
'.

[2020-07-19T19:01:00.574Z] [ error] Customization process returned with error.

[2020-07-19T19:01:00.574Z] [ debug] Deployment result = -1.

[2020-07-19T19:01:00.574Z] [ info] Setting 'unknown' error status in vmx.

[2020-07-19T19:01:00.575Z] [ info] Transitioning from state 'INPROGRESS' to state 'ERRORED'.

[2020-07-19T19:01:00.575Z] [ info] ENTER STATE 'ERRORED'.

[2020-07-19T19:01:00.575Z] [ info] EXIT STATE 'INPROGRESS'.

[2020-07-19T19:01:00.575Z] [ debug] Setting deploy error: 'Deployment failed.The forked off process returned error code.'.

G

@PengpengSun
Copy link

@gmitch64 This looks like Dbus.service did not run or ran too late during boot, hostnamectl command requires Dbus.service to work.
Could you please run 'systemd-analyze critical-chain dbus.service' and 'systemd-analyze critical-chain open-vm-tools.service' on the failed VM, if dbus.service did not start or started much later than open-vm-tools.service, this issue could reproduce.

@richwats
Copy link

richwats commented Dec 4, 2020

I'm seeing the same issue here. My dbus.service ran in 18s but open-vm-tools in 3.5s. Is there an easy way to force open-vm-tools to delay starting?

image

@richwats
Copy link

richwats commented Dec 4, 2020

Despite supposedly being fixed, I needed to do the workaround here to get this to work.

https://kb.vmware.com/s/article/59687

@PengpengSun
Copy link

PengpengSun commented Dec 4, 2020

Hi @richwats

May I know your Ubuntu version and type? And what's the vCenter version. Please check KB https://kb.vmware.com/s/article/54986 if you are using Ubuntu Live Server.

Details:
These systemd services have dependency:

  1. open-vm-tools.service is BEFORE cloud-init-local.service
  2. cloud-init-local.service is BEFORE cloud-init.service
  3. cloud-init-local.service is BEFORE sysinit.target
  4. dbus.service has DefaultDependencies which makes it AFTER sysinit.target

So the boot sequence in your VM is:

  1. open-vm-tools.service @ 3.529s
  2. cloud-init-local.service @ 3.535s
  3. cloud-init.service.@ 17.593s
  4. sysinit.target @ 18.426s
  5. dbus.service @ 18.466+ s

When cloud-init*.service are enabled, dbus.service will start pretty late.
If you are using traditional guest customization, please disable cloud-init*.service or remove cloud-init from your template.

@wixaw
Copy link

wixaw commented Jan 13, 2021

Despite supposedly being fixed, I needed to do the workaround here to get this to work.

https://kb.vmware.com/s/article/59687

Hello
This not fix for me
We have vSphere Client version 6.7.0.45000 and open-vm-tools 11.1.5-1

[2021-01-12T12:31:32.024Z] [   error] Customization command failed with stderr: 'Failed to create bus connection: No such file or directory
'.
[2021-01-12T12:31:32.024Z] [   error] Customization process returned with error.

it seems that dependency is ignored with "systemd-analyze dot"

Thanks

@PengpengSun
Copy link

@wixaw
Hi William, Which Linux distro are you using? could you please share the full /var/log/vmware-imc/toolsDeployPkg.log.

@wixaw
Copy link

wixaw commented Jan 14, 2021

Hello @PengpengSun
We deploy Ubuntu2004 from ansible on Centos8
toolsDeployPkg.log

Thanks

@PengpengSun
Copy link

From the log, "hostnamectl status 2" command kept failing since dbus is not running, I suspect there is cloud-init.service running which make dbus.service start late at booting. See "To use perl script as the customization engine:" section in KB https://kb.vmware.com/s/article/54986
Please follow the KB to disable or remove cloud-init, and then try again.

@wixaw
Copy link

wixaw commented Jan 14, 2021

I'm sorry because I realize that this is the same solution just above and that indeed it works well with the touch
Thanks you so much !

@PengpengSun
Copy link

No problem, I mentioned this KB on Dec 4, 2020 in this thread, I'm glad it works for you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

10 participants