serial-uartlite: Fix the unbind path

Currently the clocks are not enabled at probe but when the port is used.
Remove the unconditional disable at remove.

Fixes the below
[   77.660196] ------------[ cut here ]------------
[   77.664749] WARNING: CPU: 0 PID: 1992 at drivers/clk/clk.c:622
clk_core_disable+0x78/0x80
[   77.672892] Modules linked in:
[   77.675930] CPU: 0 PID: 1992 Comm: sh Not tainted 4.14.0 #23
[   77.681570] Hardware name: xlnx,zynqmp (DT)
[   77.685736] task: ffffffc879e2e580 task.stack: ffffff800be30000
[   77.691641] PC is at clk_core_disable+0x78/0x80

Signed-off-by: Shubhrajyoti Datta <shubhrajyoti.datta@xilinx.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>

drivers/tty/serial/uartlite.c

@@ -860,7 +860,7 @@ static int ulite_remove(struct platform_device *pdev)
 	struct uartlite_data *pdata = port->private_data;
 	int rc;
 
-	clk_disable_unprepare(pdata->clk);
+	clk_unprepare(pdata->clk);
 	rc = ulite_release(&pdev->dev);
 	pm_runtime_disable(&pdev->dev);
 	pm_runtime_set_suspended(&pdev->dev);