CDAP-13795 clean up run resources on remote hadoop provisioner

Added some cleanup logic to the remote hadoop provisioner to
delete the run directory on the remote host when 'deprovisioning'
the cluster.

Ideally, this logic would reside within CDAP instead of in the
provisioner, as CDAP is the one that knows where it copies the
files to. However, it requires a bit more work to make sure
the cleanup logic always runs, even across master death, due to
the asynchronous nature of how run status is recorded and how
deprovisioning is kicked off. There would also ideally need to be
some way for other provisioners to tell CDAP that it does not
need to run cleanup.

cdap-runtime-ext-remote-hadoop/src/main/java/io/cdap/cdap/runtime/spi/provisioner/remote/RemoteHadoopProvisioner.java

@@ -125,7 +125,20 @@ public Cluster getClusterDetail(ProvisionerContext context, Cluster cluster) {
 
   @Override
   public void deleteCluster(ProvisionerContext context, Cluster cluster) {
-    // no-op
+    // delete jars copied over by CDAP
+    // TODO: (CDAP-13795) move this logic into the platform
+    String programName = context.getProgramRun().getProgram();
+    String runId = context.getProgramRun().getRun();
+    String remoteIP = getMasterExternalIp(cluster);
+    try (SSHSession session = createSSHSession(context, remoteIP)) {
+      LOG.debug("Cleaning up remote cluster resources for program {} run {}", programName, runId);
+      session.executeAndWait(String.format("rm -rf %s", runId));
+      LOG.debug("Completed remote cluster clean up for program {} run {}", programName, runId);
+    } catch (IOException e) {
+      LOG.warn("Unable to clean up resources for program {} run {} on the remote cluster. "
+                 + "The run directory may need to be manually deleted on cluster node {}.",
+               programName, runId, remoteIP, e);
+    }
   }
 
   @Override