[Bug][DBT] Git clone fails on all but the first run because folder already exists

[himanshu-garg-zepto]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

When using an advanced blueprint based on the DBT plugin with a remote git project url, the "Git" task of DBT fails on every subsequent run.

It's because it tries to git clone the project which succeeds the first time but fails on every subsequent run due to "folder already exists".

If I manually delete the folder from within the container before running the DBT pipeline again, it works.

What do you expect to happen

I expect DBT based blueprints to not fail every time after the first

How to reproduce

• setup devlake
• add any project (github, gitlab etc)
• add a new advanced blueprint with dbt template with remote git url of a dummy dbt project
• run the blueprint (should succeed)
• run the blueprint again (should fail at the Git step)

Anything else

No response

Version

1.0

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dosubot]

Hello @himanshu-garg-zepto! I'm Dosu, your friendly bot here to help you solve bugs, answer questions, and assist you in becoming a contributor. I'm here while you wait for a human maintainer to jump in. Let's tackle this issue together!

To address the issue where the "Git" task fails on subsequent runs due to the "folder already exists" error, you can modify the Git function to check if the directory already exists before attempting to clone the repository. If the directory exists, you can skip the cloning process. Here is a modified version of the Git function:

func Git(taskCtx plugin.SubTaskContext) errors.Error {
    logger := taskCtx.GetLogger()
    data := taskCtx.GetData().(*DbtTaskData)
    if data.Options.ProjectGitURL == "" {
        return nil
    }
    
    // Check if the directory already exists
    projectDir := "your_project_directory" // Replace with the actual directory path
    if _, err := os.Stat(projectDir); !os.IsNotExist(err) {
        logger.Info("Project directory already exists, skipping clone.")
        return nil
    }
    
    cmd := exec.Command("git", "clone", data.Options.ProjectGitURL)
    logger.Info("start clone dbt project: %v", cmd)
    out, err := cmd.CombinedOutput()
    if err != nil {
        logger.Error(err, "clone dbt project failed")
        return errors.Convert(err)
    }
    logger.Info("clone dbt project success: %v", string(out))
    return nil
}

This code checks if the project directory already exists and skips the cloning process if it does, thus preventing the error ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}