multi_state xmv attempts to migrate data sources?

[jkstpierre]

I am trying to run some migrations within my TF CI/CD system using tfmigrate. Specifically, I am trying to move all the resources and modules from one state to another using the wildcard syntax like so:

`migration "multi_state" "xmv_alert_to_task" {
from_dir = "../workspaces/sysadmin/alert"
to_dir = "../workspaces/sysadmin/task"

actions = [
"xmv * $1"
]
}`

However, when I do so, I get a bizarre error from tfmigrate saying:

Cannot move to data.aws_secretsmanager_secret_version.cloudflare_api_token: there is already a resource instance at that address in the current state.

Data sources are not tracked in the Terraform state so this seems quite bizarre to me that tfmigrate would be trying to migrate them when using wildcard syntax. Also, since that data source it's complaining about is being generated dynamically by Terragrunt for initializing my Cloudflare provider, I can't just remove it temporarily for the sake of these migrations. This effectively forces me to not rely on the wildcard syntax as I would like to simply move all the resources from one workspace to another.

[minamijoyo]

Hi @jkstpierre, Thank you for reporting the issue!

Data sources are stored in tfstate; you can see them in tfstate as plain text and move them like resources. I have confirmed it works as expected in the sandbox environment, and I cannot reproduce your error. Could you provide the minimal reproduction case so we can dig it further?

[jkstpierre]

Oh well if that's the expected behavior then it's not a bug. The reason why it was problematic for me though is because I am using Terragrunt which I have configured to create a providers.tf file in each workspace before running Terraform commands. Inside this providers.tf file would be the data source in question because it is required for initializing the cloudflare provider. And because of this, it is impossible for me to have the data source appear in only the destination workspace's providers.tf when trying to do a full migration from one state into another like I would be able to with other resources / modules inside main.tf.

Given all this then, it would be super fantastic if there was a way to tell tfmigrate to skip migration of resources/modules/data sources that match a given pattern. That way I could still wildcard everything I want, but also have it not apply to this data source in my providers.tf.

[jkstpierre]

Also, I love your tool by the way. I managed to work around this limitation just fine and completed the migration that I was working on when I made this issue.

[minamijoyo]

What I came up with is to allow not only a simple *glob pattern match but also regular expressions. One concern is that the regular expression in the Go standard library is RE2, not PCRE, so a negative look-ahead such as ^(?!data).*$ is impossible. Thus, we can't simply write that it doesn't start with data..

Since the regex function in Terraform is also RE2, if we use the PCRE's regex engine, Terraform users would be confused. Is it possible to write an exclusion rule for data sources in the syntax of RE2? Any idea?