Run python-terraform multiple times (in loop)

[dragan1979]

Hi,

i'm new to python. I created a script to iterate through JSON file to find field email address, and remove part after @. This value is username for EC2 instance which will be created using terraform.For every user number of instances will be prompted interactively. (Terraform will be invoked from python script). Everything works fine when only one email is specified in JSON,but if more than one user is specified, code will be execured only once, I need when one iteration is finishes, run again for new user (ask for number of instances), create machine then run for as many users found in JSON file

Here is my code:

`#!/bin/python
import json
from pprint import pprint
from python_terraform import *

def myfunc(int):

tf = Terraform(working_dir='/home/ja/terraform-course/demo-2b', variables={'count':enter,'INSTANCE_USERNAME':user})
tf.plan(no_color=IsFlagged, refresh=False, capture_output=True)
approve = {"auto-approve": True}
print(tf.plan())
print(tf.apply(**approve))
return

json_data=open('./my.json')
data = json.load(json_data)

json_data.close()

for i in range (0, len (data['customers'])):
#print data['customers'][i]['email']
k=data['customers'][i]['email']
#print(k.split('@')[0])
user=k.split('@')[0]
if (i) !=0:
continue
#print(user)
enter = int(input('Enter number of instances: '))
myfunc(enter)`

[beelit94]

@dragan1979 I think this is an issue relative to your python script not about this module. More appropriate place for this question would be stackoverflow. But I would love to help you look into it as long as you can provide

1. well formatted python script(above is not appropriate formatted)
2. example json data
3. expected output on your console
4. actual output on your console

Cheers

[dragan1979]

Hi, I’m forbidden to post on stack overflow because I “didn’t search enough” I found an issue, it’s on terraform side, but I had slightly to modify python script https://devops.stackexchange.com/questions/3353/create-multiple-ec2-instances-using-python-script-in-loop-iteration Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10 From: beelit94<mailto:notifications@github.com> Sent: 14. фебруар 2018 22:58 To: beelit94/python-terraform<mailto:python-terraform@noreply.github.com> Cc: dragan1979<mailto:dragan.vucanovic@hotmail.rs>; Mention<mailto:mention@noreply.github.com> Subject: Re: [beelit94/python-terraform] Run python-terraform multiple times (in loop) (#34) @dragan1979<https://github.com/dragan1979> I think this is an issue relative to your python script not about this module. More appropriate place for this question would be stackoverflow. But I would love to help you look into it as long as you can provide 1. well formatted python script(above is not appropriate formatted) 2. example json data 3. expected output on your console 4. actual output on your console Cheers — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#34 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/Ae6Vcr0MRWr2TTfOXSIuUdjgWBRWZdA5ks5tU1cegaJpZM4SFrwU>.

[dragan1979]

It's on terraform side, i had to create keypair for every new user, after that issue gone away, is this only way to use same tf file for multiple instance creations ?

[beelit94]

@dragan1979 no, you actually don't need to use python loop to create multiple instances. You can simply use "count" variable in terraform
https://github.com/terraform-providers/terraform-provider-aws/tree/master/examples/count
https://www.terraform.io/docs/configuration/resources.html#using-variables-with-count

[dragan1979]

i know it, but we're using json file to get username and number of instances, so python script reads that file,and for every user creates x instances, script works fine but facing issues that if changing something in tf file (username variable) instance got recreated, i created a module https://1drv.ms/f/s!AizscpxS0QM4ae6fU8I48xN-oF0 but same issue,if i run same code in different zone no issuues, but that's not a solution.
Is there any way not to destroy instance if variable value is different ?

[beelit94]

@dragan1979
tweaking your tf file for ec2 instance to update data of instance instead of recreating is tricky. I've spent hours as well on my tf file. You can see
hashicorp/terraform#2423
hashicorp/terraform#1887
From your tf file I think if only username changed and count not changed, it shouldn't destory after the above issue fixed(terraform 0.8.8 and after)
I would suggest you try to move user_data to template file
https://www.terraform.io/docs/providers/template/d/file.html
and try again.
If still not working, you should file a bug on terraform github.

[dragan1979]

Yes, only username variable is changed on next terraform apply, and still instance is destroyed, newest terraform version, moved user data to template file-same issue. In next version feature to prevent this behavior should be added Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10 From: beelit94<mailto:notifications@github.com> Sent: 15. фебруар 2018 20:20 To: beelit94/python-terraform<mailto:python-terraform@noreply.github.com> Cc: dragan1979<mailto:dragan.vucanovic@hotmail.rs>; Mention<mailto:mention@noreply.github.com> Subject: Re: [beelit94/python-terraform] Run python-terraform multiple times (in loop) (#34) @dragan1979<https://github.com/dragan1979> tweaking your tf file for ec2 instance to update data instance of recreate is tricky. I've spent hours as well on my tf file. You can see hashicorp/terraform#2423<hashicorp/terraform#2423> hashicorp/terraform#1887<hashicorp/terraform#1887> From your tf file I think if only username changed and count not changed, it shouldn't destory after the above issue fixed(terraform 0.8.8 and after) I would suggest you try to move user_data to template file https://www.terraform.io/docs/providers/template/d/file.html and try again. If still not working, you should file a bug on terraform github. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#34 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/Ae6VclSNyt8s6nBDdXjZq1HhvUAXXON2ks5tVIOPgaJpZM4SFrwU>.

[ghost]

finally got it working:turned out i had to create subfolder for each user (./terraform/user1,./terraform/user2....), copy all tf files to these folders,create new security group for every user and only that machines stopped recreating, for every user new machine has been created without destroying it for previous user

#!/bin/python
import json
import os.path
import shutil
from os import mkdir
from pprint import pprint
from python_terraform import *


json_data=open('./my.json')
data = json.load(json_data)

json_data.close()



def myfunc():

  tf = Terraform(working_dir=final_path, variables={'count':count,'INSTANCE_USERNAME':user})
  tf.plan(no_color=IsFlagged, refresh=True, capture_output=False)
  approve = {"auto-approve": True}
  print(tf.init(reconfigure=True))
  print(tf.plan())
  print(tf.apply(**approve))
  return







for i in range (0, len (data['customers'])):
    #print data['customers'][i]['email']
    k=data['customers'][i]['email']
    #print(k.split('@')[0])
    user=k.split('@')[0]
    #print(user)
    count=data['customers'][i]['instances']
    #print(count)
    #enter = int(input('Enter number of instances: '))
    start_path="/home/ja/terraform-course/demo-2b/"
    final_path=os.path.join(start_path,user)
    if not os.path.exists(final_path):
       os.makedirs(final_path)
    shutil.copy2('./vars.tf', final_path)
    shutil.copy2('./sg.tf', final_path)
    shutil.copy2('./windows.tf', final_path)
    shutil.copy2('./provider.tf', final_path)
    shutil.copy2('./test.txt', final_path)
    final=os.path.join(final_path,'sg.tf')
    final1=os.path.join(final_path,'windows.tf')
    with open(final, 'r') as file :
      filedata = file.read()
    filedata = filedata.replace('allow-all', user)
    with open(final, 'w') as file:
      file.write(filedata)
    with open(final1, 'r') as file :
      filedata = file.read()
    filedata = filedata.replace('allow-all', user)
    with open(final1, 'w') as file:
      file.write(filedata)
    myfunc()