module 'tensorflow' has no attribute 'variable_scope'

[systats]

Hi I updated tensorflow but get this error for no-limit Texas Holdem:

AttributeError: module 'tensorflow' has no attribute 'variable_scope'

Seems like a migration issue? tensorflow/tensorflow#7285
Thanks for any advice or fixes.

Best, Sy

[daochenzha]

@systats May I know your Tensorflow version? Currently, RLCard only supports Tensorflow lower than 2.0

[systats]

Well it is 2.0 :D Thanks and great library so far!

[systats]

Now the version is 1.15 and by calling DQNAgent I get the following error

TypeError: 'int' object is not iterable

Any idea?

[daochenzha]

1.15 should be fine. Could you provide more details? Such as what Python version you are using and which line of code raises this error.

[systats]

I use python 3.7.3 and the error raises at

TypeError: 'int' object is not iterable
_build_model at dqn_agent.py#255
__init__ at dqn_agent.py#241
__init__ at dqn_agent.py#103
DQNAgent(sess, scope = "dqn", action_num = env$action_num, replay_memory_init_size = memory_init_size, train_every = train_every, state_shape = env$state_shape, mlp_layers = c(512, 512))

I am calling your package from R what works usually well. Thanks for your help!

[daochenzha]

@systats Thanks for letting me know. We didn't consider the use case of calling from R before. We didn't test the package in R either. We may need some time to figure out the problem.

Could you provide more information so that we can better reproduce the errors? It would be nice if you could provide the complete R code that calls the package, your R version, and maybe also your operating system.

[systats]

Sure, this is how far I got in reproducing this example of a Deep-Q Agent on Leduc Hold'em

tf <- reticulate::import("tensorflow")
os <- reticulate::import("os")
rlcard <- reticulate::import("rlcard")

tf$version

DQNAgent <- rlcard$agents$dqn_agent$DQNAgent
RandomAgent <- rlcard$agents$random_agent$RandomAgent
set_global_seed <- rlcard$utils$utils$set_global_seed
Logger <- rlcard$utils$logger$Logger

# Make environment
env = rlcard$make('no-limit-holdem')
eval_env = rlcard$make('no-limit-holdem')

# Set the iterations numbers and how frequently we evaluate the performance
evaluate_every = 100
evaluate_num = 1000
episode_num = 100000

# The intial memory size
memory_init_size = 1000

# Train the agent every X steps
train_every = 1

# The paths for saving the logs and learning curves
log_dir = '/Volumes/storage/MEGA/projects/pokerflow/notebooks/experiments'

# Set a global seed
set_global_seed(0L)
sess <- tf$Session()
# Initialize a global step
global_step = tf$Variable(0, name='global_step', trainable=F)

# Set up the agents
agent = DQNAgent(
  sess,
  scope='dqn',
  action_num=env$action_num,
  replay_memory_init_size = memory_init_size,
  train_every=train_every,
  state_shape=env$state_shape,
  mlp_layers=c(512, 512)
)

random_agent = RandomAgent(action_num=eval_env$action_num)
env$set_agents(list(agent, random_agent))
eval_env$set_agents(list(agent, random_agent))

# Initialize global variables
sess$run(tf$global_variables_initializer())

# Init a Logger to plot the learning curve
logger = Logger(log_dir)

### This is sourced as a script
reticulate::source_python("mypython.py")
create_poker(episode_num, env, eval_env, evaluate_num, agent, logger)

# def create_poker(episode_num, env, eval_env, evaluate_num, agent, logger):
#    for episode in range(episode_num):
#      
#      # Generate data from the environment
#      trajectories, _ = env.run(is_training = True)
# 
#      # Feed transitions into agent memory, and train the agent
#      for ts in trajectories[0]:
#          agent.feed(ts)
# 
#      # Evaluate the performance. Play with random agents.
#      if episode % evaluate_every == 0:
#          logger.log_performance(env.timestep, tournament(eval_env, evaluate_num)[0])


# Close files in the logger
logger$close_files()

# Plot the learning curve
logger$plot('DQN')

# Save model
save_dir = 'models/nolimit_holdem_dqn'
saver = tf$train$Saver()
saver$save(sess, os$path$join(save_dir, 'model'))

And my setup looks like...

sessionInfo()

R version 3.6.0 (2019-04-26)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS  10.15.1

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_3.6.0 tools_3.6.0    knitr_1.28     xfun_0.12      packrat_0.5.0  pacman_0.5.1  

Thanks for your kind support!

[daochenzha]

@systats Hi, the main reason is wrong inputs to DQN.

1. global_step = tf$Variable(0, name='global_step', trainable=F) should be global_step = tf$Variable(0L, name='global_step', trainable=F) to make sure that the step is integer

2. In state_shape=env$state_shape,, RLCard needs a list of integers. However, the R somehow interprets it as a single value. To solve this issue, I transform int to list in DQN so that the single value will also work.

3. In mlp_layers=c(512, 512), the R interfaces input a float value. However, RLCard requires 512 to be an integer. To solve this issue, I transform mlp_layers to a list of integers in DQN.

If you reclone and reinstall the code of RLCard in the master branch. The script should be able to run.

[systats]

Works fine thank you so much!

[dipampatel18]

Replace
tf.variable_scope
with
tf.compat.v1.variable_scope

[BurningTyfoon]

Replace
tf.variable_scope
with
tf.compat.v1.variable_scope

thx, it works for me!