Removing tensorflow bottlenecks #177

nwittler · 2022-03-01T16:24:19Z

What

Fixes the control flow of the simulation to allow for efficient evaluation as compiled Tensorflow function.

Why

This removes performance bottlenecks, allowing for better scaling and parallelization.
Closes #74

How

Introducing one @tf.function decorator at the goal function level in the optimizer class. And several fixes along the way to make the code compatible with graph compilation, e.g. supported operations and function names, avoiding state dependent logic that would require retracing.

Remarks

This is a proof of concept and base for discussion. Solutions are currently bolted onto the Experiment class and should probably move to a new Propagation class. Depending on the method, a number of pre-computations might be beneficial to make the pure math portion be as lean and efficient as possible.

Checklist

Please include and complete the following checklist. Your Pull Request is (in most cases) not ready for review until the following have been completed. You can create a draft PR while you are still completing the checklist. Check the Contribution Guidelines for more details. You can mark an item as complete with the - [x] prefix

Tests - Added unit tests for new code, regression tests for bugs and updated the integration tests if required
Formatting & Linting - black and flake8 have been used to ensure styling guidelines are met
Type Annotations - All new code has been type annotated in the function signatures using type hints
Docstrings - Docstrings have been provided for functions in the numpydoc style
Documentation - The tutorial style documentation has been updated to explain changes & new features
Notebooks - Example notebooks have been updated to incorporate changes and new features
Changelog - A short note on this PR has been added to the Upcoming Release section

lazyoracle · 2022-03-01T16:35:36Z

The formatting and linting test should be fixed by merging (or preferably rebasing on) latest dev and then either amending your latest commit if you have pre-commit installed or running black c3/ if you have black installed in your virtual environment.

lazyoracle · 2022-03-01T16:38:38Z

test/test_envelope.py

@@ -47,3 +47,6 @@ def test_envelope_netzero() -> None:
        shape.numpy()
        == np.array([1.0, 1.0, 1.0, 1.0, 1.0, 0.0, -1.0, -1.0, -1.0, -1.0, -1.0, -0.0])
    )
+
+
+test_envelope_netzero()


This looks like leftover code?

nwittler · 2022-03-02T11:27:19Z

The formatting and linting test should be fixed by merging (or preferably rebasing on) latest dev and then either amending your latest commit if you have pre-commit installed or running black c3/ if you have black installed in your virtual environment.

Weird, I actually ran black manually and I have the pre-commits in the same setup I'm always using. But yes, there's a lot of rough stuff still left.

lazyoracle · 2022-03-02T11:30:24Z

The formatting and linting test should be fixed by merging (or preferably rebasing on) latest dev and then either amending your latest commit if you have pre-commit installed or running black c3/ if you have black installed in your virtual environment.

Weird, I actually ran black manually and I have the pre-commits in the same setup I'm always using. But yes, there's a lot of rough stuff still left.

You probably have an older version of black. Maybe try pip install -U black and then black c3/?

codecov · 2022-03-30T09:42:49Z

Codecov Report

Merging #177 (2815cb4) into dev (0c2a80a) will increase coverage by 0.58%.
The diff coverage is 92.23%.

@@            Coverage Diff             @@
##              dev     #177      +/-   ##
==========================================
+ Coverage   72.65%   73.24%   +0.58%     
==========================================
  Files          37       37              
  Lines        5559     5561       +2     
==========================================
+ Hits         4039     4073      +34     
+ Misses       1520     1488      -32

Impacted Files	Coverage Δ
c3/optimizers/modellearning.py	`89.82% <ø> (-0.27%)`	⬇️
c3/qiskit/c3_backend.py	`92.92% <ø> (ø)`
c3/generator/devices.py	`67.99% <79.59%> (-0.05%)`	⬇️
c3/experiment.py	`77.04% <86.95%> (+1.69%)`	⬆️
c3/c3objs.py	`84.95% <91.66%> (-1.60%)`	⬇️
c3/parametermap.py	`93.16% <95.65%> (ø)`
c3/signal/pulse.py	`81.17% <96.00%> (+2.00%)`	⬆️
c3/signal/gates.py	`90.59% <96.42%> (-0.12%)`	⬇️
c3/generator/generator.py	`91.08% <100.00%> (-0.18%)`	⬇️
c3/libraries/algorithms.py	`55.00% <100.00%> (+0.17%)`	⬆️
... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0c2a80a...2815cb4. Read the comment docs.

lazyoracle · 2022-03-30T13:24:22Z

The failing notebook - two_qubit_entangling_gate.ipynb is due to a dead kernel which might indicate scalability/compute/memory issues. I checked by running it locally and ran into the same issue. I also tried reducing the maxfun for optimization to 10 instead of 25 and the kernel continues to crash.

lazyoracle · 2022-03-31T06:28:11Z

There seems to be some memory leak issues (or is this a feature of the new TF graph mode computation?) since the two_qubit_entangling_gate.ipynb now takes 12 GB (6x) more RAM and double the time compared to the version in dev. Additionally you get the following error:

Error Trace (Click to expand)

jupyter nbconvert --to notebook --inplace --execute examples/two_qubit_entangling_gate.ipynb
[NbConvertApp] Converting notebook examples/two_qubit_entangling_gate.ipynb to notebook
Traceback (most recent call last):
  File "/home/users/anurag/miniconda3/envs/c3-py39/bin/jupyter-nbconvert", line 8, in <module>
    sys.exit(main())
  File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/jupyter_core/application.py", line 264, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/traitlets/config/application.py", line 846, in launch_instance
    app.start()
  File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/nbconvertapp.py", line 369, in start
    self.convert_notebooks()
  File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/nbconvertapp.py", line 541, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/nbconvertapp.py", line 506, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/nbconvertapp.py", line 435, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/exporters/exporter.py", line 190, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/exporters/exporter.py", line 208, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/exporters/notebook.py", line 32, in from_notebook_node
    nb_copy, resources = super().from_notebook_node(nb, resources, **kw)
  File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/exporters/exporter.py", line 147, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/exporters/exporter.py", line 334, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
  File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__
    return self.preprocess(nb, resources)
  File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/preprocessors/execute.py", line 83, in preprocess
    self.preprocess_cell(cell, resources, index)
  File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/preprocessors/execute.py", line 104, in preprocess_cell
    cell = self.execute_cell(cell, index, store_history=True)
  File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbclient/util.py", line 84, in wrapped
    return just_run(coro(*args, **kwargs))
  File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbclient/util.py", line 62, in just_run
    return loop.run_until_complete(coro)
  File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbclient/client.py", line 965, in async_execute_cell
    await self._check_raise_for_error(cell, cell_index, exec_reply)
  File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbclient/client.py", line 862, in _check_raise_for_error
    raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)
nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
------------------
parameter_map.print_parameters()
print(opt.current_best_goal)
------------------

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [29], in <cell line: 1>()
----> 1 parameter_map.print_parameters()
      2 print(opt.current_best_goal)

File ~/dev/c3/c3/parametermap.py:463, in ParameterMap.print_parameters(self, opt_map)
    459 """
    460 Print current parameters to stdout.
    461 """
    462 opt_map = self.get_opt_map(opt_map)
--> 463 print(self.str_parameters(opt_map))

File ~/dev/c3/c3/parametermap.py:450, in ParameterMap.str_parameters(self, opt_map)
    448 key = par_id
    449 par = self._pars[key]
--> 450 ret.append(f"{key:38}: {par}\n")
    451 if len(equiv_ids) > 1:
    452     for eid in equiv_ids[1:]:

File ~/dev/c3/c3/c3objs.py:225, in Quantity.__str__(self)
    224 def __str__(self):
--> 225     val = self.numpy()
    226     ret = ""
    227     for entry in np.nditer(val):

File ~/dev/c3/c3/c3objs.py:245, in Quantity.numpy(self)
    241 """
    242 Return the value of this quantity as numpy.
    243 """
    244 # TODO should be removed to be consistent with get_value
--> 245 return self.get_value().numpy() / self.pref

File ~/dev/c3/c3/c3objs.py:256, in Quantity.get_value(self)
    247 def get_value(self) -> tf.Tensor:
    248     """
    249     Return the value of this quantity as tensorflow.
    250
   (...)
    254     dtype: tf.dtypes
    255     """
--> 256     return self.scale * (self.value + 1) / 2 + self.offset

File ~/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py:153, in filter_traceback.<locals>.error_handler(*args, **kwargs)
    151 except Exception as e:
    152   filtered_tb = _process_traceback_frames(e.__traceback__)
--> 153   raise e.with_traceback(filtered_tb) from None
    154 finally:
    155   del filtered_tb

File ~/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/execute.py:54, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     52 try:
     53   ctx.ensure_initialized()
---> 54   tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     55                                       inputs, attrs, num_outputs)
     56 except core._NotOkStatusException as e:
     57   if name is not None:

TypeError: <tf.Tensor 'sub:0' shape=() dtype=float64> is out of scope and cannot be used here. Use return values, explicit Python locals or TensorFlow collections to access it.
Please see https://www.tensorflow.org/guide/function#all_outputs_of_a_tffunction_must_be_return_values for more information.

<tf.Tensor 'sub:0' shape=() dtype=float64> was defined here:
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/runpy.py", line 197, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/runpy.py", line 87, in _run_code
      exec(code, run_globals)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel_launcher.py", line 16, in <module>
      app.launch_new_instance()
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/traitlets/config/application.py", line 846, in launch_instance
      app.start()
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/kernelapp.py", line 677, in start
      self.io_loop.start()
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 199, in start
      self.asyncio_loop.run_forever()
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/asyncio/base_events.py", line 596, in run_forever
      self._run_once()
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/asyncio/base_events.py", line 1890, in _run_once
      handle._run()
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/asyncio/events.py", line 80, in _run
      self._context.run(self._callback, *self._args)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 473, in dispatch_queue
      await self.process_one()
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 462, in process_one
      await dispatch(*args)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 369, in dispatch_shell
      await result
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 664, in execute_request
      reply_content = await reply_content
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/ipkernel.py", line 355, in do_execute
      res = shell.run_cell(code, store_history=store_history, silent=silent)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/zmqshell.py", line 532, in run_cell
      return super().run_cell(*args, **kwargs)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2863, in run_cell
      result = self._run_cell(
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2909, in _run_cell
      return runner(coro)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
      coro.send(None)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3106, in run_cell_async
      has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3309, in run_ast_nodes
      if await self.run_code(code, result, async_=asy):
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3369, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File "/tmp/ipykernel_3537389/4248971210.py", line 3, in <cell line: 3>
      opt.optimize_controls()
    File "/home/users/anurag/dev/c3/c3/optimizers/optimalcontrol.py", line 147, in optimize_controls
      self.algorithm(
    File "/home/users/anurag/dev/c3/c3/libraries/algorithms.py", line 418, in lbfgs
      return minimize(
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_minimize.py", line 681, in minimize
      res = _minimize_lbfgsb(fun, x0, args, jac, bounds,
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_lbfgsb_py.py", line 308, in _minimize_lbfgsb
      sf = _prepare_scalar_function(fun, x0, jac=jac, args=args, epsilon=eps,
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_optimize.py", line 263, in _prepare_scalar_function
      sf = ScalarFunction(fun, x0, args, grad, hess,
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_differentiable_functions.py", line 158, in __init__
      self._update_fun()
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_differentiable_functions.py", line 251, in _update_fun
      self._update_fun_impl()
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_differentiable_functions.py", line 155, in update_fun
      self.f = fun_wrapped(self.x)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_differentiable_functions.py", line 137, in fun_wrapped
      fx = fun(np.copy(x), *args)
    File "/home/users/anurag/dev/c3/c3/optimizers/optimizer.py", line 286, in fct_to_min_autograd
      goal, grad = self.goal_run_with_grad(current_params)
    File "/home/users/anurag/dev/c3/c3/optimizers/optimizer.py", line 208, in goal_run_with_grad
      goal = self.goal_run(current_params)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
      return fn(*args, **kwargs)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 915, in __call__
      result = self._call(*args, **kwds)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 963, in _call
      self._initialize(args, kwds, add_initializers_to=initializers)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 785, in _initialize
      self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 2983, in _get_concrete_function_internal_garbage_collected
      graph_function, _ = self._maybe_define_function(args, kwargs)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 3292, in _maybe_define_function
      graph_function = self._create_graph_function(args, kwargs)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 3130, in _create_graph_function
      func_graph_module.func_graph_from_py_func(
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/func_graph.py", line 1161, in func_graph_from_py_func
      func_outputs = python_func(*func_args, **func_kwargs)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 677, in wrapped_fn
      out = weak_wrapped_fn().__wrapped__(*args, **kwds)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 3831, in bound_method_wrapper
      return wrapped_fn(*args, **kwargs)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/func_graph.py", line 1136, in autograph_handler
      return autograph.converted_call(
    File "/home/users/anurag/dev/c3/c3/optimizers/optimalcontrol.py", line 174, in goal_run
      self.pmap.set_parameters_scaled(current_params)
    File "/home/users/anurag/dev/c3/c3/parametermap.py", line 364, in set_parameters_scaled
      for equiv_ids in opt_map:
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 440, in for_stmt
      _py_for_stmt(iter_, extra_test, body, None, None)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 489, in _py_for_stmt
      body(target)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 455, in protected_body
      original_body(protected_iter)
    File "/home/users/anurag/dev/c3/c3/parametermap.py", line 367, in set_parameters_scaled
      for par_id in equiv_ids:
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 440, in for_stmt
      _py_for_stmt(iter_, extra_test, body, None, None)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 489, in _py_for_stmt
      body(target)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 455, in protected_body
      original_body(protected_iter)
    File "/home/users/anurag/dev/c3/c3/parametermap.py", line 371, in set_parameters_scaled
      par.set_opt_value(values[val_indx : val_indx + par_len])
    File "/home/users/anurag/dev/c3/c3/c3objs.py", line 318, in set_opt_value
      self.value = tf.acos(bound_val) / np.pi * 2 - 1
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
      return fn(*args, **kwargs)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/ops/math_ops.py", line 1406, in binary_op_wrapper
      return func(x, y, name=name)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
      return fn(*args, **kwargs)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/util/dispatch.py", line 1082, in op_dispatch_handler
      return dispatch_target(*args, **kwargs)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/ops/math_ops.py", line 548, in subtract
      return gen_math_ops.sub(x, y, name)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/ops/gen_math_ops.py", line 11139, in sub
      _, _, _op, _outputs = _op_def_library._apply_op_helper(
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/op_def_library.py", line 740, in _apply_op_helper
      op = g._create_op_internal(op_type_name, inputs, dtypes=None,
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/func_graph.py", line 693, in _create_op_internal
      return super(FuncGraph, self)._create_op_internal(  # pylint: disable=protected-access
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 3776, in _create_op_internal
      ret = Operation(
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 2175, in __init__
      self._traceback = tf_stack.extract_stack_for_node(self._c_op)

The tensor <tf.Tensor 'sub:0' shape=() dtype=float64> cannot be accessed from here, because it was defined in FuncGraph(name=goal_run, id=140493482171936), which is out of scope.
TypeError: <tf.Tensor 'sub:0' shape=() dtype=float64> is out of scope and cannot be used here. Use return values, explicit Python locals or TensorFlow collections to access it.
Please see https://www.tensorflow.org/guide/function#all_outputs_of_a_tffunction_must_be_return_values for more information.

<tf.Tensor 'sub:0' shape=() dtype=float64> was defined here:
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/runpy.py", line 197, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/runpy.py", line 87, in _run_code
      exec(code, run_globals)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel_launcher.py", line 16, in <module>
      app.launch_new_instance()
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/traitlets/config/application.py", line 846, in launch_instance
      app.start()
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/kernelapp.py", line 677, in start
      self.io_loop.start()
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 199, in start
      self.asyncio_loop.run_forever()
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/asyncio/base_events.py", line 596, in run_forever
      self._run_once()
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/asyncio/base_events.py", line 1890, in _run_once
      handle._run()
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/asyncio/events.py", line 80, in _run
      self._context.run(self._callback, *self._args)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 473, in dispatch_queue
      await self.process_one()
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 462, in process_one
      await dispatch(*args)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 369, in dispatch_shell
      await result
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 664, in execute_request
      reply_content = await reply_content
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/ipkernel.py", line 355, in do_execute
      res = shell.run_cell(code, store_history=store_history, silent=silent)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/zmqshell.py", line 532, in run_cell
      return super().run_cell(*args, **kwargs)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2863, in run_cell
      result = self._run_cell(
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2909, in _run_cell
      return runner(coro)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
      coro.send(None)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3106, in run_cell_async
      has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3309, in run_ast_nodes
      if await self.run_code(code, result, async_=asy):
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3369, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File "/tmp/ipykernel_3537389/4248971210.py", line 3, in <cell line: 3>
      opt.optimize_controls()
    File "/home/users/anurag/dev/c3/c3/optimizers/optimalcontrol.py", line 147, in optimize_controls
      self.algorithm(
    File "/home/users/anurag/dev/c3/c3/libraries/algorithms.py", line 418, in lbfgs
      return minimize(
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_minimize.py", line 681, in minimize
      res = _minimize_lbfgsb(fun, x0, args, jac, bounds,
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_lbfgsb_py.py", line 308, in _minimize_lbfgsb
      sf = _prepare_scalar_function(fun, x0, jac=jac, args=args, epsilon=eps,
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_optimize.py", line 263, in _prepare_scalar_function
      sf = ScalarFunction(fun, x0, args, grad, hess,
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_differentiable_functions.py", line 158, in __init__
      self._update_fun()
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_differentiable_functions.py", line 251, in _update_fun
      self._update_fun_impl()
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_differentiable_functions.py", line 155, in update_fun
      self.f = fun_wrapped(self.x)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_differentiable_functions.py", line 137, in fun_wrapped
      fx = fun(np.copy(x), *args)
    File "/home/users/anurag/dev/c3/c3/optimizers/optimizer.py", line 286, in fct_to_min_autograd
      goal, grad = self.goal_run_with_grad(current_params)
    File "/home/users/anurag/dev/c3/c3/optimizers/optimizer.py", line 208, in goal_run_with_grad
      goal = self.goal_run(current_params)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
      return fn(*args, **kwargs)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 915, in __call__
      result = self._call(*args, **kwds)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 963, in _call
      self._initialize(args, kwds, add_initializers_to=initializers)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 785, in _initialize
      self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 2983, in _get_concrete_function_internal_garbage_collected
      graph_function, _ = self._maybe_define_function(args, kwargs)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 3292, in _maybe_define_function
      graph_function = self._create_graph_function(args, kwargs)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 3130, in _create_graph_function
      func_graph_module.func_graph_from_py_func(
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/func_graph.py", line 1161, in func_graph_from_py_func
      func_outputs = python_func(*func_args, **func_kwargs)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 677, in wrapped_fn
      out = weak_wrapped_fn().__wrapped__(*args, **kwds)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 3831, in bound_method_wrapper
      return wrapped_fn(*args, **kwargs)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/func_graph.py", line 1136, in autograph_handler
      return autograph.converted_call(
    File "/home/users/anurag/dev/c3/c3/optimizers/optimalcontrol.py", line 174, in goal_run
      self.pmap.set_parameters_scaled(current_params)
    File "/home/users/anurag/dev/c3/c3/parametermap.py", line 364, in set_parameters_scaled
      for equiv_ids in opt_map:
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 440, in for_stmt
      _py_for_stmt(iter_, extra_test, body, None, None)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 489, in _py_for_stmt
      body(target)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 455, in protected_body
      original_body(protected_iter)
    File "/home/users/anurag/dev/c3/c3/parametermap.py", line 367, in set_parameters_scaled
      for par_id in equiv_ids:
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 440, in for_stmt
      _py_for_stmt(iter_, extra_test, body, None, None)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 489, in _py_for_stmt
      body(target)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 455, in protected_body
      original_body(protected_iter)
    File "/home/users/anurag/dev/c3/c3/parametermap.py", line 371, in set_parameters_scaled
      par.set_opt_value(values[val_indx : val_indx + par_len])
    File "/home/users/anurag/dev/c3/c3/c3objs.py", line 318, in set_opt_value
      self.value = tf.acos(bound_val) / np.pi * 2 - 1
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
      return fn(*args, **kwargs)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/ops/math_ops.py", line 1406, in binary_op_wrapper
      return func(x, y, name=name)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
      return fn(*args, **kwargs)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/util/dispatch.py", line 1082, in op_dispatch_handler
      return dispatch_target(*args, **kwargs)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/ops/math_ops.py", line 548, in subtract
      return gen_math_ops.sub(x, y, name)
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/ops/gen_math_ops.py", line 11139, in sub
      _, _, _op, _outputs = _op_def_library._apply_op_helper(
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/op_def_library.py", line 740, in _apply_op_helper
      op = g._create_op_internal(op_type_name, inputs, dtypes=None,
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/func_graph.py", line 693, in _create_op_internal
      return super(FuncGraph, self)._create_op_internal(  # pylint: disable=protected-access
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 3776, in _create_op_internal
      ret = Operation(
    File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 2175, in __init__
      self._traceback = tf_stack.extract_stack_for_node(self._c_op)

The tensor <tf.Tensor 'sub:0' shape=() dtype=float64> cannot be accessed from here, because it was defined in FuncGraph(name=goal_run, id=140493482171936), which is out of scope.

jupyter nbconvert --to notebook --inplace --execute   835.31s user 11.63s system 99% cpu 14:10.59 total

nwittler · 2022-03-31T10:24:51Z

Some increased memory requirement is expected but I don't have the experience on how much is reasonable here. I suspect the response function module to be a major cause of trouble. Removing it from the example chains might improve things. I'll check.

lgtm-com · 2022-03-31T12:59:48Z

This pull request introduces 1 alert when merging 2815cb4 into 52d0a12 - view on LGTM.com

new alerts:

1 for `__init__` method calls overridden method

lazyoracle

Ready to merge.
Agree that the pending issues in codeclimate or lgtm are better handled in a redesign/refactor work

nwittler added 18 commits January 18, 2022 14:52

Types and supported ops for graph mode.

1b3fa63

Switched test to unitary fid.

f3fd7fe

Temp disabled logging for graph mode.

6852972

Saving signal stack per channel

5975f8f

Temp disabled logging for graph optim

d2b2d1b

Refactoring

fe44058

Graph mode tweaks

8224add

Fixed device signatures, updated tests.

1b65798

Restored check for vanishing grads, storing best goal.

d8deaf2

Fixed types.

1e56804

Graph compilation doesn't like mangled names (e.g. __fieldname)

5ad7334

Removed questionable response function

2458c01

Added experimental tree multiplication.

34ccd48

Used experimental matmul

a5297b1

Binary tree matmul for @tf.function mode

fb3be66

Enabled graph mode for goal function eval.

1d5e1d4

Modifications for graph mode, to be cleaned

fa7d329

Docstring.

69b735c

pull-request-size bot added the size/XL label Mar 1, 2022

lazyoracle reviewed Mar 1, 2022

View reviewed changes

nwittler added 2 commits March 2, 2022 12:28

Merge branch 'dev' into speed

c734587

Removed left over debug line

ac67cfe

nwittler added 5 commits March 2, 2022 12:30

Re-blacking

f52430b

Removed mangling

c417c11

Merge branch 'dev' into speed

8dbf642

Added options for tree matmul and fixed signal gen to start at 0

d08e797

Deleted obsolete logging.

4fe3df0

nwittler added 8 commits March 29, 2022 15:42

Typing for devices

cafc7af

No switching based on type

8bad733

Default simulation resolution

83b5840

Deleted annoying double checks.

f8e7786

Logging of parameters

a0ac371

Comparing angular freqs.

73a8181

Switched to unitary fidelity.

ba9e153

Initial rework to enable graph mode.

7ff7770

q-optimize deleted a comment from lgtm-com bot Mar 29, 2022

nwittler added 3 commits March 30, 2022 11:32

Added explicit loading of simulation resolution

bb8293d

Saving simulation resolution to file and loading.

90de000

Logging for different param types (tf, np, list)

836f17b

nwittler added 4 commits March 31, 2022 12:43

Removed response function from chains.

2d66f1b

Prefactor in units (like 2pi)

098d46e

Loading optimized point into param map

e3f620b

Updated examples.

202fdf2

pull-request-size bot added size/XXL and removed size/XL labels Mar 31, 2022

Comparing technical frequencies.

2815cb4

q-optimize deleted a comment from lgtm-com bot Mar 31, 2022

lazyoracle approved these changes Mar 31, 2022

View reviewed changes

nwittler merged commit 381bdc3 into q-optimize:dev Apr 1, 2022

nwittler deleted the speed branch April 1, 2022 09:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Removing tensorflow bottlenecks #177

Removing tensorflow bottlenecks #177

nwittler commented Mar 1, 2022 •

edited

lazyoracle commented Mar 1, 2022

lazyoracle Mar 1, 2022

nwittler commented Mar 2, 2022

lazyoracle commented Mar 2, 2022

codecov bot commented Mar 30, 2022 •

edited

lazyoracle commented Mar 30, 2022

lazyoracle commented Mar 31, 2022 •

edited

nwittler commented Mar 31, 2022

lgtm-com bot commented Mar 31, 2022

lazyoracle left a comment

Removing tensorflow bottlenecks #177

Removing tensorflow bottlenecks #177

Conversation

nwittler commented Mar 1, 2022 • edited

What

Why

How

Remarks

Checklist

lazyoracle commented Mar 1, 2022

lazyoracle Mar 1, 2022

Choose a reason for hiding this comment

nwittler commented Mar 2, 2022

lazyoracle commented Mar 2, 2022

codecov bot commented Mar 30, 2022 • edited

Codecov Report

lazyoracle commented Mar 30, 2022

lazyoracle commented Mar 31, 2022 • edited

nwittler commented Mar 31, 2022

lgtm-com bot commented Mar 31, 2022

lazyoracle left a comment

Choose a reason for hiding this comment

nwittler commented Mar 1, 2022 •

edited

codecov bot commented Mar 30, 2022 •

edited

lazyoracle commented Mar 31, 2022 •

edited