Integrate native Tensorflow Learning API to C3 Model Learning

[lazyoracle]

Is your feature request related to a problem? Please describe

Current implementation of Model Learning (C_3 step) suggests duplication of code and lack of integration/reuse with existing solutions (possible case of NIH syndrome).

Describe the solution you'd like

Integrate C3 Model Learning with the Tensorflow ML ecosystem by extending Model and Layer to wrap C3 computations and state.

Advantages of this approach

1. Easy access to the whole suite of developments in the ML ecosystem in and around Tensorflow
2. Standardized way to try out various Loss Functions
3. Standardized way to try out various Optimization functions
4. Use readily available Hyperparameter Optimization tools for systematic tuning
5. Easily integrate Tensorboard and associated logging tools
6. Allow for mixed-design composable models (visualised below)
7. Provide serialised models ready to make predictions allowing better tune-up cycles
8. Easily distribute and reuse pre-trained models for prediction or fine-tuning for Transfer Learning
9. Easily and systematically experiment by turning on/off different layers, training/freezing parameters etc

Describe alternatives you've considered

1. Current implementation - possibly works but difficult to extend, iterate and experiment
2. Ditch C3 Model Learning (JK lol no)

Status/Steps

Useful refs here and here

• Define a custom layer to encapsulate the computations and state (model parameters) of the c3-tf-simulator
  • Map the parameters in (QPU + Electronics) model, eg, qubit frequency, anharmonicity, Hilbert dimensions, couplings, LO/AWG properties etc to typical neural net weights and biases (w and b)
  • Define a forward pass for our simulator layer (call()) based on the gateset and the sequences as defined in the experimental data
  • Optionally define non-trainable weights (parameters in c3 language)
  • Delayed weight creation with lazy building
  • Check if recursive layer stacking works as expected
  • Provide loss tensors to be used during training by providing an add_loss() function
  • Provide add_metric() function to track a FOM during training
• Define a custom model to expose the fit(), evaluate() and predict() (ref here)
• Check if a custom training loop is more useful (ref here)
• Optionally expose save(), save_weights() etc
• Optionally check if composing models with a mix of c3-tf-simulator and general neural network layers as defined in Tensorflow works as expected
• Explore choice of Loss Function (L1 vs L2, ref here and here)
• Explore choice of Optimization Algorithms (SGD, Adam, AdaDelta eg)

Possible Gotchas in this approach

• Given how tightly coupled state (C3-style model parameters) and computations (the actual computational operations eg matrix exponentiation) are in the current codebase, it might be tricky wrapping them up efficiently in a proper Layer style class
• There is a lot of ad-hoc hacky approaches involving the use of eg tf.function decorators without adequate reasoning, tf.Variable vs tf.Constant usage, GradientTape etc which might make integration with the standard native TF learning API possibly inefficient, buggy and difficult to keep up with TF changes and developments. However, adopting the TF API ensures we are continuously able to build on and fully tap into the large ecosystem of ML tools and solutions.