[Frontend][Runtime] A plugin system for the backend devices

[maliasadi]

Description of the Change:

• Update the device info communication API. We pass a tuple of (rtd_name, rtd_lib, rtd_kwargs) representing the device name, the shared library path, and other specifications (number of shots, S3, ARN, etc.) from the frontend to the backend.
• Better encapsulating the runtime utility functions in the frontend by adding runtime.py.
• Re-structure the runtime to fully split the backend devices build system from the runtime CAPI and execution context manager.
• Update the initial dynamic loading support (the custom device interface) in runtime by adding the unique device factory method and simplifying the API.
• Package all 3 backend devices separately from the runtime and add them all to the new plugin system of Catalyst. Users are now able to dynamically load (custom) devices at runtime.
• Update frontend and backend tests to fully support the plugin system.
• Test the support with Address Sanitizer and gcov
• Test the support on macOS and Linux

Performance:
The new plugin system of Catalyst is backed by dlopen to load libraries at runtime instead of linking them at compile time. This approach may introduce negligible overhead depending on your QJIT workflow. I only observed a slight slowdown when loading multiple devices at runtime inside a JIT-decorated workflow.

Running all frontend tests on a c5.9xlarge machine,
Without the plugin system (backend='lightning.qubit'):

command: make pytest
number of workers: 18 [962 passed]
mean of 5 calls 374.85 sec

With the plugin system (backend='lightning.qubit'):

command: make pytest
number of workers: 18 [962 passed]
mean of 5 calls: 377.22 sec

Note that to support the plugin system on CIs, a few frontend tests were skipped. I also skipped those tests in the benchmark.

[sc-46412]

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (86ca6fe) 99.63% compared to head (ab5c257) 99.44%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #343      +/-   ##
==========================================
- Coverage   99.63%   99.44%   -0.20%     
==========================================
  Files          44       42       -2     
  Lines        7731     6981     -750     
  Branches      467      467              
==========================================
- Hits         7703     6942     -761     
- Misses         14       23       +9     
- Partials       14       16       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[josh146]

Looks good on my end, however I only reviewed the documentation so would still need additional technical review :)

[dime10]

Only some initial comments from a quick fly by so far, but I'm excited about merging this in 🤩

[dime10]

Great work 🥳 I just left some improvements to the documentation to be more specific and that should hopefully make it very clear how to create a custom device for third parties.