Experiments with end-to-end training

[jpata]

• merge GNN code from @jmduarte Benchmarking GNN #10
• retrain all baseline models with Run3 TTbar dataset
• Improved graph_data.py to prepare a more coherent set of input elements x ordered by block id, with the output candidates y_candidates padded within each block to have the same length.
• add additional tunable elements to distance matrix based on locality to improve graph connectivity for message passing approaches
• implemented an efficient block-by-block loss function inspired by awkward-array, such that we can compare the average pt,eta,phi of the true and predicted candidates within each true block
• tested various end-to-end training approaches to regress PFCandidates directly from the elements via an intermediate clustering step
  • Message Passing GNN with EdgePooling clustering+declustering (cannot overtrain, max pt correlation ~0.6....0.7)
  • GCN (as above)
  • GraphUNet (as above)
  • all to all dense baseline (can overtrain by memorizing dataset)

[jpata]

This is now ready to be merged, we have a baseline end-to-end training for elements to PFCandidates that seems to work reasonably well.

The following plots are done on 1k testing events not seen during training.

Confusion matrix of true vs predicted candidate pdgid (0 - no candidate):
confusion.pdf

Number of true vs predicted candidates per event:
num_corr.pdf

True vs predicted pt of 1000 candidates in one testing event:
pt_corr_0.pdf