Modular/Refactor FlashLoader

[zain-sohail]

Summary

In this PR, the loader is modularized. Key changes include the introduction of separate classes for distinct functionalities:

1. DataFrameCreator:

   • Restructured the process of generating dataframes, organizing them by channel format (electron, pulse, train).
   • Implemented class properties for easy access to the dataframes, each tailored to the unique logic of creating the respective dataframes.
   • Electron dataframe loading is 3x faster due to directly loading all dld channels at once from the dataset.
   • Using concatenation instead of the individual join operation before (Not tested but should be faster)
   • Notably, the electron dataframe now checks and sorts pulses for monotonicity, resulting in unique index tuples of trainId, pulseId, and electronId (This was apparently not the case).

2. BufferFileHandler:

   • Manages the creation of buffer files, offering the flexibility of serial or parallel generation as needed.
   • Conducts schema checks against the configuration file for existing buffer files.

3. ParquetHandler:

   • Handles the creation of relevant filenames for storing files. This functionality is separated to accommodate both the loader when saving dataframes (df and df_timed) and the buffer handler, which also deals with storing parquets.

Additionally:

• Breaking change: dldAux Modification:

  • Moved the treatment of dldAux to the df_train class.
  • Maybe we include a check and suggestion for users to switch to the correct format if necessary.

• Config Handling:

  • Both DataFrameCreator and BufferFileHandler now utilize the config["dataframe"] section. Hence I needed to change split_dld_time_from_sector_id that.
  • I suggest modification to split_dld_time_from_sector_id to directly accept relevant parameters, omitting the need for the entire config file.

• Tests

  • I have tried multiple times to test the individual dataframe features but testing based on data is rather difficult (there are multiple commented out tests)
  • Likely, tests for the buffer and parquet should be done and can be done much more easily.

This refactoring also addresses the referenced issues.

[coveralls]

Pull Request Test Coverage Report for Build 7535787913

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.6%) to 91.189%

---

Totals
Change from base Build 7437492641:	0.6%
Covered Lines:	5972
Relevant Lines:	6549

---

💛 - Coveralls