Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flash minor changes (Merge to #469) #479

Merged
merged 10 commits into from
Jul 23, 2024
Merged

Conversation

zain-sohail
Copy link
Member

@zain-sohail zain-sohail commented Jul 18, 2024

Some minor refactoring. Should be merged to fix-459 and then to V1. Mainly:

  • Allowing for a auxiliary channel alias and it's subchannels. The structure is generally not my favorite but I can't think of a better way as of now. Suggestions welcome.
  • All col metadata is fetched from parquets now. Will be useful to display info later.
  • Minor structural changes to how this column metadata is represented
  • Some naming changes in the buffer module
  • Buffer module now uses dtypes to save to correct dtype (maybe nans in timed df need be dropped to get correct dtypes)

@coveralls
Copy link
Collaborator

coveralls commented Jul 18, 2024

Pull Request Test Coverage Report for Build 10064325275

Details

  • 101 of 105 (96.19%) changed or added relevant lines in 9 files are covered.
  • 1 unchanged line in 1 file lost coverage.
  • Overall coverage increased (+0.006%) to 92.688%

Changes Missing Coverage Covered Lines Changed/Added Lines %
sed/loader/flash/dataframe.py 16 20 80.0%
Files with Coverage Reduction New Missed Lines %
sed/loader/flash/dataframe.py 1 91.09%
Totals Coverage Status
Change from base Build 9987758895: 0.006%
Covered Lines: 7073
Relevant Lines: 7631

💛 - Coveralls

Copy link
Member

@rettigl rettigl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks all good to me, a few suggestions in the comments.

sed/loader/flash/dataframe.py Outdated Show resolved Hide resolved
sed/config/flash_example_config.yaml Outdated Show resolved Hide resolved
sed/loader/flash/buffer_handler.py Outdated Show resolved Hide resolved
sed/loader/flash/dataframe.py Outdated Show resolved Hide resolved
@@ -220,15 +220,16 @@ def get_elapsed_time(self, fids: Sequence[int] = None, **kwds) -> float | list[f
raise KeyError(
"File statistics missing. Use 'read_dataframe' first.",
) from exc
time_stamp_alias = self._config["dataframe"].get("time_stamp_alias", "timeStamp")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

`We should add this to the default config. It is also required e.g. in the normalization histogram function for timestamps

sed/loader/flash/utils.py Outdated Show resolved Hide resolved
try:
dtypes[channel] = channels_dict["dldAux"][channel].get("dtype")
except KeyError:
dtypes[channel] = None
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Put as default float32 here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's a bit risky if the data is coerced to 32 without users knowledge. I am not sure if a warning is provided by pandas if the data is float64.

sed/loader/flash/buffer_handler.py Outdated Show resolved Hide resolved
@zain-sohail zain-sohail merged commit 85c1315 into fix-459 Jul 23, 2024
5 checks passed
@zain-sohail zain-sohail deleted the flash-minor-changes branch July 23, 2024 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants