Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor inputdata #14

Merged
merged 151 commits into from
Mar 19, 2024
Merged

Refactor inputdata #14

merged 151 commits into from
Mar 19, 2024

Conversation

TCatshoek
Copy link
Contributor

This PR refactors the code that parses the input files for flexfringe.

There are now separate classes for parsing csv and abbadingo formatted files, and their code has been made more independent of each other. This should fix the issue where certain csv files could not be parsed correctly because they contained characters that were used as delimiters in the abbadingo file format.

There are still some things to do before this can be merged, as streaming mode is currently broken and I am not sure what the best way to fix it is. Discussions with @RobBa underway :)

@TCatshoek TCatshoek linked an issue Jan 2, 2023 that may be closed by this pull request
@github-actions
Copy link

github-actions bot commented Jan 2, 2023

Unit Test Results

    1 files      1 suites   1s ⏱️
  61 tests   61 ✔️ 0 💤 0
284 runs  284 ✔️ 0 💤 0

Results for commit 5ec8661.

♻️ This comment has been updated with latest results.

@TCatshoek TCatshoek self-assigned this Jan 2, 2023
Copy link

github-actions bot commented Mar 6, 2024

Unit Test Results (x64-linux)

    1 files  ±    0      1 suites  ±0   1s ⏱️ -1s
  61 tests +  57    61 ✔️ +  57  0 💤 ±0  0 ±0 
284 runs  +277  284 ✔️ +277  0 💤 ±0  0 ±0 

Results for commit 46db414. ± Comparison against base commit e1e0547.

This pull request removes 1 and adds 58 tests. Note that renamed tests count towards both.
runtests.global ‑ 2: Empty tail does not have a future
runtests.global ‑ AbbadingoReader: smoke test
runtests.global ‑ CSVHeaderParser: column name, duplicate other name
runtests.global ‑ CSVHeaderParser: column name, duplicate symbol attribute name
runtests.global ‑ CSVHeaderParser: column name, duplicate trace attribute name
runtests.global ‑ CSVHeaderParser: column name, incomplete attr spec
runtests.global ‑ CSVHeaderParser: column name, incomplete attr spec 2
runtests.global ‑ CSVHeaderParser: column name, just name
runtests.global ‑ CSVHeaderParser: column name, with attribute specifier
runtests.global ‑ CSVHeaderParser: column name, with column type specifier
runtests.global ‑ CSVHeaderParser: custom label
…

♻️ This comment has been updated with latest results.

Copy link

github-actions bot commented Mar 6, 2024

Unit Test Results (x64-osx)

    1 files  ±    0      1 suites  ±0   3s ⏱️ ±0s
  61 tests +  57    61 ✔️ +  57  0 💤 ±0  0 ±0 
284 runs  +277  284 ✔️ +277  0 💤 ±0  0 ±0 

Results for commit 46db414. ± Comparison against base commit e1e0547.

This pull request removes 1 and adds 58 tests. Note that renamed tests count towards both.
runtests.global ‑ 2: Empty tail does not have a future
runtests.global ‑ AbbadingoReader: smoke test
runtests.global ‑ CSVHeaderParser: column name, duplicate other name
runtests.global ‑ CSVHeaderParser: column name, duplicate symbol attribute name
runtests.global ‑ CSVHeaderParser: column name, duplicate trace attribute name
runtests.global ‑ CSVHeaderParser: column name, incomplete attr spec
runtests.global ‑ CSVHeaderParser: column name, incomplete attr spec 2
runtests.global ‑ CSVHeaderParser: column name, just name
runtests.global ‑ CSVHeaderParser: column name, with attribute specifier
runtests.global ‑ CSVHeaderParser: column name, with column type specifier
runtests.global ‑ CSVHeaderParser: custom label
…

♻️ This comment has been updated with latest results.

Copy link

github-actions bot commented Mar 6, 2024

Unit Test Results (x64-windows)

    1 files  ±    0      1 suites  ±0   1s ⏱️ ±0s
  61 tests +  57    61 ✔️ +  57  0 💤 ±0  0 ±0 
284 runs  +277  284 ✔️ +277  0 💤 ±0  0 ±0 

Results for commit 46db414. ± Comparison against base commit e1e0547.

This pull request removes 1 and adds 58 tests. Note that renamed tests count towards both.
runtests.exe.global ‑ 2: Empty tail does not have a future
runtests.exe.global ‑ AbbadingoReader: smoke test
runtests.exe.global ‑ CSVHeaderParser: column name, duplicate other name
runtests.exe.global ‑ CSVHeaderParser: column name, duplicate symbol attribute name
runtests.exe.global ‑ CSVHeaderParser: column name, duplicate trace attribute name
runtests.exe.global ‑ CSVHeaderParser: column name, incomplete attr spec
runtests.exe.global ‑ CSVHeaderParser: column name, incomplete attr spec 2
runtests.exe.global ‑ CSVHeaderParser: column name, just name
runtests.exe.global ‑ CSVHeaderParser: column name, with attribute specifier
runtests.exe.global ‑ CSVHeaderParser: column name, with column type specifier
runtests.exe.global ‑ CSVHeaderParser: custom label
…

♻️ This comment has been updated with latest results.

…llutes the namespace of all other files including them
…functionality is now handled upon parsing the input file
…s added by read_json aren't predicted on as well
@TCatshoek
Copy link
Contributor Author

Ok, predict seems to work now. I added a little convenience method to copy the alphabet mapping from the inputdata that gets loaded from the apta json, so we can easily create a new inputdata for the traces we want to predict that has the same mapping.

…e traces were read and predicted one by one to save on memory
@TCatshoek TCatshoek merged commit 94f415b into main Mar 19, 2024
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CSV Reader escaping special characters when parsing Web API Calls
2 participants