Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strax output for pax #720

Merged
merged 14 commits into from Sep 13, 2018
Merged

Strax output for pax #720

merged 14 commits into from Sep 13, 2018

Conversation

petergaemers
Copy link
Contributor

@petergaemers petergaemers commented Aug 29, 2018

Added files for plugin to let pax output numpy arrays in strax format.

Strax uses a different output then is done by pax. This plugin gives pax the possibility to write numpy arrays as output in the format required for loading records in Strax.
To use set --output_type strax

In paxer added a line to include the WriteStrax class.
In data_model added a to_strax function

strax.py contains the WriteStrax class
strax_functions contain the functions converting to the strax format.

@tunnell
Copy link
Member

tunnell commented Aug 29, 2018

Flake8 detected 34 issues on ed8d3c9
Visit https://sider.review/gh/21694081/pull_requests/720 to review the issues.

@@ -189,6 +189,9 @@ def to_bson(self, fields_to_ignore=None, nan_to_none=False):
return bson.BSON.encode(self.to_dict(convert_numpy_arrays_to='bytes',
fields_to_ignore=fields_to_ignore,
nan_to_none=nan_to_none))

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Flake8]

[W293] blank line contains whitespace


def open(self, filename):
self.current_file = open(filename, mode='wb')

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Flake8]

[W293] blank line contains whitespace

from pax.FolderIO import WriteToFolder

class WriteStrax(WriteToFolder):

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Flake8]

[W293] blank line contains whitespace


def close(self):
self.current_file.close()

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Flake8]

[W293] blank line contains whitespace

class WriteStrax(WriteToFolder):


"""Write raw data to a folder of strax files
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Flake8]

[E303] too many blank lines (2)

for rec_i in range(n_records):
r = records[output_record_index]
r['time'] = (self.start_time
+ p.left * 10
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Flake8]

[W291] trailing whitespace

def pax_to_records(self, samples_per_record=110, events_per_chunk=10):
results = []
samples_per_record = 110

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Flake8]

[W293] blank line contains whitespace

n_records = records_needed(p.length, samples_per_record)
for rec_i in range(n_records):
r = records[output_record_index]
r['time'] = (self.start_time
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Flake8]

[W291] trailing whitespace

# There's more chunks coming, so we store a full chunk
n_store = samples_per_record
if not condition: p.length > samples_per_record * (rec_i + 1)
raise AssertionError()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Flake8]

[E113] unexpected indentation

# There's more chunks coming, so we store a full chunk
n_store = samples_per_record
if not condition: p.length > samples_per_record * (rec_i + 1)
raise AssertionError()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Flake8]

[E999] IndentationError: unexpected indent

if rec_i != n_records - 1:
# There's more chunks coming, so we store a full chunk
n_store = samples_per_record
if not condition: p.length > samples_per_record * (rec_i + 1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Flake8]

[E701] multiple statements on one line (colon)

# (that would be zero if we have to store a full record)
n_store = p.length - samples_per_record * rec_i

if not condition: 0 <= n_store <= samples_per_record
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Flake8]

[E701] multiple statements on one line (colon)

n_store = p.length - samples_per_record * rec_i

if not condition: 0 <= n_store <= samples_per_record
raise AssertionError()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Flake8]

[E113] unexpected indentation

@tunnell
Copy link
Member

tunnell commented Aug 31, 2018

Can you add a few sentences in the description above and remove the general stuff about how to make a pull request? See another pull request for an example. @petergaemers

@petergaemers petergaemers changed the title Peter Strax output for pax Aug 31, 2018
@feigaodm
Copy link
Member

@tunnell @JelleAalbers Would you like to update the review status so that we can decide when to merge it? At a later stage more analyzers can use this to learn Strax.

Copy link
Member

@tunnell tunnell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's fine. Does not affect 1T.

@tunnell tunnell merged commit 12427d3 into master Sep 13, 2018
@tunnell tunnell deleted the peter branch September 13, 2018 09:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants