Add support for reading raw data #19
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Minor corrections to manual and to setup.py long description.
Try to correct weird bug in PI Web API where read-type MAX timestamps end of interval while all the other summaries stamp start of interval by shifting all timestamps one interval down.
PI Web API returns timestamps with sub-second part for RAW, otherwise without. pd.to_datetime() crashes if format is specified wrong, so make sure both cases are handled. Different cases result in timezone naive or aware stamps, so ensure all are aware by specifying utc=True during conversion.
Slicing as used for INT and aggregated values doesn't work for reads with indeterminate lengths. Therefore moving enforcing of length for RAW into handler and changing client logic to investigate length of returned results to determine whether done or not. Should probably use similar solution for all read types, not just RAW.
Previous algorithm: Client identifies stop time for next query based on absolute stop time and max query length. Client expect handler to return exactly what is asked for. This works fine for fixed-interval time series, but not for uneven time series such as RAW. New algorithm: Client asks handler for all missing data. Handler ensures response is limited by max query length, and may return less data than what the client asked for. Upon receiving new data, the client identifies what data is still missing. This works fine for both fixed-interval and uneven time series. Also improved warning when tags are invalid and added tests.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added support for ReaderType.RAW for all handlers.
Rewrote tag reading mechanism by moving responsibility for limiting amount of data points returned from client to handler. Client now investigates length of returned result to determine start time of next query, instead of querying predetermined fixed-length intervals. This makes a more unified read algorithm for fixed-length intervals (such as INT) and uneven time series (such as RAW).
Reduced max rows per query from PI Web API to default value 10000.
Warn instead of crash when user tries to read an invalid tag.