Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

EN-37698 Clarify any_failed() which was confusing #52

Merged
merged 1 commit into from
Feb 4, 2020
Merged

EN-37698 Clarify any_failed() which was confusing #52

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 28 additions & 34 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,6 @@ Python SDK for the Socrata Data Management API
- [transform](#transform)
+ [OutputSchema](#outputschema)
- [add_column](#add_column)
- [any_failed](#any_failed)
- [build_config](#build_config)
- [change_column_metadata](#change_column_metadata)
- [change_column_transform](#change_column_transform)
Expand Down Expand Up @@ -1046,15 +1045,15 @@ ArgSpec



#### [add_to_revision](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L416)
#### [add_to_revision](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L419)
```
ArgSpec
Args: revision
```

Associate this Source with the given revision.

#### [blob](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L211)
#### [blob](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L214)
```
ArgSpec
Args: file_handle
Expand Down Expand Up @@ -1124,7 +1123,7 @@ Examples:
source = source .change_parse_option('header_count').to(2) .change_parse_option('column_header').to(2) .run()
```

#### [csv](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L235)
#### [csv](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L238)
```
ArgSpec
Args: file_handle
Expand All @@ -1148,7 +1147,7 @@ Examples:
upload = upload.csv(f)
```

#### [df](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L391)
#### [df](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L394)
```
ArgSpec
Args: dataframe
Expand All @@ -1173,7 +1172,7 @@ Examples:
upload = upload.df(df)
```

#### [geojson](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L368)
#### [geojson](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L371)
```
ArgSpec
Args: file_handle
Expand All @@ -1197,7 +1196,7 @@ Examples:
upload = upload.geojson(f)
```

#### [kml](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L345)
#### [kml](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L348)
```
ArgSpec
Args: file_handle
Expand Down Expand Up @@ -1228,7 +1227,7 @@ Get a list of the operations that you can perform on this
object. These map directly onto what's returned from the API
in the `links` section of each resource

#### [load](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L180)
#### [load](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L183)


Forces the source to load, if it's a view source.
Expand All @@ -1238,12 +1237,12 @@ Returns:
Source: Returns the new Source
```

#### [open_in_browser](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L479)
#### [open_in_browser](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L482)


Open this source in your browser, this will open a window

#### [shapefile](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L323)
#### [shapefile](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L326)
```
ArgSpec
Args: file_handle
Expand All @@ -1267,7 +1266,7 @@ Examples:
upload = upload.shapefile(f)
```

#### [tsv](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L301)
#### [tsv](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L304)
```
ArgSpec
Args: file_handle
Expand All @@ -1291,7 +1290,7 @@ Examples:
upload = upload.tsv(f)
```

#### [ui_url](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L462)
#### [ui_url](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L465)


This is the URL to the landing page in the UI for the sources
Expand All @@ -1301,17 +1300,17 @@ Returns:
url (str): URL you can paste into a browser to view the source UI
```

#### [wait_for_finish](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L449)
#### [wait_for_finish](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L452)
```
ArgSpec
Args: progress, timeout, sleeptime
Defaults: progress=<function noop at 0x7f2091a387b8>, sleeptime=1
Defaults: progress=<function noop at 0x7f83097f67b8>, sleeptime=1
```

Wait for this dataset to finish transforming and validating. Accepts a progress function
and a timeout.

#### [xls](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L257)
#### [xls](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L260)
```
ArgSpec
Args: file_handle
Expand All @@ -1335,7 +1334,7 @@ Examples:
upload = upload.xls(f)
```

#### [xlsx](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L279)
#### [xlsx](https://github.com/socrata/socrata-py/blob/master//socrata/sources.py#L282)
```
ArgSpec
Args: file_handle
Expand Down Expand Up @@ -1527,7 +1526,7 @@ the

This is data as transformed from an InputSchema

#### [add_column](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L184)
#### [add_column](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L209)
```
ArgSpec
Args: field_name, display_name, transform_expr, description
Expand Down Expand Up @@ -1558,11 +1557,6 @@ new_output_schema = output
.run()
```

#### [any_failed](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L65)


Whether or not any transform in this output schema has failed

#### [build_config](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L50)
```
ArgSpec
Expand All @@ -1572,7 +1566,7 @@ ArgSpec
Create a new ImportConfig from this OutputSchema. See the API
docs for what an ImportConfig is and why they're useful

#### [change_column_metadata](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L247)
#### [change_column_metadata](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L272)
```
ArgSpec
Args: field_name, attribute
Expand Down Expand Up @@ -1604,7 +1598,7 @@ Examples:
.run()
```

#### [change_column_transform](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L277)
#### [change_column_transform](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L302)
```
ArgSpec
Args: field_name
Expand Down Expand Up @@ -1634,7 +1628,7 @@ Examples:
.run()
```

#### [drop_column](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L223)
#### [drop_column](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L248)
```
ArgSpec
Args: field_name
Expand Down Expand Up @@ -1666,7 +1660,7 @@ Get a list of the operations that you can perform on this
object. These map directly onto what's returned from the API
in the `links` section of each resource

#### [rows](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L101)
#### [rows](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L126)
```
ArgSpec
Args: offset, limit
Expand All @@ -1676,7 +1670,7 @@ ArgSpec
Get the rows for this OutputSchema. Acceps `offset` and `limit` params
for paging through the data.

#### [run](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L305)
#### [run](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L330)


Run all adds, drops, and column changes.
Expand Down Expand Up @@ -1705,7 +1699,7 @@ Examples:
.run()
```

#### [schema_errors](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L109)
#### [schema_errors](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L134)
```
ArgSpec
Args: offset, limit
Expand All @@ -1715,7 +1709,7 @@ ArgSpec
Get the errors that resulted in transforming into this output schema.
Accepts `offset` and `limit` params

#### [schema_errors_csv](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L116)
#### [schema_errors_csv](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L141)


Get the errors that results in transforming into this output schema
Expand All @@ -1724,7 +1718,7 @@ as a CSV stream.
Note that this returns a Reponse, where Reponse
is a python requests Reponse object

#### [set_row_id](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L157)
#### [set_row_id](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L182)
```
ArgSpec
Args: field_name
Expand All @@ -1747,7 +1741,7 @@ Examples:
new_output_schema = output.set_row_id('the_id_column')
```

#### [validate_row_id](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L131)
#### [validate_row_id](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L156)
```
ArgSpec
Args: field_name
Expand All @@ -1765,11 +1759,11 @@ Returns:
boolean
```

#### [wait_for_finish](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L71)
#### [wait_for_finish](https://github.com/socrata/socrata-py/blob/master//socrata/output_schema.py#L96)
```
ArgSpec
Args: progress, timeout, sleeptime
Defaults: progress=<function noop at 0x7f2091a387b8>, sleeptime=1
Defaults: progress=<function noop at 0x7f83097f67b8>, sleeptime=1
```

Wait for this dataset to finish transforming and validating. Accepts a progress function
Expand Down Expand Up @@ -1799,7 +1793,7 @@ in the `links` section of each resource
```
ArgSpec
Args: progress, timeout, sleeptime
Defaults: progress=<function noop at 0x7f2091a387b8>, sleeptime=1
Defaults: progress=<function noop at 0x7f83097f67b8>, sleeptime=1
```

Wait for this dataset to finish transforming and validating. Accepts a progress function
Expand Down
27 changes: 26 additions & 1 deletion socrata/output_schema.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,11 +63,36 @@ def build_config(self, uri, name, data_action):
return Config(self.auth, res, None)

def any_failed(self):
self.wait_for_finish()
"""
Whether or not any transform in this output schema has failed
This is probably not the function you are looking for.

This returns whether or not any transform in this output schema has failed. "Failed" in this
case means an internal error (which is unexpected), not a data error (which is expected). This
function will wait for processing to complete if it hasn't yet.

For data errors:

Tell whether or not there are data errors
output_schema.any_errors()
Get the count of data errors
output_schema.attributes['error_count']
Get the errors themselves
output_schema.schema_errors(offset = 0, limit = 20)
"""
return any([column['transform']['failed_at'] for column in self.attributes['output_columns']])


def any_errors(self):
self.wait_for_finish()
"""
Whether or not any transform returned a data error in this schema. This
function will wait for processing to complete if it hasn't yet.
"""

return self.attributes['error_count'] > 0


def wait_for_finish(self, progress = noop, timeout = None, sleeptime = 1):
"""
Wait for this dataset to finish transforming and validating. Accepts a progress function
Expand Down
2 changes: 2 additions & 0 deletions test/output_schema_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,8 @@ def test_get_errors_csv(self):

assert 'Unable to convert' in out_csv

assert output_schema.any_errors()

def test_get_rows(self):
output_schema = create_good_output_schema(self.create_input_schema())
output_schema = output_schema.wait_for_finish()
Expand Down