add python support for rdd apis #7
Conversation
| ping = json.dumps({ | ||
| k: v for k, v in kwargs.items() if v is not None | ||
| k: v for k, v in event.items() if v is not None |
There was a problem hiding this comment.
renamed kwargs to event to better match scala implementation
Codecov Report
@@ Coverage Diff @@
## master #7 +/- ##
=====================================
Coverage 100% 100%
=====================================
Files 5 5
Lines 164 176 +12
Branches 17 18 +1
=====================================
+ Hits 164 176 +12
Continue to review full report at Codecov.
|
relud
force-pushed
the
python_rdd
branch
3 times, most recently
from
6e6a592
to
8a05a67
Compare
| @@ -9,6 +9,8 @@ RUN dpkg -i /var/cache/apt/archives/*.deb | |||
| RUN pip install pyspark | |||
| COPY python/setup.py /app/python | |||
| RUN mkdir -p src/mozdata && pip install .[dev] && pip uninstall -y mozdata | |||
| # fix for https://github.com/spulec/moto/issues/1793 | |||
| RUN pip install 'boto3<1.8' | |||
There was a problem hiding this comment.
This isn't in the install_requires because it's only needed by tests?
| "test": { | ||
| "prefix": "test", | ||
| "metadata_prefix": "test", | ||
| "bucket": "%s" |
There was a problem hiding this comment.
| "bucket": "%s" | |
| "bucket": "{}" |
| "bucket": "%s" | ||
| } | ||
| } | ||
| """ % bucket) |
There was a problem hiding this comment.
| """ % bucket) | |
| """.format(bucket)) |
There was a problem hiding this comment.
Format strings are nicer? Although it doesn't really make a difference.
|
|
||
| def test_read_rdd(spark_fake, rdd): | ||
| # override Dataset.records to return the dataset because we don't need | ||
| # to test that .records() works and the mock.patch for boto3 breaks it |
There was a problem hiding this comment.
Even if we don't need to test records, it seems misleading to assert properties about the Dataset class instead of an actual RDD.
There was a problem hiding this comment.
Is this the error you were running into?
botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the ListObjects operation: The AWS Access Key Id you provided does not exist in our records.
There was a problem hiding this comment.
i mean, that's one of them.
There was a problem hiding this comment.
i also ran into a different exception related to pickling a non-matching class, as a result of combining mock patching and spark workers
There was a problem hiding this comment.
That's the one I got from setting boto3<1.8 and adding a spark fixture. It seems like a lot of the prep-work in the conftest fixture will go to waste :/
|
manually ran circleci tests, passed in python 2.7 and 3.7 filed #8 to deal with improving |
No description provided.