Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vladiate enhancements #62

Open
wants to merge 13 commits into
base: master
Choose a base branch
from
Open

Vladiate enhancements #62

wants to merge 13 commits into from

Conversation

mmuru
Copy link

@mmuru mmuru commented Jul 28, 2018

This PR extends the current Vladiate to support our CSV file validation requirements.

The following enhancements have been added

  • Ability to overrides or accept custom field names instead of inferring them from CSV header.
  • Able to pass S3 credentials to S3File inputs.
  • Gzip compressed CSV file support for LocalFile and S3File inputs.
  • Add commonly used CSV data types validators.
  • Ability to turn off console log.

@di
Copy link
Owner

di commented Jul 31, 2018

@mmuru Can you get the CI passing? Looks like you'll need to add dateutil as a dependency at the very least.

@mmuru
Copy link
Author

mmuru commented Aug 3, 2018

@di: Added python-dateutil as a dependency and fixed flake8 lint errors. CI passed for python 2.7 but not for other python 3.x versions. I don't have a python 3 environment so any help is appreciated.

@@ -31,8 +32,13 @@ def __init__(self, filename):
self.filename = filename

def open(self):
with open(self.filename, 'r') as f:
return f.readlines()
with open(self.filename, 'rb') as f:
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mmuru I think the current test failure is because you changed the mode here from r to rb. The check for .gz extension should probably happen before trying to open the file.

@di di mentioned this pull request Mar 18, 2019
@di
Copy link
Owner

di commented Mar 18, 2019

@mmuru Are you able to get this PR passing? Or shall we close it?

@mmuru
Copy link
Author

mmuru commented Mar 19, 2019 via email

@mmuru
Copy link
Author

mmuru commented Mar 24, 2019

@di: Now this PR is passing and let me know if you have any questions otherwise merge it.

Copy link
Owner

@di di left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs documentation and tests for the new validators, and details on how to use the new arguments for S3File and Vlad.

vladiate/inputs.py Outdated Show resolved Hide resolved
vladiate/inputs.py Show resolved Hide resolved
vladiate/vlad.py Outdated Show resolved Hide resolved
vladiate/vlad.py Show resolved Hide resolved
vladiate/vlad.py Show resolved Hide resolved
@di
Copy link
Owner

di commented Mar 26, 2019

Also, if you can add a -q/--quiet argument which turns off the logging, that'd be awesome. See #53 for an example (this PR is creating more verbose logging).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants