Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tap doesn't clean special characters in headers and hence fails #202

Open
Atif8Ted opened this issue Feb 9, 2023 · 0 comments
Open

Tap doesn't clean special characters in headers and hence fails #202

Atif8Ted opened this issue Feb 9, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@Atif8Ted
Copy link

Atif8Ted commented Feb 9, 2023

Describe the bug
A clear and concise description of what the bug is.
The Tap has no functionality to handle junk characters/special characters in headers ( that is not visible to naked eyes) . Also it doesn't remove spaces in between headers ("column name" -> column_name) to make it more suitable for databases and it doesn't remove the trailing white spaces in headers. These all are very crucial since some third-party apps exports there data directly to s3 and we don't have much say in controlling the headers. and as per the standard column names should not contain spaces or special character.

To Reproduce
Steps to reproduce the behavior:

  1. Prepare the data as '...' put non-ascii values or junk values in colunm name for example "\ufeffmessage_id" and try to load it with passing key_properties as ["message_id"], it will fail since it would not be able to find message_id
  2. Run the command '....'
  3. See error

Expected behavior
A clear and concise description of what you expected to happen.
if the column has special character, it should be removed and spaces should be replaced with _,
ex: "\ufeffmessage_id" -> "message_id"
"message id" -> "message_id"
message%%id-> message_id
Screenshots
If applicable, add screenshots to help explain your problem.

Your environment

  • Version of tap: [e.g. 2.0.0]
  • Version of python [e.g. 3.8]

Additional context
Add any other context about the problem here.

@Atif8Ted Atif8Ted added the bug Something isn't working label Feb 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant