Data Cleansing

[Cassandra-Clark]

As a user, I often encounter data with common formatting issues that need to be resolved, but which require relatively complex solutions to accommodate this. As an example, removing leading or trailing whitespace requires knowledge of a specific regular expression most commonly to resolve. We can dramatically improve the user experience for these common functions with a Data Cleansing component which encapsulates these as easy to understand descriptions of the operation, allowing multiple operations to be executed in sequence against an array of columns.

a rough outline of the proposed API is as follows:

data_cleansing Vector (Text) Vector (Integer | Text | Regex) -> Table
data_cleansing self (operations=Data_Cleansing) (columns=self.column_names) =

This should support the following operations:

• Duplicate_Whitespace
• Leading_Whitespace
• Trailing_Whitespace
• All_Whitespace
• Leading_Numbers
• Trailing_Numbers
• Punctuation
• Numbers
• Symbols
• Non_ASCII
• Tabs
• Letters
• Replace_Empty_With_Blank
• Replace_Empty_With_Zero
• Replace_Empty_With_Null
• Replace_Zero_With_Empty
• Replace_Null_With_Empty
• Replace_Blank_With_Empty
• Duplicate_Characters

These operations should execute in a consistent order, regardless of the order that the user selects them in. For example, if choosing {Punctuation, Special_Characters, Tabs}, this should execute in the same sequence as {Tabs, Special_Characters, Punctuation}.

These should operate on applicable columns within the selection, so Replace_Empty_With_Zero will apply to integer, decimal, float, but would not apply to a selected String field. This should not produce a warning.

[Cassandra-Clark]

Removed case modification as it doesn't fit the pattern of removing or replacing existing values