Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use of base_columns should be allowed as alias for base_column with multiple base_columns #47

Closed
ronanstokes-db opened this issue Jul 15, 2021 · 0 comments · Fixed by #56
Assignees
Labels
enhancement New feature or request

Comments

@ronanstokes-db
Copy link
Contributor

Code is present already to do this but does not work for following code snippet:

import dbldatagen as dg
from pyspark.sql.types import StructType, StructField, StringType

shuffle_partitions_requested = 8
partitions_requested = 8
data_rows = 10000000

dataspec = (dg.DataGenerator(spark, rows=10000000, partitions=8)
.withColumn("name", percent_nulls=1.0, template=r'\w \w|\w a. \w')
.withColumn("payment_instrument_type", values=['paypal', 'visa', 'mastercard', 'amex'], random=True)
.withColumn("payment_instrument", minValue=1000000, maxValue=10000000, template="dddd dddddd ddddd")
.withColumn("email", template=r'\w.\w@\w.com')
.withColumn("md5_payment_instrument",
expr="md5(concat(payment_instrument_type, ':', payment_instrument))",
base_columns=['payment_instrument_type', 'payment_instrument'])
)
df1 = dataspec.build()

df1.display()

@ronanstokes-db ronanstokes-db added the bug Something isn't working label Jul 15, 2021
@ronanstokes-db ronanstokes-db self-assigned this Jul 15, 2021
@ronanstokes-db ronanstokes-db added wontfix This will not be worked on enhancement New feature or request and removed bug Something isn't working wontfix This will not be worked on labels Jul 27, 2021
@ronanstokes-db ronanstokes-db added this to the initial-release milestone Jul 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant