Packages & Modules

General Utils

Note

The module contains several general util functions with no specific technology or SDK binding (e.g., Google SDK)

• read_file_from_path: Read a file from local path

BigQuery Connector

Note

Defines a Connector class in order to interact with BigQuery datasets and tables.

• execute_query_from_config: Execute a query from local path and with a certain set of parameter configurations. The query can either read data or create a table on BigQuery.

• table_exists: Check if a table exists in a dataset.

• wrap_dictionary_to_query_config: Converts a dictionary of Query Configurations into a BQQueryConfig object.

Data Preparation

Note

The module includes functions for implementing data transformations

• generate_embeddings: Generate the embeddings from the input texts through the method specified in embeddings_config.method.

• compress_embeddings: Compress the input embeddings with the corresponding selected method in compress_embeddings_config.method.

• encode_text: Encode an input text through embeddings and compress their dimensionality.

• extract_date_information: Extract date information from a column included in the config.column_name like the year, the month, etc.

• standardise_features: Apply the specific standardisation method in config.standardisation on the data column config.column_name.

• drop_outliers: Apply the specific drop outliers method in config.drop_outliers on the data column config.column_name

• manage_nan_values: Apply the specific drop outliers method in config.nan_values on the data column config.column_name

• prepare_numerical_features: Apply a set of transformation to the selected column in config.column_name.

• create_flag_feature: Create a flag feature from the column in config.column_name.

PostgreSQL Connector

Note

The module includes the PostgreSQL connector to interact with the PostgreSQL database.

• execute_query_from_config: Execute a query from local path and with a certain set of parameter configurations.

• schema_exists: Check if a schema exists in the database.

• tables_exists: Check if a table exists in the database.

• upload_dataframe: Upload a DataFrame to a PostgreSQL table.