Grok Glue Logstash Scala
Testing data products: BDD for data engineers
Apache Spark with Cucumber for Behavioral-Driven Development
Testing strategies for Apache Spark based projects
Test Driven Development in Grok
Test-Driven Development (TDD) tests with ScalaTest
AWS Glue offers an optimized Apache Parquet writer
Glue:: l'extraction-l'enrichissement- le chargement et l'organisation DataBrew
Amazon EMR to run a PySpark job using Python 3.x
Cribl: grok-patterns-library Cribl on AWS
introduction-to-logstash-grok-patterns
Glue ETL Scala:: gluecontext , AWS
Scala jobs go with Cats, Cats-Effect, Akka, Kafka, Spark
Creating a PySpark project with pytest, pyenv, and egg files
FAQs Les cibles pour Glue Elastic Views actuellement prises en charge sont Amazon Redshift, Amazon S3 et Amazon OpenSearch Service (successeur d'Amazon Elasticsearch Service). Amazon Aurora, Amazon RDS et Amazon DynamoDB seront prochainement pris en charge.
Testing PySpark DataFrame transformations
« connectionType » : « dynamodb » comme collecteur Migrating to DynamoDB using Parquet and Glue
Apache Kafka-> Glue -> Redshift
AWS Amplify, Pinpoint and Kinesis Firehose-> AWS Glue -> Spark ETL -> AWS Athena
PySpark write/read to JSON file
Glue Jupyter Glue Jupyter Glue Jupyter
Oracle to DynamoDB (Financial Ledger and Accounting Systems Hub (FLASH)). The Financial Ledger and Accounting Systems Hub (FLASH) is a suite of microservices that ingests these financial transactions, performs complex and business-critical functions to substantiate the general ledger, and generates financial statements such as balance sheet, cash flow, and income. ( moved from 90 instances of Oracle to a managed DynamoDB).
Relating AWS Schema Conversion Tool to AWS Glue ETL Actuellement, seules les conversions Oracle ETL, Microsoft SSIS et Teradata BTEQ vers AWS Glue sont pris en charge.
Evaluation of Fault Finding Ability of Assertions
SQL-Based ETL with Apache Spark on Amazon EKS Implementation Guide
ReInvent 2020: Glue Studio, Partition Indexes, Custom Connectors, DataBrew, Schema Registry
Desarrollo y pruebas locales de scripts de ETL mediante la biblioteca de ETL de AWS Glue
pip3 install --upgrade boto3 pytz tzlocal
pip show boto3
PS C:\Users\Greg> gh repo clone awslabs/aws-glue-libs fatal: destination path 'aws-glue-libs' already exists and is not an empty directory. exit status 128
================
https://issueexplorer.com/issue/awslabs/aws-glue-libs/100
I installed each prerequisites and still getting No module named 'awsglue' error.
AWS Glue version 3.0,
Apache Maven from the following location: https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-common/apache-maven-3.6.0-bin.tar.gz
AWS Glue version 3.0: https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-3.0/spark-3.1.1-amzn-0-bin-3.2.1-amzn-3.tgz
SPARK_HOME is setup
run glue-setup.sh from \\wsl$\Ubuntu-20.04\home\my_user\aws_ds\glue_libs\aws-glue-libs\bin
Please help on debbuging this as I don't know where to start else. marcin2x4 wrote this answer on 2021-10-23
Working solution:
- Make sure your Glue script is ran in the aws-glue-libs folder
- Sync jar files between
jarsv1inaws-glue-libsandjarsinyour_spark_folder(quavajar may have two versions, leave latest one) re: et Spark on WSL2: https://phoenixnap.com/kb/install-spark-on-ubuntu Remember to runglue-setup.shfromaws-glue-libs\binas a last step of Setting up Glue locally
where the content of glue-setup.sh is here https://github.com/awslabs/aws-glue-libs/blob/master/bin/glue-setup.sh
and rub from git bash is >$ bash glue-setup.sh
================ Glue in Docker (once again ) https://www.youtube.com/watch?v=2-zOXnSDdZ0
=================== EC2 policy to access S3 ========================
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:", "s3:GetBucketLocation", "s3:ListBucket", "s3:ListAllMyBuckets", "s3:GetBucketAcl", "ec2:DescribeVpcEndpoints", "ec2:DescribeRouteTables", "ec2:CreateNetworkInterface", "ec2:DeleteNetworkInterface", "ec2:DescribeNetworkInterfaces", "ec2:DescribeSecurityGroups", "ec2:DescribeSubnets", "ec2:DescribeVpcAttribute", "iam:ListRolePolicies", "iam:GetRole", "iam:GetRolePolicy", "cloudwatch:PutMetricData" ], "Resource": [ "" ] }, { "Effect": "Allow", "Action": [ "s3:CreateBucket", "s3:PutBucketPublicAccessBlock" ], "Resource": [ "arn:aws:s3:::aws-glue-" ] }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::aws-glue-/", "arn:aws:s3:::/aws-glue-/" ] }, { "Effect": "Allow", "Action": [ "s3:GetObject" ], "Resource": [ "arn:aws:s3:::crawler-public", "arn:aws:s3:::aws-glue-" ] }, { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents", "logs:AssociateKmsKey" ], "Resource": [ "arn:aws:logs:::/aws-glue/" ] }, { "Effect": "Allow", "Action": [ "ec2:CreateTags", "ec2:DeleteTags" ], "Condition": { "ForAllValues:StringEquals": { "aws:TagKeys": [ "aws-glue-service-resource" ] } }, "Resource": [ "arn:aws:ec2:::network-interface/", "arn:aws:ec2:::security-group/", "arn:aws:ec2:::instance/*" ] } ] }
=========================================