- 🌎
Engenharia de Software & Engenharia de Dados; - 🔎
Autodidata, analítico, multidisciplinar e detalhista; - 🧐
Crio e desenvolvo documentos autorais teóricos e práticos de estudo; - 📖
Minhas referências: (Documentações oficiais, livros e cursos qualificados);
| DATA SOURCES | Data Ingestion | Data Processing | Data Storage | Workflow scheduling | Monitoring data pipelines |
|---|---|---|---|---|---|
| SGBD SQL - MySQL - Postgres - MS SQL SERVER - AWS Aurora SGBD NoSQL (document) - MongoDB - Azure Cosmos DB - ElasticSearch (data retrieval) SGBD NoSQL (key-value) - AWS DynamoDB - Redis SGBD NoSQL (wide-column) - GCP Bigtable - Apache HBase - Apache Cassandra SGBD NoSQL (graph) - AWS Neptune - Neo4j TELEMETRIA - Application logs - Device sensors FORNECEDORES DE DADOS EXTERNOS - APIs (ex: RESTful, SOAP) - Web scraping (ex: BeautifulSoup, Scrapy) |
Event Streaming - Apache Kafka (c/ RabbitMQ) - AWS Kinesis (c/ AWS SNS & SQS) - Azure Event Hubs Data Movement - Apache NiFi - StreamSets - Talend - AWS Glue Messaging - RabbitMQ - AWS SNS & SQS - Azure Service Bus |
Batch Processing: - Apache Spark - Apache Flink - Apache Hive - Apache Pig Stream Processing: - Apache Kafka - Apache Storm - AWS Kinesis |
"batch" Distributed File Systems - Hadoop HDFS - Azure Data Lake Storage "hybrid" Data Lake (Storage for Pre-processing) - AWS S3 - Azure Blob Storage Gen2 - GCP Cloud Storage "hybrid" Data Lake (Storage for Post-processing) - Delta Lake - Apache Hudi - Apache Iceberg Data Warehouses - Azure Synapse Analytics - AWS Redshift - Snowflake Data Marts - Apache Druid - Apache Pinot |
Workflow scheduling - Apache Airflow - Azure Data Factory |
Monitoring data pipelines - Datadog - Prometheus - Sentry - Grafana |