-
-
Notifications
You must be signed in to change notification settings - Fork 0
/
portfolio-details-8.html
369 lines (325 loc) · 26 KB
/
portfolio-details-8.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta content="width=device-width, initial-scale=1.0" name="viewport">
<title>Portfolio Details</title>
<meta content="" name="description">
<meta content="" name="keywords">
<!-- Favicons -->
<link href="assets/img/Favicon-1.png" rel="icon">
<link href="assets/img/Favicon-1.png" rel="apple-touch-icon">
<!-- Google Fonts -->
<link href="https://fonts.googleapis.com/css?family=Open+Sans:300,300i,400,400i,600,600i,700,700i|Raleway:300,300i,400,400i,500,500i,600,600i,700,700i|Poppins:300,300i,400,400i,500,500i,600,600i,700,700i" rel="stylesheet">
<!-- Vendor CSS Files -->
<link href="assets/vendor/aos/aos.css" rel="stylesheet">
<link href="assets/vendor/bootstrap/css/bootstrap.min.css" rel="stylesheet">
<link href="assets/vendor/bootstrap-icons/bootstrap-icons.css" rel="stylesheet">
<link href="assets/vendor/boxicons/css/boxicons.min.css" rel="stylesheet">
<link href="assets/vendor/glightbox/css/glightbox.min.css" rel="stylesheet">
<link href="assets/vendor/swiper/swiper-bundle.min.css" rel="stylesheet">
<!-- Creating a python code section-->
<link rel="stylesheet" href="assets/css/prism.css">
<script src="assets/js/prism.js"></script>
<!-- Template Main CSS File -->
<link href="assets/css/style.css" rel="stylesheet">
<!-- To set the icon, visit https://fontawesome.com/account-->
<script src="https://kit.fontawesome.com/5d25c1efd3.js" crossorigin="anonymous"></script>
<!-- end of icon-->
<!-- =======================================================
* Template Name: iPortfolio
* Updated: Sep 18 2023 with Bootstrap v5.3.2
* Template URL: https://bootstrapmade.com/iportfolio-bootstrap-portfolio-websites-template/
* Author: BootstrapMade.com
* License: https://bootstrapmade.com/license/
======================================================== -->
</head>
<body>
<!-- ======= Mobile nav toggle button ======= -->
<i class="bi bi-list mobile-nav-toggle d-xl-none"></i>
<!-- ======= Header ======= -->
<header id="header">
<div class="d-flex flex-column">
<div class="profile">
<img src="assets/img/myphoto.jpeg" alt="" class="img-fluid rounded-circle">
<h1 class="text-light"><a href="index.html">Arun</a></h1>
<div class="social-links mt-3 text-center">
<a href="https://www.linkedin.com/in/arunp77/" target="_blank" class="linkedin"><i class="bx bxl-linkedin"></i></a>
<a href="https://github.com/arunp77" target="_blank" class="github"><i class="bx bxl-github"></i></a>
<a href="https://twitter.com/arunp77_" target="_blank" class="twitter"><i class="bx bxl-twitter"></i></a>
<a href="https://www.instagram.com/arunp77/" target="_blank" class="instagram"><i class="bx bxl-instagram"></i></a>
<a href="https://arunp77.medium.com/" target="_blank" class="medium"><i class="bx bxl-medium"></i></a>
</div>
</div>
<nav id="navbar" class="nav-menu navbar">
<ul>
<li><a href="index.html#hero" class="nav-link scrollto active"><i class="bx bx-home"></i> <span>Home</span></a></li>
<li><a href="index.html#about" class="nav-link scrollto"><i class="bx bx-user"></i> <span>About</span></a></li>
<li><a href="index.html#resume" class="nav-link scrollto"><i class="bx bx-file-blank"></i> <span>Resume</span></a></li>
<li><a href="index.html#portfolio" class="nav-link scrollto"><i class="bx bx-book-content"></i> <span>Portfolio</span></a></li>
<li><a href="index.html#skills-and-tools" class="nav-link scrollto"><i class="bx bx-wrench"></i> <span>Skills and Tools</span></a></li>
<!-- <li><a href="index.html#services" class="nav-link scrollto"><i class="bx bx-server"></i> <span>Services</span></a></li>-->
<li><a href="index.html#professionalcourses" class="nav-link scrollto"><i class="bx bx-book-alt"></i> <span>Professional Certification</span></a></li>
<li><a href="index.html#publications" class="nav-link scrollto"><i class="bx bx-news"></i> <span>Publications</span></a></li>
<li><a href="index.html#extra-curricular" class="nav-link scrollto"><i class="bx bx-rocket"></i> <span>Extra-Curricular Activities</span></a></li>
<li><a href="index.html#contact" class="nav-link scrollto"><i class="bx bx-envelope"></i> <span>Contact</span></a></li>
</ul>
</nav><!-- .nav-menu -->
</div>
</header><!-- End Header -->
<main id="main">
<!-- ======= Breadcrumbs ======= -->
<section id="breadcrumbs" class="breadcrumbs">
<div class="container">
<!-- ======= Breadcrumbs ======= -->
<div class="d-flex justify-content-between align-items-center">
<h2>Portfolio Details</h2>
<ol>
<li><a href="Data-engineering.html" class="clickable-box">Content</a></li>
<li><a href="index.html#portfolio" class="clickable-box">Portfolio</a></li>
</ol>
</div>
</div>
</section><!-- End Breadcrumbs -->
<!------ right dropdown menue ------->
<div class="right-side-list">
<div class="dropdown">
<button class="dropbtn"><strong>Shortcuts:</strong></button>
<div class="dropdown-content">
<ul>
<li><a href="cloud-compute.html"><i class="fas fa-cloud"></i> Cloud</a></li>
<li><a href="AWS-GCP.html"><i class="fas fa-cloud"></i> AWS-GCP</a></li>
<li><a href="amazon-s3.html"><i class="fas fa-cloud"></i> AWS S3</a></li>
<li><a href="ec2-confi.html"><i class="fas fa-server"></i> EC2</a></li>
<li><a href="Docker-Container.html"><i class="fab fa-docker" style="color: rgb(2, 2, 2);"></i> Docker</a></li>
<li><a href="Jupyter-nifi.html"><i class="fab fa-python" style="color: rgb(5, 5, 5);"></i> Jupyter-nifi</a></li>
<li><a href="snowflake-task-stream.html"><i class="fas fa-snowflake"></i> Snowflake</a></li>
<li><a href="data-model.html"><i class="fas fa-database"></i> Data modeling</a></li>
<li><a href="sql-basics.html"><i class="fas fa-table"></i> QL</a></li>
<li><a href="sql-basic-details.html"><i class="fas fa-database"></i> SQL</a></li>
<li><a href="Bigquerry-sql.html"><i class="fas fa-database"></i> Bigquerry</a></li>
<li><a href="scd.html"><i class="fas fa-archive"></i> SCD</a></li>
<li><a href="sql-project.html"><i class="fas fa-database"></i> SQL project</a></li>
<!-- Add more subsections as needed -->
</ul>
</div>
</div>
</div>
<!-- ======= Portfolio Details Section ======= -->
<section id="portfolio-details" class="portfolio-details">
<div class="container">
<div class="row gy-4">
<h1>Automated Real-Time Data Streaming Pipeline using Apache Nifi, AWS, Snowpipe, Stream & Task</h1>
<i>"Unlocking the Power of Real-Time Data in the Cloud"</i>
<h3><b>Introduction</b></h3>
<p>In this project, I embarked on an exciting adventure in data engineering. I orchestrated the real-time flow of data using a fusion of modern technologies,
aiming to build a smart system. This system not only generates random data but also swiftly sends it to an AWS S3 storage space. Then, it processes this data
instantly with Snowflake, all under the watchful eye of Apache Nifi, which acts as the conductor of this intricate data symphony.
</p>
<div class="flex-container">
<div class="text">
<h3><b>Prerequisites</b></h3>
<p>Before diving into the project, you'll need the following prerequisites:</p>
<ul style="list-style-type: disc; margin-left: 30px;">
<li>Access to <a href="amazon-s3.html">AWS services</a>.</li>
<li>An <a href="https://docs.aws.amazon.com/ec2/" target="_blank">EC2 instance</a> with specific configurations: 8GB RAM, 100GB memory space.</li>
<li><a href="https://docs.docker.com/desktop/" target="_blank">Docker container</a> expertise.</li>
<li>Proficiency in <a href="https://github.com/arunp77/Python-programming" target="_blank"> Python</a>, including the
'<a href="https://faker.readthedocs.io/en/master/" target="_blank">Faker</a>' library.</li>
<li>Familiarity with <a href="https://docs.snowflake.com/en/user-guide/intro-key-concepts" target="_blank">Snowflake for cloud data warehousing</a>.</li>
<li><a href="https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-zookeeper.html" target="_blank">Basic understaidng of Apache ZooKeeper</a>.</li>
<li><a href="https://nifi.apache.org/docs.html" target="_blank">Apache Nifi</a> setup and configuration.</li>
<li><a href="https://quickstarts.snowflake.com/guide/getting_started_with_streams_and_tasks/index.html#0" target="_blank">Streaming & Task Processing</a>.</li>
<li>Understanding of <a href="https://github.com/arunp77/Learning-git" target="_blank"> Version Control (Git)</a>.</li>
<li>Basic understanding of <a href="https://github.com/arunp77/SQL" target="_blank">SQL</a> for Data Manipulation</li>
</ul>
</div>
<div class="image">
<img src="assets/img/portfolio/Apache-nifi-logo.png" alt="Image Description">
</div>
</div>
<!-- project overview start here-->
<h3><b>Project overview</b></h3>
<p>This project demonstrates my ability to design and implement a real-time data streaming pipeline using Apache Nifi, AWS, Snowpipe,
Stream, and Task. The pipeline continuously ingests data from a JupyterLab notebook, processes it using Apache Nifi, and loads it into a Snowflake
data warehouse. The pipeline is designed to handle both new data and updates to existing data.</p>
<!-- project overview ends here-->
<!-- project Architecture starts here-->
<h3><b>Project Architecture</b></h3>
<p>The architecture of our project involves several interconnected components:</p>
<ul style="list-style-type: disc; margin-left: 30px;">
<li><b>EC2 Instance</b>: This serves as the foundation of the project, where we deploy our Docker container and essential tools.</li>
<li><b>Docker Container</b>: Housed within the EC2 instance, this container contains Python, Apache Nifi, and Apache ZooKeeper, ensuring a consistent and easily replicable environment.</li>
<li><b>JupyterLab and Apache Nifi</b>: These are the workstations for data engineers. JupyterLab is accessible at 'IP/4888,' and Apache Nifi is reachable at 'IP/2080' (where IP is the
<a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-instance-addressing.html" target="_blank">EC2 machine IP</a>).</li>
<li><b>Data Generation</b>: Using Python in JupyterLab, we utilized the 'Faker' library to create random data records, including customer information.</li>
<li><b>Data Streaming to <a href="amazon-s3.html">AWS S3</a></b>: Apache Nifi was used to establish connections for transferring the generated data to an AWS S3 bucket, providing scalable storage and real-time access.</li>
<li><b>Real-Time Data Processing with Snowflake</b>: Leveraging Snowpipe, we ensured that any new data or changes in existing data were automatically incorporated into the dataset. Slowly changing dimensions were used to track these changes.</li>
<li><b>Target Table Creation</b>: A task was executed to create target tables, ensuring the final dataset was always up-to-date and accurate.</li>
<img src="assets/img/portfolio/ec2-nifi-snowflake-pipeline.png" alt="Project Architecture" style="width: 800px; height: auto;">
</ul>
<!-- project Architecture ends here-->
<!-- project setup start here-->
<h3><b>Project Setup</b></h3>
<p>Let's dive into the details of the setup, like taking a closer look at the instruments in our orchestra:</p>
<ul style="list-style-type: disc; margin-left: 30px;">
<li><b><a href="ec2-confi.html">EC2 Instance Configuration</a>:</b>
I carefully set up an AWS EC2 instance with 8GB of RAM, selecting a powerful t2.xlarge instance for the best performance on the Amazon Linux system.
We opened ports from 4000 to 38888 and established SSH access. Think of it as preparing a concert hall for our performance.</li>
<li><b><a href="Docker-Container.html">Docker Container Setup</a>:</b> Within the EC2 instance, a Docker container was created and populated with Python,
Apache Nifi, and Apache ZooKeeper. Think of it as preparing our instruments, tuning them to perfection.</li>
<li><b><a href="Jupyter-nifi.html">JupyterLab and Apache Nifi Configuration</a>: </b>JupyterLab and Apache Nifi were configured to run on specific ports, making them accessible for data processing and orchestration.
It's like setting up the conductor's podium and the sheet music stand just right.</li>
<ul>
<li><b>Data Generation: </b>In JupyterLab, Python code was crafted to generate random data, simulating customer information. This is where the composer writes the notes.</li>
<li><b>Data Streaming: </b>Apache Nifi was utilized to set up connections, ensuring the seamless transfer of generated data to an AWS S3 bucket. It's the conductor guiding the instruments.</li>
</ul>
<li><b><a href="snowflake-task-stream.html">Real-Time Data Processing</a>: </b>Snowpipe, Snowflake streams, and tasks were set up to handle real-time data processing, enabling automatic updates as new data arrived.
It's the conductor guiding the orchestra to play in harmony.</li>
<li><strong><a href="scd.html">Slowly Changing Dimensions (SCD)
Implementation</a>:</strong></li>
<p>Within our orchestration, we employed the concept of Slowly Changing Dimensions (SCD) in Snowflake to maintain the integrity and history of our datasets.
Think of it as a conductor's keen ear for detecting subtle changes in the music.</p>
<ul>
<li><strong>SCD-1 Method (Flagging Changes):</strong> With the SCD-1 approach, we tracked changes by flagging modified records. It's akin to marking specific musical
notes that need adjustment. As data evolved, we efficiently identified and marked the records that underwent changes, ensuring data accuracy.</li>
<li><strong>SCD-2 Method (Historical Versioning):</strong> In parallel, we implemented the SCD-2 method, creating a symphony of historical versions for our datasets.
Each change introduced a new version, preserving a complete history of our data, much like recording various performances of a musical composition.
We maintained records with attributes such as start dates, end dates, and version numbers, allowing us to trace the evolution of our datasets over time.</li>
</ul>
</ul>
<!-- project setup end here-->
<!-- Key component start here-->
<h3><b>Key Components</b></h3>
<p>The key components are like our orchestra members:</p>
<ul style="list-style-type: disc; margin-left: 30px;">
<li><b><a href="amazon-s3.html">AWS EC2</a>:</b> (The Stage Where the Symphony is Performed) The AWS EC2 instance serves as the stage for our data symphony. Here's how
it was created and configured for the project:
<ul style="list-style-type: disc; margin-left: 20px;">
<li><b>Creation: </b> To set up an EC2 instance, I began by accessing the AWS Management Console and launched an EC2 instance.
I selected an instance type with 8GB of RAM and 100GB of memory space, specifically choosing a t2.xlarge instance for optimal performance. </li>
<li><b>Storage and Security: </b> In the process, I configured the instance with 100GB of storage, ensuring that there was sufficient space to store data and project components. Additionally, I created a security group that allowed incoming traffic on the
specified ports and enabled SSH access for remote management.</li>
<li><b>Connection: </b> After launching the EC2 instance, I connected to it via SSH, providing a secure channel for executing commands and configuring the environment.</li>
</ul>
</li>
<li><strong><a href="amazon-s3.html">Amazon S3 Terminology</a>: </strong>Before we get started moving data, let’s establish some basic terminology:</li>
<ul style="list-style-type: disc; margin-left: 30px;">
<li><strong>Identity and Access Management (IAM)</strong> – Controls for making and controlling who and what can interact with your AWS resources.</li>
<li><strong>Access Keys</strong> – These are your access credentials to use AWS. These are not your typical username/password — they are generated using access identity management.</li>
<li><strong>Bucket</strong> – A grouping of similar files that must have a unique name. These can be made publicly accessible and are often used to host static objects.</li>
<li><strong>Folder</strong> – Much like an operating system folder, these exist within a bucket to enable organization.</li>
</ul>
<li><b>Docker: </b>ocker played a crucial role in maintaining the tools and dependencies used in the project. Here's how it was set up:</li>
<ul style="list-style-type: disc; margin-left: 20px;">
<li><b>Installation:</b> To prepare the EC2 instance for Docker, I executed a series of commands:</li>
<pre>
<code class="language-python">
sudo yum update -y
sudo yum install docker
</code>
</pre>
<li><b>Installing Docker Compose:</b> I also installed Docker Compose to manage multi-container Docker applications.
This was achieved with the following commands:</li>
<pre>
<code class="language-python">
sudo curl -L "https://github.com/docker/compose/releases/download/1.29.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
</code>
</pre>
<li><b>Adding User to Docker Group:</b> To ensure that I had the necessary permissions to interact with Docker without using sudo,
I added my user to the Docker group and activated the changes with:</li>
<pre>
<code class="language-python">
sudo gpasswd -a $USER docker
newgrp docker
</code>
</pre>
<li><b>Python installation:</b>I installed Python and the required package:</li>
<pre>
<code class="language-python">
sudo yum install python-pip
</code>
</pre>
<li><b>Docker-compose: </b>I've installed a customized Docker container that includes essential images for JupyterLab and Apache NiFi. A docker-compose
file is configuration file used with Docker compose, a tool that simplifies the management of multi-container Docker applications. </li>
<p>To start the continer:</p>
<pre><code>docker-compose up</code></pre>
</ul>
<li><b>Jupyterlab</b></li>
<p>The JupyterLab can be access in browser:</p>
<pre><code>http://[your-ec2-instance-ip]:4888/lab?</code></pre>
<li><b>Apache Nifi: The Conductor </b></li>
<p>Apache Nifi was the conductor of our data symphony, orchestrating the flow of data. The apache-nifi can be access in browser:</p>
<pre><code>http://[your-ec2-instance-ip]:2080/nifi?</code></pre>
<li><b>Snowflake, snowpipe, snowflake stream and Task</b></li>
<ul style="list-style-type: circle;">
<li><b>Snowpipe:</b> Automates the process of ingesting data from your S3 bucket into Snowflake. It continuously loads data as it arrives, making it an
ideal solution for real-time data processing.</li>
<li><b>Snowflake Stream:</b> Captures changes to specific tables, including new data generated in JupyterLab and transferred to S3 by Apache NiFi. This stream ensures you're always working with the latest data.</li>
<li><b>Snowflake Task: </b> These tasks are configured to automatically execute SQL statements or stored procedures in response to changes in Snowflake Streams.
They enable real-time data processing and transformations, ensuring that new data is immediately available for analysis.</li>
<p></p>
<p>In my workflow, this combination of Snowflake, Snowpipe, Streams, and Tasks enables the seamless automation of data loading and processing as new data arrives,
eliminating manual intervention and ensuring up-to-the-minute data availability for your analytics and reporting needs.</p>
</ul>
</ul>
<p>The orchestration of these components is analogous to preparing an orchestra for a grand performance. The EC2 instance serves as the stage,
Docker maintains our instruments, and Apache Nifi conducts the symphony of data, guiding it through the intricacies of real-time processing and streaming.</p>
<h3><b>Project Results</b></h3>
<p>The project successfully achieved real-time data generation, streaming, and processing, showcasing the efficiency and scalability of the architecture.
It exemplified my proficiency in working with a diverse set of tools and technologies. Like any symphony, it was a harmonious blend of various elements,
each playing its role to create a beautiful performance.</p>
<h3><b>Challenges Faced:</b></h3>
<p>Although the project achieved great results, we encountered some hurdles. Think of it as fine-tuning a complex musical performance.
We had to make our data processing pipelines work faster for big datasets, ensure our real-time components were strong, and keep everything
moving smoothly. We tackled these challenges by carefully testing and making adjustments, just like a composer perfecting a symphony.</p>
<h3><b>Future Improvements</b></h3>
<p>To further enhance this project, future improvements may include:</p>
<ul style="list-style-type: disc; margin-left: 30px;">
<li>Implementing data quality checks for ensuring data accuracy.</li>
<li>Integrating machine learning models for more sophisticated data analysis.</li>
<li>Automating deployment and scaling of the architecture for handling larger and more extensive data streams, much like a symphony growing in scale and grandeur.</li>
</ul>
<p>This project is a testament to my expertise in data engineering and real-time data processing.
It's more than just a project; it's a data symphony, where diverse tools and technologies come together to create a harmonious and ever-evolving performance.</p>
<div class="navigation">
<a href="index.html#portfolio" class="clickable-box">
<span class="arrow-left">Go home</span>
</a>
<a href="Data-engineering.html" class="clickable-box">
<span class="arrow-right">Content</span>
</a>
</div>
</div>
</div>
</section><!-- End Portfolio Details Section -->
</main><!-- End #main -->
<!-- ======= Footer ======= -->
<footer id="footer">
<div class="container">
<div class="copyright">
© Copyright <strong><span>Arun</span></strong>
</div>
</div>
</footer><!-- End Footer -->
<a href="#" class="back-to-top d-flex align-items-center justify-content-center"><i class="bi bi-arrow-up-short"></i></a>
<!-- Vendor JS Files -->
<script src="assets/vendor/purecounter/purecounter_vanilla.js"></script>
<script src="assets/vendor/aos/aos.js"></script>
<script src="assets/vendor/bootstrap/js/bootstrap.bundle.min.js"></script>
<script src="assets/vendor/glightbox/js/glightbox.min.js"></script>
<script src="assets/vendor/isotope-layout/isotope.pkgd.min.js"></script>
<script src="assets/vendor/swiper/swiper-bundle.min.js"></script>
<script src="assets/vendor/typed.js/typed.umd.js"></script>
<script src="assets/vendor/waypoints/noframework.waypoints.js"></script>
<script src="assets/vendor/php-email-form/validate.js"></script>
<!-- Template Main JS File -->
<script src="assets/js/main.js"></script>
<script>
document.addEventListener("DOMContentLoaded", function () {
hljs.initHighlightingOnLoad();
});
</script>
</body>
</html>