-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
11 changed files
with
276 additions
and
27 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,110 @@ | ||
| <!DOCTYPE HTML> | ||
| <!-- | ||
| Massively by HTML5 UP | ||
| html5up.net | @ajlkn | ||
| Free for personal and commercial use under the CCA 3.0 license (html5up.net/license) | ||
| --> | ||
| <html> | ||
| <head> | ||
| <title>Michael Tezak Portfolio</title> | ||
| <meta charset="utf-8" /> | ||
| <meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no" /> | ||
| <link rel="stylesheet" href="../../assets/css/main.css" /> | ||
| <link rel="icon" type="image/svg+xml" href="../../assets/icons/favicon.svg" /> | ||
| <noscript><link rel="stylesheet" href="../../assets/css/noscript.css" /></noscript> | ||
| </head> | ||
|
|
||
| <body class="is-preload"> | ||
|
|
||
| <!-- Wrapper --> | ||
| <div id="wrapper"> | ||
|
|
||
| <!-- Header --> | ||
| <header id="header"></header> | ||
|
|
||
| <!-- Nav --> | ||
| <nav id="nav"> | ||
| <ul class="links"> | ||
| <li class="active"><a href="../projekte.html">Projekte</a></li> | ||
| <li><a href="../qualifikationen.html">Qualifikationen</a></li> | ||
| <li><a href="../weitere-interessen.html">Weitere-Interessen</a></li> | ||
| </ul> | ||
| <ul class="icons"> | ||
| <li><a href="https://github.com/mgtezak" target="_blank" class="icon brands fa-github"><span class="label">GitHub</span></a></li> | ||
| <li><a href="https://www.linkedin.com/in/mgtezak/" target="_blank" class="icon brands fa-linkedin"><span class="label">LinkedIn</span></a></li> | ||
| </ul> | ||
| </nav> | ||
|
|
||
| <!-- Main --> | ||
| <div id="main"> | ||
|
|
||
| <!-- Post --> | ||
| <section class="post"> | ||
| <header class="major"> | ||
| <!-- <span class="date">1. November 2023</span> --> | ||
| <h1>Advent of Code Datenanaylse</h1> | ||
| <p> | ||
| Web Scraping, Analyse und Visualisierung öffentlicher Benutzerdaten von Advent of Code. <br/> | ||
| Streamlit, Requests, BeautifulSoup, Matplotlib, Seaborn | ||
| </p> | ||
| </header> | ||
| <div class="image main"><img src="../../images/aoc-submission-times.png" alt="image: aoc-submission-times" /></div> | ||
|
|
||
| <p> | ||
| Nachdem ich meine | ||
| <a href="aoc-solver.html">Advent of Code Puzzle Solver App</a> entwickelt hatte, dachte ich, dass es schön wäre, | ||
| einen Bereich für Datenanalytik mit vielen weihnachtlich gestalteten Visualisierungen hinzuzufügen. | ||
| Zuerst fügte ich einige Visualisierungen meiner persönlichen Statistiken hinzu, | ||
| dann wollte ich einige globale Community-Statistiken, Ranglisten und allgemeine Partizipationsquoten visualisieren. | ||
| Da es keine öffentliche API gab, von der aus ich diese Daten abrufen konnte, beschloss ich, | ||
| sie direkt von der Website mit Hilfe von Pythons <i>Requests</i> und <i>BeautifulSoup</i> Bibliotheken zu scrapen. | ||
| Dabei lernte ich viel über empfohlene Vorgehensweisen bei ethischem Web Scraping. | ||
| Zum Beispiel sollte man seine Kontaktdaten im User-Header seiner Requests mitsenden und | ||
| die Frequenz von Requests zeitlich drosseln um den Server nicht zu überlasten. | ||
| Nachdem ich die Daten hauptsächlich mit Matplotlib und etwas Seaborn visualisiert hatte, | ||
| präsentierte ich meine Ergebnisse in einem öffentlichen | ||
| <a href="https://www.reddit.com/r/adventofcode/comments/1aeoazh/aoc_public_stats_visualizations_20152023/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button" target="_blank">Reddit-Post</a>. | ||
| Es war erfreulich zu sehen, wie viele Leute in der Community Interesse zeigten und mir konstruktives Feedback gaben. | ||
| </p> | ||
| <ul class="actions special"> | ||
| <li><a href="https://aoc-puzzle-solver.streamlit.app/" target="_blank" class="button primary">Zur Website</a></li> | ||
| </ul> | ||
| </section> | ||
|
|
||
| <footer> | ||
| <div class="pagination"> | ||
| <a href="pomodino.html" class="previous">Prev</a> | ||
| <a href="mohssche-härte.html" class="next">Next</a> | ||
| </div> | ||
| </footer> | ||
|
|
||
| </div> | ||
|
|
||
| <!-- Footer --> | ||
| <footer id="footer"> | ||
| <section class="split contact"> | ||
| <section> | ||
| <h3>Kontakt</h3> | ||
| <p><a>mgtezak@gmail.com</a></p> | ||
| <p><span class="language-toggle"><a href="../../en/projects/aoc-analytics.html">English version</a></span></p> | ||
| </section> | ||
| </section> | ||
| </footer> | ||
|
|
||
| <!-- Copyright --> | ||
| <div id="copyright"></div> | ||
|
|
||
| </div> | ||
|
|
||
| <!-- Scripts --> | ||
| <script src="../../assets/js/jquery.min.js"></script> | ||
| <script src="../../assets/js/jquery.scrollex.min.js"></script> | ||
| <script src="../../assets/js/jquery.scrolly.min.js"></script> | ||
| <script src="../../assets/js/browser.min.js"></script> | ||
| <script src="../../assets/js/breakpoints.min.js"></script> | ||
| <script src="../../assets/js/util.js"></script> | ||
| <script src="../../assets/js/main.js"></script> | ||
|
|
||
|
|
||
| </body> | ||
| </html> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| <!DOCTYPE HTML> | ||
| <!-- | ||
| Massively by HTML5 UP | ||
| html5up.net | @ajlkn | ||
| Free for personal and commercial use under the CCA 3.0 license (html5up.net/license) | ||
| --> | ||
| <html> | ||
| <head> | ||
| <title>Michael Tezak Portfolio</title> | ||
| <meta charset="utf-8" /> | ||
| <meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no" /> | ||
| <link rel="stylesheet" href="../../assets/css/main.css" /> | ||
| <link rel="icon" type="image/svg+xml" href="../../assets/icons/favicon.svg" /> | ||
| <noscript><link rel="stylesheet" href="../../assets/css/noscript.css" /></noscript> | ||
| </head> | ||
|
|
||
| <body class="is-preload"> | ||
|
|
||
| <!-- Wrapper --> | ||
| <div id="wrapper"> | ||
|
|
||
| <!-- Header --> | ||
| <header id="header"></header> | ||
|
|
||
| <!-- Nav --> | ||
| <nav id="nav"> | ||
| <ul class="links"> | ||
| <li class="active"><a href="../projects.html">Projects</a></li> | ||
| <li><a href="../technical-skills.html">Technical-Skills</a></li> | ||
| <li><a href="../other-interests.html">Other-Interests</a></li> | ||
| </ul> | ||
| <ul class="icons"> | ||
| <li><a href="https://github.com/mgtezak" target="_blank" class="icon brands fa-github"><span class="label">GitHub</span></a></li> | ||
| <li><a href="https://www.linkedin.com/in/mgtezak/" target="_blank" class="icon brands fa-linkedin"><span class="label">LinkedIn</span></a></li> | ||
| </ul> | ||
| </nav> | ||
|
|
||
| <!-- Main --> | ||
| <div id="main"> | ||
|
|
||
| <!-- Post --> | ||
| <section class="post"> | ||
| <header class="major"> | ||
| <!-- <span class="date">1. November 2023</span> --> | ||
| <h1>Advent of Code Data Analytics</h1> | ||
| <p> | ||
| Web scraping, analysis and visualization of Advent of Code's public user data. | ||
| <br/> | ||
| Streamlit, Requests, BeautifulSoup, Matplotlib, Seaborn | ||
| </p> | ||
| </header> | ||
| <div class="image main"><a href="https://aoc-puzzle-solver.streamlit.app/" target="_blank"><img src="../../images/aoc-submission-times.png" alt="image: aoc-solver" /></a></div> | ||
|
|
||
| <!-- <span class="image right"><img src="../../images/aoc_public_completions2.png" alt="" /></span> --> | ||
|
|
||
| <p> | ||
| After having built my <a href="aoc-solver.html">Advent of Code Puzzle Solver</a> app, | ||
| I figured, that it would be nice to add a data analytics part with lots of Christmas-themed visualizations to the website. | ||
| First I added some visualizations of my personal stats, then I wanted to visualize some global community stats, leaderboard rankings and overall puzzle completion rates. | ||
| Since there wasn't any public API from which to pull these data, | ||
| I decided to scrape them directly from the website using Python's <i>Requests</i> and <i>BeautifulSoup</i> libraries. | ||
| In the process I learned a lot about best practices in ethical web scraping. | ||
| For example, one should put their contact info in the user-header of each request and throttle the speed at which requests are sent so as not to overwhelm the server. | ||
| After visualizing the data using mostly Matplotlib and some Seaborn I presented my results in a public | ||
| <a href="https://www.reddit.com/r/adventofcode/comments/1aeoazh/aoc_public_stats_visualizations_20152023/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button" target="_blank">reddit post</a>. | ||
| It was gratifying to see how many people in the community took an interest and gave me constructive feedback. | ||
|
|
||
| <!-- All in all, this has been the biggest, most multifaceted and gratifying of my projects so far. | ||
| I'm glad to say that, in little more than a month after its launch, it has received more than 2500 unique visitors, some of whom have become regular guests. | ||
| I've also received a lot of positive feedback by the Advent of Code community. --> | ||
|
|
||
| </p> | ||
| <ul class="actions special"> | ||
| <li><a href="https://aoc-puzzle-solver.streamlit.app/" target="_blank" class="button primary">Visit Website</a></li> | ||
| </ul> | ||
| </section> | ||
| <footer> | ||
| <div class="pagination"> | ||
| <a href="pomodino.html" class="previous">Prev</a> | ||
| <a href="mohs-hardness.html" class="next">Next</a> | ||
| </div> | ||
| </footer> | ||
| </div> | ||
|
|
||
| <!-- Footer --> | ||
| <footer id="footer"> | ||
| <section class="split contact"> | ||
| <section> | ||
| <h3>Contact</h3> | ||
| <p><a>mgtezak@gmail.com</a></p> | ||
| <p><span class="language-toggle"><a href="../../de/projekte/aoc-analytics.html">Deutsche Version</a></span></p> | ||
|
|
||
| </section> | ||
| </section> | ||
| </footer> | ||
|
|
||
| <!-- Copyright --> | ||
| <div id="copyright"></div> | ||
|
|
||
| </div> | ||
|
|
||
| <!-- Scripts --> | ||
| <script src="../../assets/js/jquery.min.js"></script> | ||
| <script src="../../assets/js/jquery.scrollex.min.js"></script> | ||
| <script src="../../assets/js/jquery.scrolly.min.js"></script> | ||
| <script src="../../assets/js/browser.min.js"></script> | ||
| <script src="../../assets/js/breakpoints.min.js"></script> | ||
| <script src="../../assets/js/util.js"></script> | ||
| <script src="../../assets/js/main.js"></script> | ||
|
|
||
|
|
||
| </body> | ||
| </html> |
Oops, something went wrong.