Skip to content

Maikxx/frontend-data

BranchesTags

Repository files navigation

Frontend Data

Table of contents

Installation

git clone git@github.com:Maikxx/frontend-data.git
cd frontend-data
  1. Install the dependencies with: yarn or npm install.
  2. Copy the .env.example file with cp .env.example .env and provide the API keys.
  3. Run the server with yarn serve or npm run serve.
  4. Run the client with yarn start:client or npm run start:client.
  5. Start linting with yarn lint or npm run lint.

To build the code locally use yarn build:client and change the links of the CSS and JS in the index.html file from /client... to /dist/client....

If you are experiencing CORS errors, please consider installing this Chrome plugin

Concept

Note: These designs might be a little outdated, but the global concept still stands

I originally planned to make a graph, which would hold all the variables of the books from the OBA in it (some early sketches can be found here and here). However, this felt like it was not interesting enough. I also received feedback that I needed to make a visualization which was interesting to me personally.

When I heard that, I immediately looked at a way to implement airplanes in the visualization. I eventually came up with a map, which holds all the cities in which atleast one books is published.

I decided to go for a map in D3, because I heard good things about that from Tim, Titus and Lifely.

The landing page can be seen in this image:

Concept image 1

Here you can select an airplane and then select a city on the map. You could also zoom out on the map to reveal more of the world. When you click on a specific point you can explore the books that are published at this location from a popup. My initial way to visualize this can be seen in the following image:

Concept image 2

However, after exploring the amount of books I had per location, I quickly realized this was not possible. I have thought of something new when you read this, check out the live visualization.

On the next two images another interaction state can be seen. The color of the city changes and there is a line being drawn from the selected city to the destination city, which is always Amsterdam.

Concept image 3 Concept image 4

The main thought behind this visualization was to have the user explore the world to find their favorite books. I coupled the flight data with this, because I thought it would make the visualization a lot more interesting.

Disclaimers

  • There is no way the flight times are 100% accurate, since the calculation takes the cruising speed of a plane, in other words, the take-off and landing speeds are not taken into account.
  • The speed is not live, which means that flight times may always vary with wind speeds.
  • Probably the most books don't arrive in the Netherlands by plane, and even may or may not come from the publication city.

Future enhancements

For an up to date list of future enhancements, you can take a look here.

Process

During these two weeks I learned a lot more than I had initially thought. The learning process started with, what was a mystery for me, that GitHub doesn't like to have large files stored on it's servers.

I had to use a git tool, called Git-LFS to store the data file I originally had. This was a text file containing all the geographic coordinates of cities in the world. I eventually stopped using this file, because my computer could not handle the amount of computations required to filter and map through the data to transform it.

I then choose to use an external API to load the coordinates by city name (at this point I had transformed all the cities to cleaned names). I thought these requests (±230) could be fired from the client, however I quickly found out that the API only allows 1 request per second, which would make the load time of the map equivalent to the amount of requests, which was way too long.

Finally, in this chain of events I choose to take an approach I think is rather unnecessary, but after started to apreciate. One might call this the long route, which involved me writing a data file to my computer for each significant step in the data transformation process. You can find all these resulting files here.

Because you write a file every significant step, getting lost in the data is harder, which is a good thing. Because of this approach the client only has to do two requests, which is to this and this this.

More things I learned these past weeks include, of course D3, but also MapBox, which is a great Map provider, which makes interaction with it and placing svg layers on top super easy.

As for the concepting part of this project, well... The moment I heard from Titus that I should make something that is fun for me to make, I was just flying (pun intended) to do something with airplanes for once after two years of not having the time to make something like that happen.

The concept really just evolved over time, and at this moment it is still a work in progress, which I think is great, because I can add a lot of things to the visualization when it progresses.

I initially thought I was going to hate D3, because everyone I know said it was awefull, however the reality is that I love it. It makes a lot of things so much more easy. The things that I could not easily write myself or solve with D3, I fixed by using Lodash.

The further the project got, the more I started enjoying it, and now I would like to keep this project up to date and develop it to the next level when I have the time for it. You can look at my GitHub issues to see what I want to add in the future.

Finally, I am starting to like working with GitHub issues and git branches each time a little bit more, which I would not have imagined I would ever say for a solo project.

The full log of my process can be found here!

Development

Code attribution

Pretty much all the code I wrote is made by myself. This includes the complete development environment setup, as well as the server every file in the api folder.

The server and api code is mainly used for transforming the raw data from the OBA to usable geoJson. The reason I complete this task on the server instead of the client, is because otherwise the client would need to send out 230 requests, as I explained earlier.

Pretty much the whole graph is also made by me, except for the gradient part of the scaleLegend function and the code that transforms a line into an arc. For both of these, the motivation to not write it myself, is because I couldn't understand how they would work. For the line to arc transformation, this was pretty much the only decent example that I could find and could get to work with the structure I had built at that moment.

All of the other code is written by myself, unless otherwise stated.

Data

In the visualization I used books that have a publication location. The base data came from Wouter, who had exported about 4500 hits in a JSON file, which made developing way easier, instead of having to wait for like 500000 books to load. Thus, this visualization is made up of a subset of the data from the OBA.

What I used in terms of variables to tie it all together:

  • Book title
  • Publication location of a book
  • Lat, long coordinates based on the publication locations, revrieved from LocationIQ
  • The distance between cities, calculated with the help of MapBox
  • Names and cruising speeds of a select amount of airplanes, attribution can be found here
  • Fly time between two points
  • Amount of books per location
  • Wind speed and bearing for the selected city, by sending a request to Dark Sky.

I originally planned to make the select have more options by attaching it to another API, from Wikipedia for example, but due to the simple fact that most planes have similar cruising speeds, this was not usefull enough for me to start looking in it, for now.

Interaction

If, for some reason, the interaction is not clearly visible at first, no worries, I got you covered here.

  • Zooming in and out on the map
  • Clicking on the Amsterdam legend item flies you back to Amsterdam if you get lost on the map
  • Selecting an airplane from the dropdown
  • Clicking on a point on the map to draw a line between this city and Amsterdam in order for the statistics in the panel on the right to update with accurate approximate fly times and distances
  • When you click on a city without selecting an airplane you will get an error toast🍞

Tools and APIs

The graphs base is using MapBox, because I heard a lot of good things about this before starting at this task and started to wonder if I could make something happen with an additional SVG layer on top.

Code

Server

  • server - Base for the server.

  • types - TypeScript type definition files.

  • filters - Contains some general modularized filter functions.

  • getters - Most of the transformation logic happens here.

  • processors - The brains of the server, fires getters, filters and queries.

  • queries - Contains a function that sends a request to the LocationIQ API to get coordinates by the name of a book.

Client

  • client - Client initializer, which fires all sub files.

  • animations/pageLoad - Function that animates in page content.

  • drawers/cities - Functions that draw dots and lines on the map.

  • drawers/scaleLegend - Function that creates the legend in the bottom right of the page.

  • feedback/error - Function that handles showing of errors to the page.

  • filters/cities - Handles filtering of a specific line when clicked.

  • getters/amountOfBooks - Functions that have to do with calculating the amount of books for a given spot.

  • getters/cities - Functions that return something that has something to do with cities.

  • getters/planes - Functions that return something that has something to do with planes.

  • getters/scales - Functions that return a certain scale.

  • setters/interaction - Functions that set data to the state or UI, which have something to do with interaction.

  • setters/planes - Functions that set data to the state or UI, which have something to do with planes.

  • setters/ui - Main file that contains UI interactions.

  • updaters/cities - Functions that update the data in the state for cities.

  • updaters/general - General updaters.

Data storage

Honourable mentions

  • Chelsea: Chelsea helped me writing the data transformators and getting me up and running with D3.
  • Jessie: Jessie helped me getting up and running with D3.
  • Wouter: Wouter created the boilerplate for pagination with the OBA API.
  • Tim: Tim helped me solve some complex issues, mainly things that involved the connection between MapBox and D3.

Sources

License

This repository is licensed as MIT by Maikel van Veen.

About

Study - A repository for the course frontend-data

maikxx.github.io/frontend-data/index.html

Topics

javascript d3 html data node typescript js ts

Resources

Readme

License

MIT license
Activity

Stars

0 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases 1

Finish Frontend Data course Latest
Nov 24, 2018

Packages

No packages published

Languages