Data management (tutorial)

In this tutorial I will show 3 different ways of storing data. (15 min read)

Author: Wouter van der Heijde - June 2020

For this tutorial I'll be using a node.js server and expect the reader to have at least some basic understanding of Javascript.

Different types of data require different methods of storing

Because not all the data is equally important to the owner of the web page, you can choose the approach you find fitting to your needs. In this tutorial I tend to explain when it's best to use one of the three different approaches handled in this tutorial. Here is a heads up of the three different approaches I'll be explaining:

1. Local storage ★✩✩ (store data in the browser, personal data of the user)
2. Server ★★✩ (store the data as a JSON file, usually done for small projects or prototypes)
3. Database ★★★ (store large sets of data and multi user data-objects)

Local storage

Difficulty level - ★✩✩ (beginner)

What is local storage? localStorage is a built-in web storage API that can be used to securely store data as key/value pairs. In chrome you can find it in the developer tools under "Application" see image below.

Before you store the formData there are some things you need to know:

• Local storage only allow "strings" to be stored
• Data in local storage has no expiration time

Local storage (chrome)	Similar storage method: sessionStorage
	Session storage works exactly the same as localStorage but with one major exception: all data in the session storage gets cleared when the page session ends.* *whenever the user closes the page.

When should I use this approach?

Lets say you have a form on your website but the form is rather long and your user doesn't feel like completing it in one turn. This is a perfect opportunity to use local storage. You can store the form data as key = form question and value = user's answer. Before you can store the data you need to parse the form data to a JSON format. This can be easily done using the javascript built-in FormData. Every time the user fills in an answer you can trigger the following function to parse the form into JSON-object.

index.js

const form = document.querySelector('#myForm') // first get acces to the form

form.addEventListener('change', function () {

    const formData = new FormData(form) // with form available we can parse the answers of the users into our formData object
    // the FormData object itself cannot be read or parsed so we'll have to take it apart

    let formDataObject = {} // define a Object you want the formData to be parsed to

    formData.forEach((value, key) => { // here we take all the values and keys in our formdata and assign it to the formDataObject
        formDataObject[key] = value
    })
    console.log(formDataObject) // now if we change something to the form we the output will be {firstName: "Wouter", lastName: ""}
    // this is perfect because we have a JSON-object containing all our form data

    // Now we can call the function to store our formDataObject with a matching local storage name
    setLocalStorage("formData", formDataObject)
})

Now we have our formData available as an javascript-object, but before we can store this to the local storage we'll have to parse it to a string. We can simply use the javascript built-in JSON.stringyfy(formDataObject) to get the result we want. In the next function I'll show you how to save data to the local storage.

function setLocalStorage(localName, dataToStore) {
    // before we store the data check if there already is data in the localstorage
    let existing = localStorage.getItem(localName)

    // If no existing data, create an object
    // Otherwise, convert the localStorage string to an object
    existing = existing ? JSON.parse(existing) : {}

    // Add new data to localStorage Object
    existing = dataToStore

    // Save back to localStorage
    localStorage.setItem(localName, JSON.stringify(existing))
    // remember to parse your object to a string because localstorage can only store "strings"

}

And here we have our result we stored data to the local storage! The data is now accessible by calling localStorage.getItem('formData'), remember before you can use this as an javascript-object {}, you'll have to parse it to JSON again. Because the local storage only stores "strings" remember ;). For a working demo check the image below or clone this repo and try it yourself!

Server

Difficulty level - ★★✩ (intermediate)

In this tutorial w're gonna create our own user list and store the list in a file on our server. For our server side data management we'll be using the following setup:

Node.js server
NPM - packages	File structure
- EJS - template language - Express - Body-parser (used to get acces to the form post) - fs-extra (used for reading and writing files)

For our basic setup I created a simple form which takes a first- and a last name. And our server set up like this:

app.js

const express = require('express')
const routes = express.Router()
const bodyParser = require("body-parser")
const urlencodedParser = bodyParser.urlencoded({ extended: true })
const setJSON = require('./controllers/setJSON')

const config = {
    PORT: 3000
}

const app = express()

app
    .use(express.static('static'))
    .use(bodyParser.json())
    .use(urlencodedParser)
    .use('/', routes)

    .set('view engine', 'ejs')
;

routes 
    // the route required for the POST request this is the route you post to with your form
    .post('/form', setJSON.writeData, (req, res) =>{ // we call the middleware function, writeData, where we extract our form post
        let data = res.locals.users 
        // This is the unsaved data made available from the writeData function that lives in: 'controllers/setJSON.js'
        // we will serve the user a list of all users in the data.json
        res.render('pages/userlist.ejs', {
            users: data.users, // The data is now available on the userlist.ejs
            title: "A list of users"
        })
    })

    // the route to homepage
    .get('/', (req, res) =>{ 
        res.render("pages/form.ejs", {
            title: "Store data to server",
        })
    })      
;

app.listen(config.PORT, () => console.log(`Server running on: http://localhost:${config.PORT}`))

For this approach you need to create a data.json file in the root of your app. We are gonna create the file with the following contents:

data.json

{
  "users": [
    
  ]
}

This way when we open the file on our server we can use it like an javascript-object {}. First we make sure our file is available in our setJSON.js with the following statement at the top of our file: const jsonFile = "./data.json";. We can target the file and parse its contents.

fs.readFile example

fs.readFile(jsonFile, (err, content) => {
    // First we read the jsonFile and see whats inside
    if (err) return console.log(err); // catch errors as they occur

    const contentJSON = JSON.parse(content); // parse the json file to workable javascript-object

    console.log(contentJSON); // output: { users: [] }
}

We have now acces to our users = (contentJSON.users) which is at this point an empty array: [ ]. Because it's an Array we can add one object to it using contentJSON.users.push(newUser). Now contentJSON looks like this:

{
  "users": [
    { 
      "firstName": "Wouter",
      "lastName": "Heijde" 
    }
  ]
}

Nice we stored a user in our contentJSON variable next up rewrite the jsonFile with the new content:

fs.writeFile(jsonFile, JSON.stringify(contentJSON, null, 2),(err) => { 
      // I stringyfy the contentJSON before its write-able to the data.json
      // these fancy parameters I put in the stringyfy
      // makes sure the data.json gets nicely formatted
      if (err) console.log(err);
    })

Great we have our own little database on the server. But when you'll have to handle more complex and bigger datasets, this might not be an ideal approach. Next step create your own database!

Database

Difficulty level - ★★★ (advanced) We are going to create a register page and store a user to our database.

Node.js server
NPM - packages	File structure
- EJS - template language - Express - express-session - Body-parser (used to get acces to the form post) - mongoose - mongoose-unique-validator - bcrypt - passport - cookie-parser

We are using mongoose which is an extension build for mongodb with allot of built-in functions that helps us with making models and finding objects in our database.

For our database we are going to use Mongodb. Before you can start this tutorial have the following things ready:

• Register an account on Mongodb
• Clone this repository: example.3
• open your terminal and navigate to example.3
• run npm i
• .env - file to safely store our local variables

Dotenv is a zero-dependency module that loads environment variables from a .env file into process.env. For more information about dotenv and how to use it check their documentation on: dotenv - npm

After registering an account on mongoDB log in on mongodb-Atlas with your credentials, you can get the URI to connect with your database really easy. Check the example below!

For convenience matters I paste the complete URI to our .env file like this:

``

In our .env we declare the following:

MONGODB_URI = mongodb+srv://<username>:<password>@<clustername>-dklfs.azure.mongodb.net/<dbname>?retryWrites=true&w=majority
DB_NAME = YOUR_DATABASE_NAME

First step is creating a connection with our database. Since we have mongoose installed we can call the mongoose.connect() and pass it our mongodb variables, take a look at the next code example how to do this.

const uri = process.env.MONGODB_URI // Get acces to our mongoDB variables

mongoose.connect(uri, options)
mongoose.connection.on("open", function (err, doc) {
  console.log(`connection established with ${process.env.DB_NAME}`)
  if (err) throw err
})