Skip to content
/ software-development-notes-bookmarks Public

📑 Some of my own notes and bookmarks for software development

6 stars 0 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

FintanK/software-development-notes-bookmarks

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

89 Commits
README.md
README.md
 
 

Repository files navigation

📚 📒 Software Development Notes and Bookmarks

Standing on the shoulders of giants..

This markdown file (which is always a work in progress) helps me revise many of the topics that i've covered over the last 12 years in the area of software development. It includes links to some of my favourite articles / resources as well as some short write ups of my own interpretations of many topics / concepts that i've encountered along the way.

Some of my favourite websites / blogs
Organisations
Common Abbreviations
  • IAB (Internet Architecture Board)
  • IANA (Internet Assigned Numbers Authority)
  • ICANN (Internet Corporation for Assigned Names and Numbers)
  • IESG (Internet Engineering Steering Group)
  • IETF (Internet Engineering Task Force)
  • IRTF (Internet Research Task Force)
  • ISOC (Internet Society)
  • W3C (World Wide Web Consortium)
Protocols and Standards

Markdown

Markdown reference https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet

HTML (Hyper-Text Markup Language)

Hypertext Markup Language (HTML) is the standard markup language for creating web pages and web applications. With Cascading Style Sheets (CSS) and JavaScript, it forms a triad of cornerstone technologies for the World Wide Web.

HTML5 API

HTML 5 API Overview (Handy reference) https://platform.html5.org/

Canvas

"Added in HTML5, the HTML element can be used to draw graphics via scripting in JavaScript. For example, it can be used to draw graphs, make photo compositions, create animations or even do real-time video processing or rendering." by Mozilla Developer Network

https://github.com/raphamorim/awesome-canvas#readme - Curated resource for working with the HTML 5 Canvas Element

CSS (Cascading Style Sheets)

Cascading Style Sheets (CSS) is a style sheet language used for describing the presentation of a document written in a markup language like HTML.

CSS Layout

A 2018 Guide to CSS layouts https://www.smashingmagazine.com/2018/05/guide-css-layout/ Browser Media Queries https://developer.mozilla.org/en-US/docs/Web/CSS/Media_Queries/Using_media_queries

Popular CSS Libraries and Frameworks

Bootstrap https://getbootstrap.com/docs/4.0/getting-started/introduction/ Foundation https://foundation.zurb.com/ Bulma https://bulma.io/ UIkit https://getuikit.com/ Semantic UI https://semantic-ui.com/

CSS Preprocessors

A CSS preprocessor is a program that lets you generate CSS from the preprocessor's own unique syntax. There are many CSS preprocessors to choose from, however most CSS preprocessors will add some features that don't exist in pure CSS, such as mixin, nesting selector, inheritance selector, and so on.

SASS (Syntatically Awesome Style Sheets)

https://sass-lang.com/documentation - Documentation https://medium.com/@andrew_barnes/bem-and-sass-a-perfect-match-5e48d9bc3894 - BEM (Block Element Modifier)

LESS (Leaner Style Sheets)

http://lesscss.org/features/ - Documentation

Stylus

Stylus simplifies your CSS Syntax to an indented format with a lot of extra features!

  • Optional colons
  • Optional semi-colons
  • Optional commas
  • Optional braces
  • Variables
  • Interpolation
  • Mixins
  • Arithmetic
  • Type coercion
  • Dynamic importing
  • Conditionals
  • Iteration
  • Nested selectors
  • Parent referencing
  • Variable function calls
  • Lexical scoping
  • Built-in functions (over 60)
  • In-language functions
  • Optional compression
  • Optional image inlining
  • Stylus executable
  • Robust error reporting
  • Single-line and multi-line comments
  • CSS literal for those tricky times
  • Character escaping
  • TextMate bundle

http://stylus-lang.com/

CSS Coding Standards

CSS Articles

Conditional CSS Support Checks https://www.lottejackson.com/learning/supports-will-change-your-life CSS Mixins https://css-tricks.com/custom-user-mixins/

Javascript

Javascript from scratch https://javascript.info/

Promises

https://github.com/wbinnssmith/awesome-promises#readme

Standard Tools

https://github.com/standard/awesome-standard#readme

Workflow

https://github.com/maxogden/maintenance-modules#readme

Observables

https://github.com/sindresorhus/awesome-observables#readme

XML (Extensible Markup Language)

Extensible Markup Language is a markup language that defines a set of rules for encoding documents in a format that is both human-readable and machine-readable. The W3C's XML 1.0 Specification and several other related specifications—all of them free open standards—define XML

A list of popular XML Schemas used on the web https://en.wikipedia.org/wiki/List_of_types_of_XML_schemas

In my opinion the best and most concise article on XML https://www.service-architecture.com/articles/xml/index.html

Fonts / Typography on the Web

Mozilla Web Fonts Guide https://developer.mozilla.org/en-US/docs/Learn/CSS/Styling_text/Web_fonts https://github.com/deanhume/typography#readme - Curated list of resources for working with Web Typography

Audio for the web

Mozilla Web Audio Guide https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

Video for the web

Mozilla Video Web Guide https://developer.mozilla.org/en-US/docs/Learn/HTML/Multimedia_and_embedding/Video_and_audio_content#Audio_and_video_on_the_web

Images on the Web

https://kinsta.com/blog/optimize-images-for-web/ - How to opimize images for the web and performance

PNG

PNG (pronounced ping as in ping-pong; for Portable Network Graphics) is a file format for image compression that, in time, is expected to replace the Graphics Interchange Format (GIF) that is widely used on today's Internet.

JPEG, JPG

JPEG is a commonly used method of lossy compression for digital images, particularly for those images produced by digital photography. The degree of compression can be adjusted, allowing a selectable tradeoff between storage size and image quality.

SVG

Scalable Vector Graphics (SVG) is an XML-based vector image format for two-dimensional graphics with support for interactivity and animation. The SVG specification is an open standard developed by the World Wide Web Consortium (W3C) since 1999. SVG images and their behaviors are defined in XML text files.

https://github.com/willianjusten/awesome-svg#readme - Curated resources on SVG images for the web.

Responsive Images

HTML responsive images https://developer.mozilla.org/en-US/docs/Learn/HTML/Multimedia_and_embedding/Responsive_images

Bandwidth Optimization for Web Applications

In order to ensure that your application is lightweight and responds quickly to user input, it's important to take a look at how much bandwith your application uses.

A quick way to do this is to open your dev tools and take a look at the total size of all downloaded assets when loading the website.

Caching

Setting cache control policies on HTTP response headers can prevent the browser having the re-download the same resource.

Service workers are also useful for defining which assets you would like to cache locally.

File Compression

Many forms of downloaded media can be compressed. For more information on compression with HTTP take a look here https://developer.mozilla.org/en-US/docs/Web/HTTP/Compression

File Concatenation

Using a build tool like Webpack or Grunt can allow you to combine multiple css, js or other file types into a single file reducing the amount of concurrent requests the browser has to make and can help an initial page request load faster.

Client and server side detection

Detecting the user's device can be quite useful when it comes to choosing what types of media to download. For example, you can generate different sizes for an image and then serve smaller images for user's on a mobile browser.

Multimedia

https://ffmpeg.org/ - FFmpeg is the leading multimedia framework, able to decode, encode, transcode, mux, demux, stream, filter and play pretty much anything that humans and machines have created. It supports the most obscure ancient formats up to the cutting edge. No matter if they were designed by some standards committee, the community or a corporation. It is also highly portable: FFmpeg compiles, runs, and passes our testing infrastructure FATE across Linux, Mac OS X, Microsoft Windows, the BSDs, Solaris, etc. under a wide variety of build environments, machine architectures, and configurations.

https://imagemagick.org/index.php - Use ImageMagick® to create, edit, compose, or convert bitmap images. It can read and write images in a variety of formats (over 200) including PNG, JPEG, GIF, HEIC, TIFF, DPX, EXR, WebP, Postscript, PDF, and SVG. ImageMagick can resize, flip, mirror, rotate, distort, shear and transform images, adjust image colors, apply various special effects, or draw text, lines, polygons, ellipses and Bézier curves.

State Management for Web Applications

Just like the server, the browser can have different types of states to manage...

  • Server Response Data
  • User information
  • User Input
  • UI State
  • Router / location state

State Management Libraries

  • Model our app state
  • Update the state
  • Read state values
  • Monitor changs to the state

The Redux Design Pattern / Architecture

The three principles of Redux.

  • Single source of truth.
  • State is read-only.
  • Pure functions update the state

One of the biggest benefits of using state management is that the duplicate dependency injection of data services can be moved out from the components to the state management library implementation.

In Universal (Server side rendered) Angular apps you can prepopulate the store on page load and avoid the many HTTP many requests needed for this logic with an SPA (Single Page Application).

Great developer tools including time travel debugging.

Easier testing of how data retrieved in the client application is used.

Because the state is read only, our components just have to react to state changes. Our components role when using state management is to subscribe to the store and respond appropriately.

Redux Core Concepts

Single state tree

A single state tree is a single Javascript object. It is composed / built up using reducers.

It has an initial state. When the application loads.

An example state

const state = {
    todos: []
};
Actions

Actions have two properties

  • Type (usually a string, describing the action).
  • Payload (this is optional)

These are dispatched by components in order to update the application state but do not update the state themselves. This gives us an "immutable update pattern" which basically means that we clearly define how data is updated in our client applications.

const action = {
    type: 'ADD_TO_DO',
    payload: {
      label: 'Go for a jog',
      complete: false
    }
};
Reducers

These are pure functions (taken from the functional programming methodology) that update the application state. Given the same input they will always have the same output. They don't modify any data outside of their functional scope. This output is typically an updated application state.

This is a pure function. It does not mutate any data outside of it's own scope.

function reducer(state, action) {
    switch(action.type) {
       case: 'ADD_TO_DO': {
        const todo = action.payload;
        const todos = [...state.todos, todo],
        return [ todos]; // Update the state and pass back
       }
    }  
    return state; // Return the passed state by default
}

This would result in the state being updated

const state = {
    todos: [
      { labelL 'Go for a jog', complete: false }
    ]
};
Store
  • The state container. Injected into components that need it.
  • Components use the store to dispatch events
  • Components use the store to subscribe to state updates.
  • Handles responsibility of invoking reducers.
One-way data flow.

One way data flow in redux

Immutability

"An immutable object is an object that cannot be modified after creation."

Why would we make an object immutable?

  • Predictability
  • Explicit State Changes
  • Performance
  • Mutation Tracking
  • Undoing state changes

Semantic Web Applications

The first reference to the term "Semantic Web" was made by Tim Berners Lee back in 2011 in this publication Google Scholar Link

Wikipedia Article https://en.wikipedia.org/wiki/Semantic_Web

Personal Opinion: I studied this topic as part of my university thesis. I personally believe that the outlined solutions proposed by the W3C however interesting will not be accomodated any time soon by the majority of websites. Because of the flexible nature of HTML, CSS and Javascript coding practices websites will not provide structured markup required for the interopability of websites in this manner.

Content Management Systems,oding Frameworks by be able to incorporate these coding practices in the future and take the manual work of adding context to markup out of the developers workflow.

HTML Ontologies / Schemas

FOAF (Friend of a friend) http://xmlns.com/foaf/spec/ Open Graph Protocol http://ogp.me/ (Facebook uses this for links) W3C Spec on HTML Microdata https://www.w3.org/TR/microdata/ RDFa (RDFa is an extension to HTML5 that helps you markup things like People, Places, Events, Recipes and Reviews. Search Engines and Web Services use this markup to generate better search listings and give you better visibility on the Web, so that people can find your website more easily) https://rdfa.info/

Javascript Ontologies / Schemas

JSON LD (JSON Linked Data) https://json-ld.org/

Web Components

An introduction to web components from Mozilla https://developer.mozilla.org/en-US/docs/Web/Web_Components Custom Elements W3C Specification https://www.w3.org/TR/2018/NOTE-custom-elements-20180503/ Polymer (JS Library to create web components) https://www.polymer-project.org/

Shadow DOM

Shadow DOM refers to the ability of the browser to include a subtree of DOM elements into the rendering of a document, but not into the main document DOM tree.

Shadow DOM W3C Specificatin https://www.w3.org/TR/2018/NOTE-shadow-dom-20180301/

Frameworks that use web components

Whilst these frameworks might not follow the official W3C outlines for web components they are the most popular frameworks and libraries that are recognised as using "web components" by the community today.

  • Angular
  • Vue.js
  • React.js

Responsive Web Applications

Frontend Responsive Design

CSS Media Queries - https://css-tricks.com/a-complete-guide-to-css-media-queries/ CSS Flexbox - https://css-tricks.com/snippets/css/a-guide-to-flexbox/

Client Side Detection

Browser User Agent

Polyfills

Polyfills allow older browsers to handle HTML, CSS and Javascript features that would typically not be available with your client side code.

Server Side Detection

Server side detection is a very powerful technique that allows you to detect the users device, browser and browser capabilities before serving any content to their device. This allows you to customize the user experience and cater to any progresive functionality that their device and browser may support.

  • Using supported HTML5 API
  • Hardware capabilities
  • Supported network protocols
  • Video, audio
  • JAVA
  • Optimize the bandwidth used for the website when serving HTML, CSS, Javascript, Images, Fonts etc.

Server Side Detection Solutions

DeviceAtlas https://deviceatlas.com/ Wurfl - http://wurfl.sourceforge.net/ Scientamobile - https://www.scientiamobile.com/

IDE (Integrated Development Environments)

VSCode

WebStorm

Webtorm Guide https://www.jetbrains.com/help/webstorm/getting-started-with-webstorm.html#ws_getting_started_open_project Use GIT Bash for the terminal in Webstorm https://gist.github.com/sadikaya/f5fa699c435ebfafece2fb7d982bcdb5

Web Servers

Apache

Official documentation https://httpd.apache.org/docs/current/ .htcaccess file configuration https://httpd.apache.org/docs/current/howto/htaccess.html

Best documentation on Github: https://github.com/trimstray/nginx-quick-reference

Nginx

Official documentation https://nginx.org/en/docs/ Third party modules https://www.nginx.com/resources/wiki/modules/

IIS

Official website https://www.iis.net/overview Getting started https://docs.microsoft.com/en-gb/iis/get-started/whats-new-in-iis-10-version-1709/new-features-introduced-in-iis-10-1709

Software Architecture

Cloud Architecture

Serverless Architecture

Micro services & Distributed Systems

Docker and DevOps

Security

Node.js

Popular Node.js Frameworks, Libraries and Packages.

NPX

NPX is great. It simplifies how you can run node package binaries inside your project. It is now included with NPM by default.

https://blog.npmjs.org/post/162869356040/introducing-npx-an-npm-package-runner - An introduction

Express.js

The most popular Node.js module for creating REST API https://expressjs.com/

https://expressjs.com/en/guide/using-middleware.html - Express.js Middleware.

Express.js middleware functions are functions that have access to the request object (req), the response object (res), and the next middleware function in the application’s request-response cycle. The next middleware function is commonly denoted by a variable named next.

Nest.js

https://docs.nestjs.com/

This is my favourite Node.js framework at the moment.

If you're a React or Angular developer today using Typescript, this framework will be quite intuitive.

Nest.js is a backend framework that allows you to develop scalable APIs or any other type of backend service you require. It is built on top of Express.js but allows for a more structured enterprise-grade codebase using Typescript. You can switch out express for another type of middleware for better performance e.g. fastify.

Nest comes with it's own monorepo architecture so there's no need for NX.

Best using NX for a full stack project with both Angular / Vue / React AND Nest.js.

Setting Up Nest.
$ npm i -g @nest/cli
$ nest new <your_project_name>
$ nest generate application
Nest.js Project Information
nest info

Example Output

 _   _             _      ___  _____  _____  _     _____
| \ | |           | |    |_  |/  ___|/  __ \| |   |_   _|
|  \| |  ___  ___ | |_     | |\ `--. | /  \/| |     | |
| . ` | / _ \/ __|| __|    | | `--. \| |    | |     | |
| |\  ||  __/\__ \| |_ /\__/ //\__/ /| \__/\| |_____| |_
\_| \_/ \___||___/ \__|\____/ \____/  \____/\_____/\___/


[System Information]
OS Version     : Windows 10
NodeJS Version : v10.16.0
NPM Version    : 6.11.3

[Nest CLI]
Nest CLI Version : 7.5.1

[Nest Platform Information]
platform-express version : 7.0.0
common version           : 7.0.0
core version             : 7.0.0
Nest.js Adding Dependencies

Use the nest CLI to add dependencies as it will allow you to choose which apps and modules to add it to.

$ nest add <npm-module-name>
Nest.js Updating dependencies
nest update
Nest.js Scaffolding the framework design patterns.

You can preview the changes made which each of these commands by appending --dry-run

$ nest generate application <your_app_name>
$ nest generate class <path/your_class_name>        
$ nest generate configuration
$ nest generate controller <path/your_controller_name>
$ nest generate decorator <path/your_decorator_name>
$ nest generate filter  <path/your_filter_name>   
$ nest generate gateway <path/your_gateway_name>
$ nest generate guard <path/your_guard_name>
$ nest generate interceptor <path/your_interceptor_name>
$ nest generate interface <path/your_interface_name>  
$ nest generate middleware <path/your_middleware_name>  
$ nest generate module <path/your_module_name>       
$ nest generate pipe <path/your_pipe_name>      
$ nest generate provider <path/your_provider_name>
$ nest generate resolver <path/your_resolver_name>    
$ nest generate service <path/your_service_name>    
$ nest generate library <path/your_library_name>    
$ nest generate sub-app <sub_app_name>      
$ nest generate resource <path/resource_name>
Nest.js Running the app
# development
$ nest start <app_name> --watch

# production mode
$ nest build <app_name> (Generates native Javascript for Node.js)
$ node dist/<app_name> main.js
Nest.js Tests

Run all tests to make sure that no shared modules are broken between projects.

# unit tests
$ npm run test

# API Integration Tests
$ npm run test:e2e

# test coverage
$ npm run test:cov

Nodemon

https://nodemon.io/

Nodemon enables live reloading within your application. It will watch your source files for changes and trigger a rebuild / reload of your code when necessary.

RxJS

Creating and Subscribing to Observables with RXJS. Click here

Subjects (BehaviourSubject, ReplaySubject, Asynch) with RXJS. Click here

Operators with RXJS Click here

Other

  • MomentJS - Functional utility library
  • MomentJS - Date Utility Library
  • Axios - Promise based HTTP client for the browser and node.js
  • Cors - Express.js middleware to allow cross origin resource sharing
  • Body-Parser - Express.js request body parsing middleware to parse your API requests
  • Morgan - Express.js middleware to log incoming API requests.
  • PassportJS - Node and Express.js middleware for all of your authentication needs. Both JWT and OAuth.
  • Passport-Local - Username and password authentication strategy for Passport and Node.js.
  • Passport-JWT - Passport.js authentication using JSON Web Tokens
  • Raven - A standalone (Node.js) client for Sentry (Sentry is error reporting software).
  • Joi - Out of the box object schema validation. Useful for your data models and validating different types of data automatically.
  • Http-Status - Utility to interact with HTTP status code in Node.js
  • Lint-Staged - CLI Tool. Run linters on git staged files.
  • Husky - GIT Hooks simplified.
  • Prettier - CLI Tool. Prettier is an opinionated code formatter.
  • Eslint Config Prettier - ESLint rule template.
  • Istanbul - Code coverage Tools
  • Mocha - Unit Test Framework
  • Chai - JS Assertion Library
  • Supertest - Test your API Endpoints with Node.js code.
  • NPS - NPS is a package that solves the problem of large and complicated NPM scripts by allowing you to move your scripts to a package-scripts.js file. Because this file is a JavaScript file, you can do a lot more with your project scripts.
  • Mongoose - MongoDB Object Modelling for Node.js.

IPFS

Distributed file system that seeks to connect all computing devices with the same system of files https://ipfs.io/

Ramda

Purely functional Javascript utility Library

https://github.com/ramda/ramda

Typescript

Popular Typescript Libraries and Generators

Linting

TSLint https://palantir.github.io/tslint/ TSLint formatters https://palantir.github.io/tslint/formatters/ AirBNB TSLint (Just extend this in your TSLint configuration to use it in your project, you can override and customize these settings) https://www.npmjs.com/package/tslint-config-airbnb

Angular

Learning resources

https://ultimatecourses.com/angular - My most recommended course. https://www.tutorialspoint.com/angular6 - A useful overview of coure Angular concepts.

Other concepts

Angular Interceptors

Angular HTTP Interceptors are useful for modifying network requests and responses as they are send and received.

This can be particularly useful for authentication when an authorization token needs to be added to the headers of a request sent to an API.

Official documentation https://angular.io/guide/http#intercepting-requests-and-responses

Useful Angular Packages and Modules

Tip: Always check the component / module dependencies to avoid bloating the size of your static build files.

Enterprise Angular applications with Nx (I highly recommended this)

Nx is brilliant. It is built on top of the Angular CLI so it will feel quite familiar to other developers hat have used it before.

All of the existing Angular CLI commands will work but NX offers more functionality.

It allows you to manage your application with a more advanced CLI tool that automates more than the standard Angular CLI.

It provides a framework for Enterprise Angular development that avoids complications with managing shared libraries and avoiding unecessary pipelines.

It also helps developers to see where they can avoid code duplication, increase modularity and avoid introducing breaking changes.

  • Extra Code Generation
  • Workspace Management
  • NgRx State Management Code Generation
  • Data Persistence Management
  • Linters
  • Code Formatting
  • UpgradeModule and dowgradeModule helpers
  • Vulnerability scanning

Find out more at https://nrwl.io/nx/overview

How Nx structures your Enterprise Angular Applications

<project-root-directory>
apps/
  website1.com/
    src/
  website2.com
    src/
libs/
  website1.com/
    src/
  website2.com/
    src/
  ui-component-library
    src/
  data-services
    src/
  • You now don't need separate GIT repos!
  • Our applications are independenly deployable!
  • You can clearly see how your applications and shared node modules relate to each other!
  • No more CI/CD Pipelines for every shared library we create!
  • We can clearly see when things break for ANY application!
  • No GIT submodules :)
  • No cloning different GIT repos to see how a shared library works (easier to keep everything on context).

Each library has an index.ts fie so we can use shared modules in our repo as if they we normal NPM modules. For example we won't need to do this

import { CustomModule } from '@ourorgansiation/custom/src/custom.module'

We can still do this..

import { CustomModule } from '@ourorgansiation/custom'
Some good articles and videos

https://auth0.com/blog/create-custom-schematics-with-nx/ - Creating custom schemantics with Nx https://www.youtube.com/watch?v=bMkKz8AedHc - Supercharging the Angular CLI with Nx by James Henry https://blog.nrwl.io/building-full-stack-applications-using-angular-cli-and-nx-5eff205248f1 - Creating

Creating custom Angular Schematics with NX

Angular schematics are a very interesting topic for me personally. Enterprise developers can use Angular Schematics to streamline code generation in projects. The Angular CLI comes when many code generation commands which is powerful for scaffolding applications. These in themselves are Angular Schematics and you can roll your own for your projects or enterprise workspaces. Add Nx to the mix and developers can automate much of their development workflow.

  • Library generation
  • Component generation
  • Unit Test Generation

Whatever you want form your code structure really..

Benefits
  • Enforcing design patterns
  • Enforcing modularity of code
  • Enforcing naming conventions

https://auth0.com/blog/create-custom-schematics-with-nx/

Final notes on NX
Reference Commands

A comprehensive list of commands can be found here https://github.com/nrwl/nx-examples (Highly recommended)

State Management with NgRX for Angular.

NgRX at it's core is made up of NgRX Store and Effects. It utilizes the popular Rx.js library and it's observable design pattern to provide a Redux architecture for state management in Angular.

There are other libraries than can be used with NgRX but they are optional.

Note: Code scaffolding for NgRX can be achived with NX Angular Schemantics but it is important to understand how the Redux pattern and NgRX specifcally work with Angular.

What is NgRX Store?

Official documentation https://ngrx.io/guide/store

Inspired by the Redux architecture NgRX Store using RXJS Observables to provide state management in Angular.

Benefits of NgRX Store
  • Single source of truth
  • Testability
  • Performance Benefits (one-way data flow. No unexpected events in your components around data management).
  • Root and feature module support (eager ad laxy loaded modules).
Store selectors

This is probably the toughest part about NgRX in general but they are well worth implementing

A very basic component implementation interacting with the store would look something like this..

import { Store } from '@ngrx/store'
// other imports

export class YourCustomComponent implements OnInit {

   constructor(private store: Store<fromStore.ToDosState>) {
    
   }
   
   ngOnInit() {
    this.store.select<>('todos').subscribe() // etc
   }

};

You can however define functions in your store that care of all of the logic for navigating the state and getting the data your are looking for. These can be imported and used to select data specifically from your store. Your component can then store the observables and use asynch in your components to handle changes to the state reactively.

You can use a naming convention for observables that are stored on your component. Typically using the $ at the end of the variables name is a good idea.

import { Store } from '@ngrx/store'
// other imports

export class YourCustomComponent implements OnInit {

   todos$: Observable<ToDo[]>

   constructor(private store: Store<fromStore.ToDosState>) {
    
   }
   
   ngOnInit() {
    this.todos$ = this.store.select>(fromStore.getAllToDos)
   }

};

Then in your HTML for the component

<div *ngIf="!((todos$ | async)?.length)">
    These are currently no to do items
</div>
<to-do-item *ngFor="let todo of (todos$ | async)" [todo]="todo"></to-do-item>

This observables and async approach makes your components implement a reactive architecture.

What are NgRX Effects?

Official doccuentation https://ngrx.io/guide/effects

  • Listen for ngrx/store actions that are dispatched
  • Isolate side effects from components
  • Communicate outside of Angular (Handle API calls, websockets etc)

NgRX Effects Flow

Because reducers are pure functions these are not ideal places to put API calls

If an effect fetches data from an external source then it will dispatch an action of it's own with a reducer handling it and updating the state

Effects don't perform service calls themselves. They will typically offload that work to an Angular service.

NGRX Facades

This pattern is purely optional but once implemented provides a much cleaner implementation with NgRX on the component level.

I haven't used this approach myself yet but would like to. I'm curious about how it effects testing.

Its also supported by Nx Angular Schematics!

https://medium.com/@thomasburleson_11450/ngrx-facades-better-state-management-82a04b9a1e39

NGRX Entities

A great tutorial https://blog.angular-university.io/ngrx-entity/#operationssupportedbythengrxentityadapter

Can be used with NgRX Schematics and provides..

  • Entity State adapter for managing record collections by ID reference.
  • Entity provides an API to manipulate and query entity collections.
  • Reduces boilerplate for creating reducers that manage a collection of models.
  • Provides performant CRUD operations for managing entity collections.
  • Extensible type-safe adapters for selecting entity information.

Redux Dev Tools - Highly recommended

This will allow you see every single action and state change in your app.

It really showcases how reactive your application is when built using the redux pattern.

You can also step back in time to an older state to debug issues with your application by dragging the slider.

It will also visualize your state tree in an easy to understand way if you index your data with the entities approach outlined above.

You can download the extension here http://extension.remotedev.io/

Setting up NGRX Router Store

This allows us to integrate our router state to the redux store in our application which is quite useful for even more debugging with our dev tools. It also makes our application more reactive.

** I will be providing a link here to my own reference Angular application with state management **

My own lessons from implementing state management with NgRX

  • State Management with Angular requires a lot of extra code. This can be generated automatically though with the CLI once NgRX schematics have been added to your project https://ngrx.io/guide/schematics . It's even better using NX https://nrwl.io/nx/guide-setting-up-ngrx
  • Implement container components for each feature (these are basically the top most parent components for each module).
  • Hook up your router to the state
  • Make both your routing and store actions resource based (from the API down to the store for consistency, bespoke functionality can be implemented at the component level).
  • Keep your stores inside of the relevant modules only. This will allow you to keep your feature implementations modular.
  • Optimize your folder structure with index.ts files wherever possible to avoid too many imports.

Other state management solutions for Angular

NgXS - https://ngxs.gitbook.io/ngxs Akkita - https://blog.angularindepth.com/state-management-in-angular-using-akita-82f117d282dd Mobx - https://mobx.js.org/getting-started.html - can be plugged into Angular https://github.com/mobxjs/mobx-angular

https://blog.singular.uk/why-i-prefer-ngxs-over-ngrx-df727cd868b5 - A comparison between NgXS and NgRX

Angular Module / Folder Organisation Best Practices

This is a controversial topic depending on which developers / teams and organisations you ask.

I've personally learned over time that as long as the structure is modular, maintainable and as intuitive as possible for everyone then that's the best compromise.

Angular Unit Testing

Karma

Official documentation https://karma-runner.github.io/latest/index.html

Karma comes with Angular for your unit tests by default and many coverage and reporting tools are available.

You can also choose to run your unit tests with ChromeHeadless to avoid spinning up a new browser window every time.

JEST

Official documentation https://jestjs.io/docs/en/getting-started

Replacing the default karma unit test suite wih Jest can result in many benefits for your application.

https://www.xfive.co/blog/testing-angular-faster-jest/

  • Faster (does not require a browser to run the tests)
  • Easier to debug as the error messages are less verbose
  • The browser is mocked and the tests run in JSDOM instead (much, much faster!)
Useful Libraries

Angular End to End Testing

Protractor

BDD Testing with Protractor and Cucumber

Behaviour-Driven Development (BDD) is a collaborative approach to software development that bridges the communication gap between business and IT. BDD helps teams communicate requirements with more precision, discover defects early and produce software that remains maintainable over time.

https://dev.to/fzwael/bdd-with-angular-and-cucumberjs-p8

Cypress

https://www.cypress.io/ - Official website https://chrome.google.com/webstore/detail/cypress-scenario-recorder/fmpgoobcionmfneadjapdabmjfkmfekb?hl=en - Useful Chrome Extension for generating script.

Provides

  • Time Travel
  • Debuggability
  • Automatic Waiting
  • Spies, Stubs, and Clocks
  • Network Traffic Control
  • Consistent Results
  • Screenshots and Videos

https://blog.nrwl.io/nrwl-nx-7-0-better-e2e-testing-with-cypress-1b88336bef5e - Better E2E Testing with Cypress

Storybook

https://storybook.js.org/docs/react/get-started/introduction

Storybook allows for a more collaborative approach to developing UI components with React, Vue and Angular.

QA, Developers and UX Teams can use it to develop more robust components that are more general and reusable across business logic and specific use cases.

Angular Documentation Tools

Compodoc

https://compodoc.app/ - Excellent Angular project documentation tool

Source map explorer

To use source-map-explorer, first install it globally using your favorite package manager:

npm install -g source-map-explorer

yarn global add source-map-explorer

And then build your angular application with –source-map flag set to true:

ng build --source-map=true --prod

And finally run it on one or more of your generated bundles. For instance, main bundle:

npx source-map-explorer dist/main.xxx.bundle.js

https://www.youtube.com/watch?v=7aY9BoMEpG8 - Video from Google developers

Webpack Bundle Analyzer

The Angular CLI uses webpack in the background to build your application.

You can analyze the static files in your build for a breakdown of exactly what dependencies you have included.

To get started with this tool, install is a dev dependency in your project using your favorite package manager:

npm i webpack-bundle-analyzer --save-dev

yarn add webpack-bundle-analyzer –dev

Then, build your application with –stats-json flag:

ng build --prod --stats-json

And finally, run Webpack Bundle Analyzer against the generated stats.json file located under dist/app-name for angular 6, and dist for angular 5 and below:

npx webpack-bundle-analyzer dist/APP-NAME/stats.json

Your app name can be found inside your angular.json configuration file.

TSLint

You can use this to monitor the code quality of the typescript in your application.

It is highly custmizable with it's set of rules and you can even use other organisation's code standards.

For more information visit the official website https://palantir.github.io/tslint/

Angular Reporting Tools

Code Coverage Reporting Tools

Istanbul

A fantastic code coverage reporting tool. Official documentation can be found here https://istanbul.js.org/

E2E Test Reporting Tools

Protractor HTML Reporter

This is quite useful for vizualizing your E2E Test Results. It can also display screenshots for debugging.

https://github.com/etxebe/protractor-html-reporter

Progressive Web Applications

Service workers

Note: You will need HTTPS on your website to fully utilise this browser feature.

https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API - Mozilla Service Worker API Guide.

App Cache Manifest

Use a service worker instead :)

Build Tools for Web Applications

The role of a build tool in web applications is take the source development assets for your application and produce optimized assets that will be used to run your application.

This can include tasks such as

  • Copy, deleting files
  • Merging files
  • Compiling code
  • Optimizing code
  • Removing spaces
  • Producing index files
  • Optimizing images, fonts other assets.
  • Any other tasks you wish to automate with your application setup.

Grunt

Javascript Task runner with an impressive list of plugins https://gruntjs.com/

Gulp

Another Javascript build tool for automating common tasks https://gulpjs.com/

Webpack

Angular uses webpack.

Browser Build Optimization

  • Semantic browser applications: Schema generation.
  • Optimizing Images (PNG, SVG,)
  • Optimizing fonts
  • Minification of files
  • Concatenation of files

Progressive Browser Functionality

Native application development using Web Technologies

Cordova

Develop applications for native operating systems with HTML, CSS and Javascript https://cordova.apache.org/

Electron

Develop desktop applications with HTML, CSS and Javascript https://electronjs.org/

Ionic Native

  • Nativescript

NW.js (Previously node-webkit)

Lets you call all Node.js modules directly from DOM and enables a new way of writing applications with all Web technologies.

https://nwjs.io/

Version Control

Source / Version Control allows developers to share and track each individual change to a codebase.

It allows developers to work on the same piece of code and can automate much of the tedious aspects of resolving these conflicts.

Mercurial

https://www.mercurial-scm.org/

SVN (Subversion)

https://subversion.apache.org/

GIT - Now the most popular

How merkle trees work https://en.wikipedia.org/wiki/Merkle_tree

Continuous Deployment / Integration / Delivery

Continuous deployment is a strategy for software releases wherein any code commit that passes the automated testing phase is automatically released into the production environment, making changes that are visible to the software's users.

Continuous Integration (CI) is a development practice where developers integrate code into a shared repository frequently, preferably several times a day. Each integration can then be verified by an automated build and automated tests. While automated testing is not strictly part of CI it is typically implied.

Continuous delivery (CD) is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time and, when releasing the software, doing so manually. It aims at building, testing, and releasing software with greater speed and frequency. The approach helps reduce the cost, time, and risk of delivering changes by allowing for more incremental updates to applications in production. A straightforward and repeatable deployment process is important for continuous delivery.

Cloud Services for Continuous Deployment / Integration / Delivery

Microsoft Azure

https://azure.microsoft.com/en-gb/services/ - Overview of products and solutions

Google Cloud Services

https://cloud.google.com/products/ - Overview of products and solutions

Deployment Environments

This can be as simple as having a development, QA, UAT and production server but there are many different configurations that can be useful depending on your project.

Environment variables: Why and how.

Storing sensitive information in your GIT repository makes you vulnerable to unauthorised access.

Because of this it is more useful to have these values parameterised and injected later by your hosting provider or pipeline configuration.

Environment variables can be used for database credentials, secret keys or any other type of private key / password values.

It also allows for a more synchronised approach when having to update these values later on as the values can be changed in one place instead of across multiple servers in the case of scaled applications.

Private Registries / Artefact Repositories

Verdaccio (NPM Registry) JAVA - Maven Gradle repositories.

CI Platforms and Tools

IAAS (Infrastructure as a service)

https://www.netlify.com/ - All-in-one platform for automating modern web projects.

Security on registries / artifact repositories

Provisioning Tools / Virtualization / Snapshots

Virtual environments are extremely useful for ensuring that you have a consistent OS and dependency layer for your development and deployment environments.

It takes away the time consuming tasks of provisioning each individual environment and list of dependencies when scaling too.

For developers it can also keep the list of running processed on their local machines free of any bloat from previous projects.

Vagrant

https://www.vagrantup.com/intro

Simple, intituive virtual environments (they use more disk space however).

Docker

https://www.docker.com/why-docker

Container based approach. Lightweight environments.

Fastest Environment Setup

  1. Download and Install Docker Desktop for Windows

https://download.docker.com/win/stable/Docker%20Desktop%20Installer.exe

  1. Download and install AWS CLI tool

https://s3.amazonaws.com/aws-cli/AWSCLI64PY3.msi

on your terminal type

aws configure

Access Key Id - <your key>
Secret Access Key - <your secret>
Region - eu-west-1
Format - json

That should save your credentials on ~/.aws/credentials and config files consecutively.

Now you can login onto AWS ECR services

$(aws ecr get-login --no-include-email)
  1. Run docker compose to spawn each service
docker-compose up -d

This should download images from ECR repository and start services consecutively. If you have Kitematic installed you'll be able to see the logs of each container and your App should start at 127.0.0.1:8080

Install Docker and Kitematic

  1. Install https://www.docker.com/products/docker-desktop
  2. Use the Native Docker GUI (Docker Desktop)
  3. Install Kitematic (optional but recommended)

Build docker images (assuming you're in the folder where docker-compose.yml lives)

https://hub.docker.com/search?q=&type=image - Search for a docker image

docker build -t <image-name> <whatever-custom-container-name>/.

Run docker compose

Run

docker-compose up -d

and in Kitematic, view the logs for your custom container.

Helpful Docker commands and code snippets

CONTAINERS

Pull an image from the repo

docker pull <image>

This will run a container based on its image. If no entrypoint is determined it should redirect to its terminal. When running containers add tag "--name" for easy access and identification. Also adding a tag "--rm" it removes it from a dangling list once exited.

docker run -it <image:tag>

To overwrite a default entrypoint of a container

docker run -it --entrypoint bash <image:tag>

List all running containers, add "-a" to list all containers including the stopped ones

docker ps

Stop all containers

docker stop $(docker ps -a -q)

Remove all containers

docker rm -f $(docker ps -a -q)

Remove container by container ID

docker rm -f $(sudo docker ps --before="container_id_here" -q)

CLI access into a container

This works only on a running container, the reason is beacuse containers are supposed to be immutable.

docker exec -it $(docker container ls  | grep '<seach_term>' | awk '{print $1}') sh

Exec into container on windows with Git Bash

winpty docker exec -it $(docker container ls  | grep '<seach_term>' | awk '{print $1}') sh

To exit above use 'ctrl p', 'ctrl q' (don't exit or it will be in exited state)

docker rm $(docker ps -q -f status=exited) # remove all exited containers

IMAGES

List images and containers

docker images | grep "search_term_here"

Remove image(s) (must remove associated containers first)

docker rmi -f image_id_here # remove image(s) docker rmi -f $(docker images -q) # remove ALL images!!! docker rmi -f $(docker images | grep "^<none>" | awk '{print $3}') # remove all <none> images docker rmi -f $(docker images | grep 'search_term_here' | awk '{print $1}') # i.e. 2 days ago docker rmi -f $(docker images | grep 'search_1\|search_2' | awk '{print $1}')

DELETE BOTH IMAGES AND CONTAINERS

docker images && docker ps -a

Stop and remove containers and associated images with common grep search term

docker ps -a --no-trunc  | grep "search_term_here" | awk "{print $1}" | xargs -r --no-run-if-empty docker stop && \
docker ps -a --no-trunc  | grep "search_term_here" | awk "{print $1}" | xargs -r --no-run-if-empty docker rm && \
docker images --no-trunc | grep "search_term_here" | awk "{print $3}" | xargs -r --no-run-if-empty docker rmi

Stops only exited containers and delete only non-tagged images

docker ps --filter 'status=Exited' -a | xargs docker stop docker images --filter "dangling=true" -q | xargs docker rmi

DELETE NETWORKS AND VOLUMES

Clean up orphaned volumes

docker volume rm $(docker volume ls -qf dangling=true)

Clean up orphaned networks

docker network rm $(docker network ls -q)

NEW IMAGES/CONTAINERS

Create new docker container, ie. ubuntu

docker pull ubuntu:latest docker run -i -t ubuntu /bin/bash # drops you into new container as root

OTHER

Install docker first using directions for installing latest version

https://docs.docker.com/installation/ubuntulinux/#ubuntu-trusty-1404-lts-64-bit

Other great tips:

http://www.centurylinklabs.com/15-quick-docker-tips/

Fix fig / docker config:

https://gist.github.com/RuslanHamidullin/94d95328a7360d843e52

PM2 (Production Process Manager)

PM2 or Process Manager 2, is an Open Source, production Node.js process manager helping Developers and Devops manage Node.js applications in production environment. In comparison with other process manager like Supervisord, Forever, Systemd, some key features of PM2 are automatic application load balancing, declarative application configuration, deployment system and monitoring.

Commands

Fork mode

pm2 start app.js --name my-api # Name process

Cluster mode

pm2 start app.js -i 0        # Will start maximum processes with LB depending on available CPUs
pm2 start app.js -i max      # Same as above, but deprecated.
pm2 scale app +3             # Scales `app` up by 3 workers
pm2 scale app 2              # Scales `app` up or down to 2 workers total

Listing

pm2 list               # Display all processes status
pm2 jlist              # Print process list in raw JSON
pm2 prettylist         # Print process list in beautified JSON

pm2 describe 0         # Display all informations about a specific process

pm2 monit              # Monitor all processes

Logs

pm2 logs [--raw]       # Display all processes logs in streaming
pm2 flush              # Empty all log files
pm2 reloadLogs         # Reload all logs

Actions

pm2 stop all           # Stop all processes
pm2 restart all        # Restart all processes

pm2 reload all         # Will 0s downtime reload (for NETWORKED apps)

pm2 stop 0             # Stop specific process id
pm2 restart 0          # Restart specific process id

pm2 delete 0           # Will remove process from pm2 list
pm2 delete all         # Will remove all processes from pm2 list

Misc

pm2 reset <process>    # Reset meta data (restarted time...)
pm2 updatePM2          # Update in memory pm2
pm2 ping               # Ensure pm2 daemon has been launched
pm2 sendSignal SIGUSR2 my-app # Send system signal to script
pm2 start app.js --no-daemon
pm2 start app.js --no-vizion
pm2 start app.js --no-autorestart

Kubernetes Container Orchestration

https://kubernetes.io/docs/home/ - Kubernetes Documentation

Removing friction from the development workflow

Semantic Versioning

https://github.com/semantic-release/semantic-release

Automate release notes

https://github.com/semantic-release/changelog - Useful NPM library that works with the conventional commit to gather commits and generate changelogs for each release.

Avoiding scope creep

https://www.teamgantt.com/guide-to-project-management/taming-scope-creep

Software Development and Testing Strategies

Principles of Software Engineering

  • Rigor and formality
  • Separation of concerns
  • Modularity and decomposition
  • Abstraction
  • Anticipation of change
  • Generality
  • Incremental Development
  • Reliability

Software Engineering Concepts

Flux Architecture

https://facebook.github.io/flux/docs/overview.html#content - Flux Architecture

DRY Principles

https://thevaluable.dev/dry-principle-cost-benefit-example/

Service-Oriented Architecture

https://www.ibm.com/support/knowledgecenter/en/SSMQ79_9.5.1/com.ibm.egl.pg.doc/topics/pegl_serv_overview.html

Microservices

https://microservices.io/presentations/index.html - Avoid the monolith

Functional Programming

https://www.tutorialspoint.com/functional_programming/functional_programming_introduction.htm

Reactive Programming

https://medium.com/@kevalpatel2106/what-is-reactive-programming-da37c1611382

Unit Testing

https://www.tutorialspoint.com/software_testing_dictionary/unit_testing.htm

Load Testing

https://www.tutorialspoint.com/software_testing_dictionary/load_testing.htm

Code Coverage

https://www.educba.com/code-coverage/

Integration Testing

https://softwaretestingfundamentals.com/integration-testing/

BDD (Behaviour driven development)

https://www.agilealliance.org/glossary/bdd/

End to end testing

https://www.guru99.com/end-to-end-testing.html

SOLID Principles

https://medium.com/mindorks/solid-principles-explained-with-examples-79d1ce114ace

S - Single Responsibility Principle (known as SRP) O - Open/Closed Principle L - Liskov’s Substitution Principle I - Interface Segregation Principle D - Dependency Inversion Principle

Cross Browser Testing

https://www.browserstack.com/cross-browser-testing

Networking

OSI Model

OSI Model

The OSI Model (Open Systems Interconnection Model) is a conceptual framework used to describe the functions of a networking system. The OSI model characterizes computing functions into a universal set of rules and requirements in order to support interoperability between different products and software. In the OSI reference model, the communications between a computing system are split into seven different abstraction layers: Physical, Data Link, Network, Transport, Session, Presentation, and Application.

Created at a time when network computing was in its infancy, the OSI was published in 1984 by the International Organization for Standardization (ISO). Though it does not always map directly to specific systems, the OSI Model is still used today as a means to describe Network Architecture.

The 7 Layers of the OSI Model

Physical Layer

The lowest layer of the OSI Model is concerned with electrically or optically transmitting raw unstructured data bits across the network from the physical layer of the sending device to the physical layer of the receiving device. It can include specifications such as voltages, pin layout, cabling, and radio frequencies. At the physical layer, one might find “physical” resources such as network hubs, cabling, repeaters, network adapters or modems.

Data Link Layer

At the data link layer, directly connected nodes are used to perform node-to-node data transfer where data is packaged into frames. The data link layer also corrects errors that may have occurred at the physical layer.

The data link layer encompasses two sub-layers of its own. The first, media access control (MAC), provides flow control and multiplexing for device transmissions over a network. The second, the logical link control (LLC), provides flow and error control over the physical medium as well as identifies line protocols.

Network Layer

The network layer is responsible for receiving frames from the data link layer, and delivering them to their intended destinations among based on the addresses contained inside the frame. The network layer finds the destination by using logical addresses, such as IP (internet protocol). At this layer, routers are a crucial component used to quite literally route information where it needs to go between networks.

Transport Layer

The transport layer manages the delivery and error checking of data packets. It regulates the size, sequencing, and ultimately the transfer of data between systems and hosts. One of the most common examples of the transport layer is TCP or the Transmission Control Protocol.

Session Layer

The session layer controls the conversations between different computers. A session or connection between machines is set up, managed, and termined at layer 5. Session layer services also include authentication and reconnections.

Presentation Layer

The presentation layer formats or translates data for the application layer based on the syntax or semantics that the application accepts. Because of this, it at times also called the syntax layer. This layer can also handle the encryption and decryption required by the application layer.

Application Layer

At this layer, both the end user and the application layer interact directly with the software application. This layer sees network services provided to end-user applications such as a web browser or Office 365. The application layer identifies communication partners, resource availability, and synchronizes communication.

HTTP - Hypertext Transfer Protocol (HTTP)

https://developer.mozilla.org/en-US/docs/Web/HTTP

RPC - Remote Procedure Call

Most computer programs run procedures, or sets of instructions, using the computer's CPU. In other words, the instructions are processed locally on the same computer that the software is running from. Remote procedure calls, however, run procedures on other machines or devices connected to a network. Once the instructions have been run, the results of the procedure are usually returned to the local computer.

For example, a computer without a hard drive may use an RPC to access data from a network file system (NFS). When printing to a network printer, a computer might use an RPC to tell the printer what documents to print. A client system connected to a database server may execute an RPC to process data on the server.

Remote procedure calls are based on the client-server model, where multiple client computers may connect to a server and retrieve data from it. RPCs are typically written in a standard format, such as XML, so that the procedures can be understood by multiple computer platforms. For example, an XML-RPC sent by a Windows computer could be recognized by a Macintosh or Unix-based system

Databases

Database Types

SQL / RDBMS / Relational databases

RDBMS are more widely known and understood than their NoSQL cousins. Relational databases emerged in the 70’s to store data according to a schema that allows data to be displayed as tables with rows and columns. Think of a relational database as a collection of tables, each with a schema that represents the fixed attributes and data types that the items in the table will have. RDBMS all provide functionality for reading, creating, updating, and deleting data, typically by means of Structured Query Language (SQL) statements.

The tables in a relational database have keys associated with them, which are used to identify specific columns or rows of a table and facilitate faster access to a particular table, row, or column of interest.

Data integrity is of particular concern in relational databases, and RDBMS use a number of constraints to ensure that the data contained in your tables is reliable and accurate.

While there are many relational databases, over time these have become the most popular:

  • Oracle - Oracle Database (commonly referred to as Oracle RDBMS or simply as Oracle) is a multi-model database management system produced and marketed by Oracle Corporation.
  • MySQL - MySQL is an open source RDBMS based on Structured Query Language (SQL). MySQL runs on virtually all platforms, including Linux, UNIX, and Windows.
  • Microsoft SQL Server - Microsoft SQL Server is an RDBMS, that supports a wide variety of transaction processing, business intelligence and analytics applications in corporate IT environments.
  • PostgreSQL - PostgreSQL, often simply Postgres, is an object-relational database management system (ORDBMS) with an emphasis on extensibility and standards compliance.
  • DB2 - DB2 is an RDBMS designed to store, analyze and retrieve data efficiently.
Advantages
  • Relational databases are well-documented and mature technologies, and RDBMS are sold and maintained by a number of established corporations.
  • SQL standards are well-defined and commonly accepted.
  • A large pool of qualified developers have experience with SQL and RDBMS.
  • All RDBMS are ACID-compliant, meaning they satisfy the requirements of Atomicity, Consistency, Isolation, and Durability.
Disadvantages
  • RDBMSes don’t work well — or at all — with unstructured or semi-structured data, due to schema and type constraints. This makes them ill-suited for large analytics or IoT event loads.
  • The tables in your relational database will not necessarily map one-to-one with an object or class representing the same data.
  • When migrating one RDBMS to another, schemas and types must generally be identical between source and destination tables for migration to work (schema constraint). For many of the same reasons, extremely complex datasets or those containing variable-length records are generally difficult to handle with an RDBMS schema.

NoSQL / Non-relational databases

NoSQL databases emerged as a popular alternative to relational databases as web applications became increasingly complex. NoSQL/Non-relational databases can take a variety of forms. However, the critical difference between NoSQL and relational databases is that RDBMS schemas rigidly define how all data inserted into the database must be typed and composed, whereas NoSQL databases can be schema agnostic, allowing unstructured and semi-structured data to be stored and manipulated.

Types

Note that some products may fall into more than one category. For example, Couchbase is both a document database and a key-value store.

Key-Value Stores

Key-Value Stores, such as Redis and Amazon DynamoDB, are extremely simple database management systems that store only key-value pairs and provide basic functionality for retrieving the value associated with a known key.

The simplicity of key-value stores makes these database management systems particularly well-suited to embedded databases, where the stored data is not particularly complex and speed is of paramount importance.

Wide-Column Stores

Wide Column Stores, such as Cassandra, Scylla, and HBase, are schema-agnostic systems that enable users to store data in column families or tables, a single row of which can be thought of as a record — a multi-dimensional key-value store.

These solutions are designed with the goal of scaling well enough to manage petabytes of data across as many as thousands of commodity servers in a massive, distributed system.

Although technically schema-free, wide column stores like Scylla and Cassandra use an SQL variant called CQL for data definition and manipulation, making them straightforward to those already familiar with RDBMS.

Document Stores

Document Stores, including MongoDB and Couchbase, are schema-free systems that store data in the form of JSON documents. Document stores are similar to key-value or wide column stores, but the document name is the key and the contents of the document, whatever they are, are the value.

In a document store, individual records do not require a uniform structure, can contain many different value types, and can be nested. This flexibility makes them particularly well-suited to manage semi-structured data across distributed systems.

Graph Databases

Graph Databases, such as Neo4J and Datastax Enterprise Graph, represent data as a network of related nodes or objects in order to facilitate data visualizations and graph analytics.

A node or object in a graph database contains free-form data that is connected by relationships and grouped according to labels. Graph-Oriented Database Management Systems (DBMS) software is designed with an emphasis on illustrating connections between data points.

As a result, graph databases are typically used when analysis of the relationships between heterogeneous data points is the end goal of the system, such as in fraud prevention, advanced enterprise operations, or Facebook’s original friends graph.

Search Engines

Search Engines, such as Elasticsearch, Splunk, and Solr, store data using schema-free JSON documents. They are similar to document stores, but with a greater emphasis on making your unstructured or semi-structured data easily accessible via text-based searches with strings of varying complexity.

Advantages of NoSQL Databases

Since there are so many types and varied applications of NoSQL databases, it’s hard to nail these down, but generally:

  • Schema-free data models are more flexible and easier to administer.
  • NoSQL databases are generally more horizontally scalable and fault-tolerant.
  • Data can easily be distributed across different nodes. To improve availability and/or partition tolerance, you can choose that data on some nodes be "eventually consistent".
Disadvantages of NoSQL Databases

These are also dependent on the database type.

  • NoSQL databases are generally less widely adopted and mature than RDBMS solutions, so specific expertise is often required.
  • There are a range of formats and constraints specific to each database type.

Which database is right for you?

  • If ACID (Atomicity, Durability, Consistency, and Durability) compliance is your first priority, consider using RDBMS.
  • If you have a massively distributed system and can settle for eventual consistency on some nodes/partitions, you might consider a wide column store such as Cassandra or Scylla.
  • If your input data is particularly heterogeneous and difficult to encapsulate according to a normalization schema, consider using a NoSQL DBMS.
  • If your goal is to scale vertically, consider an RDBMS; conversely, if you want to scale horizontally, a NoSQL DBMS may be preferable.

Database Concepts

ACID (Atomicity, Consistency, Isolation, and Durability)

ACID properties are an important concept for databases. The acronym stands for Atomicity, Consistency, Isolation, and Durability.

Atomicity

The phrase "all or nothing" succinctly describes the first ACID property of atomicity. When an update occurs to a database, either all or none of the update becomes available to anyone beyond the user or application performing the update. This update to the database is called a transaction and it either commits or aborts. This means that only a fragment of the update cannot be placed into the database, should a problem occur with either the hardware or the software involved. Features to consider for atomicity:

a transaction is a unit of operation - either all the transaction's actions are completed or none are atomicity is maintained in the presence of deadlocks atomicity is maintained in the presence of database software failures atomicity is maintained in the presence of application software failures atomicity is maintained in the presence of CPU failures atomicity is maintained in the presence of disk failures atomicity can be turned off at the system level atomicity can be turned off at the session level

Consistency

Consistency is the ACID property that ensures that any changes to values in an instance are consistent with changes to other values in the same instance. A consistency constraint is a predicate on data which serves as a precondition, post-condition, and transformation condition on any transaction.

Isolation

The isolation portion of the ACID Properties is needed when there are concurrent transactions. Concurrent transactions are transactions that occur at the same time, such as shared multiple users accessing shared objects. This situation is illustrated at the top of the figure as activities occurring over time. The safeguards used by a DBMS to prevent conflicts between concurrent transactions are a concept referred to as isolation.

As an example, if two people are updating the same catalog item, it's not acceptable for one person's changes to be "clobbered" when the second person saves a different set of changes. Both users should be able to work in isolation, working as though he or she is the only user. Each set of changes must be isolated from those of the other users.

An important point in understanding isolation through transactions is serializability. Transactions are serializable when the effect on the database is the same whether the transactions are executed in serial order or in an interleaved fashion. The effect on the DBMS is that the transactions may execute in serial order based on consistency and isolation requirements. It is important to note that a serialized execution does not imply the first transactions will automatically be the ones that will terminate before other transactions in the serial order.

######## Degrees of isolation:

  • degree 0 - a transaction does not overwrite data updated by another user or process ("dirty data") of other transactions
  • degree 1 - degree 0 plus a transaction does not commit any writes until it completes all its writes (until the end of transaction)
  • degree 2 - degree 1 plus a transaction does not read dirty data from other transactions
  • degree 3 - degree 2 plus other transactions do not dirty data read by a transaction before the transaction commits
Durability

Maintaining updates of committed transactions is critical. These updates must never be lost. The ACID property of durability addresses this need. Durability refers to the ability of the system to recover committed transaction updates if either the system or the storage media fails.

######## Features to consider for durability:

  • recovery to the most recent successful commit after a database software failure
  • recovery to the most recent successful commit after an application software failure
  • recovery to the most recent successful commit after a CPU failure
  • recovery to the most recent successful backup after a disk failure
  • recovery to the most recent successful commit after a data disk failure

Concurrency and Locking

Concurrency control and locking is the mechanism used by DBMSs for the sharing of data. Atomicity, consistency, and isolation are achieved through concurrency control and locking.

While many people may be reading the same data item at the same time, it is usually necessary to ensure that only one application at a time can change a data item. Locking is a way to do this. Because of locking, all changes to a particular data item will be made in the correct order in a transaction.

The amount of data that can be locked with the single instance or groups of instances defines the granularity of the lock.

Granularity Types

  • Page Locking
  • Cluster Locking
  • Class or Table Locking
  • Object or Instance Locking

Page locking

Page locking (or page-level locking) concurrency control involves all the data on a specific page being locked. A page is a common unit of storage in computer systems and is used by all types of DBMSs. Locking for objects is on the left and page locking for relational tuples is on the right. If the concept of pages is new to you, just think of a page as a unit of space on the disk where multiple data instances are stored.

Cluster locking

Cluster locking or container locking for concurrency control means all data clustered together (on a page or multiple pages) will be locked simultaneously. This applies only to clusters of objects in ODBMSs (Object database management systems).

Class or Table Locking

Class or table locking means that all instances of either a class or table are locked.

Object or Instance Locking

Instance locking locks a single relational tuple in an RDBMS or a single object in an ODBMS

Database Normalisation

Database normalization is process used to organize a database into tables and columns. The idea is that a table should be about a specific topic and that only those columns which support that topic are included. For example, a spreadsheet containing information about sales people and customers serves several purposes:

  • Identify sales people in your organization
  • List all customers your company calls upon to sell product
  • Identify which sales people call on specific customers.

By limiting a table to one purpose you reduce the number of duplicate data that is contained within your database, which helps eliminate some issues stemming from database modifications.

To assist in achieving these objectives, some rules for database table organization have been developed. The stages of organization are called normal forms in database normalisation; there are three normal forms / rules most databases adhere to using.

As tables satisfy each successive database normalization form, they become less prone to database modification anomalies and more focused toward a sole purpose or topic.

There are three main reasons to normalize a database.

  • 1NF - minimize duplicate data
  • 2NF - minimize or avoid data modification issues
  • 3NF - simplify queries.

There are several additional forms, such as BCNF, but I consider those advanced, and not too necessary to learn in the beginning.

The forms are progressive, meaning that to qualify for 3rd normal form a table must first satisfy the rules for 2nd normal form, and 2nd normal form must adhere to those for 1st normal form. Before we discuss the various forms and rules in detail, let’s summarize the various forms:

First Normal Form – The information is stored in a relational table and each column contains atomic values, and there are not repeating groups of columns. Second Normal Form – The table is in first normal form and all the columns depend on the table’s primary key. Third Normal Form – the table is in second normal form and all of its columns are not transitively dependent on the primary key.

MySQL

Official documentation https://dev.mysql.com/doc/

Tools

MySQL Workbench

MS SQL Server

Official documentation https://docs.microsoft.com/en-us/sql/sql-server/sql-server-technical-documentation?view=sql-server-2017

Postgres

Official documentation https://www.postgresql.org/docs/manuals/

Postgres Privileges

Create user with global privileges

CREATE USER test ENCRYPTED PASSWORD 'password';
GRANT CONNECT ON SCHEMA "your-schema" TO test;
GRANT USAGE ON SCHEMA "your-schema" TO test;
GRANT CREATE ON SCHEMA "your-schema" TO test;
GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA "your-schema" TO test;
GRANT SELECT ON ALL SEQUENCES IN SCHEMA "your-schema" TO test;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA your-schema TO test;

Per table

GRANT ALL PRIVILEGES ON TABLE your-table TO test;

Privileges Description SELECT Ability to execute SELECT operators for a table. INSERT Ability to execute INSERT operators for a table. UPDATE Ability to execute UPDATE operators for a table. DELETE Ability to execute DELETE operators for a table. TRUNCATE Ability to execute TRUNCATE operators on a table. REFERENCES Ability to create external keys (requires privileges for both parent and child tables). TRIGGER Ability to create triggers for a table. CREATE Ability to perform CREATE TABLE operators. ALL Provides all privileges.

MongoDB

Official documentation https://docs.mongodb.com

Redis

Official documentation https://redis.io/documentation

Cassandra

Official documentation http://cassandra.apache.org/doc/latest/

CouchDB

Official Documentation http://docs.couchdb.org/en/stable/

Release Management

GIT Branching Strategies

GIT Flow

The best guide to GIT Flow (in my opinion of course) https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow

Tools

Commitzen CLI https://github.com/commitizen/cz-cli

Software Versioning

Semantic versioning guide https://semver.org/ NPM Semantic versioning guide https://docs.npmjs.com/about-semantic-versioning

Automated versioning

https://www.npmjs.com/package/semantic-release - Automate how your NPM package.json file versions are incremented when releasing a package.

Coding Standards

  • Google HTML and CSS Code Standards https://google.github.io/styleguide/htmlcssguide.htmlRDFa is an extension to HTML5 that helps you markup things like People, Places, Events, Recipes and Reviews. Search Engines and Web Services use this markup to generate better search listings and give you better visibility on the Web, so that people can find your website more easily.

Scaling applications for Enterprise use and large traffic

An Introduction to Software Scaling

You may face dozens of different issues when it comes to scaling. A few general sources of your problems may be related to:

  • Limited physical resources like memory, CPUs, etc.,
  • Wrong memory management
  • Inefficient database engine
  • The complicated database schema, bad indexing
  • Poorly performed database queries
  • Wrong server configuration
  • App server limitations
  • Overall spaghetti code
  • Inefficient caching
  • Lack of monitoring tools
  • Too many external dependencies
  • Improper background jobs design

Horizontal and Vertical Scaling

https://www.scalyr.com/blog/horizontal-scalability-software/

Server clustering

https://www.ibm.com/developerworks/aix/tutorials/clustering/clustering.html

Load Balancers

https://www.ibm.com/cloud/learn/load-balancing

Tools

Docker

Docker Swarm

https://docs.docker.com/engine/swarm/ - Docker Swarm Documentation

Docker Tools

Software Logging

TBW

Client / Browser Logging

TBW

Server Application Logging

TBW

Hardware Logging

TBW

Software Monitoring

TBW

Tools

TBW

Software Message Queuing

TBW

Tools

RabbitMQ

TBW

ActiveMQ

TBW

IT Law

PCI ( Payment Card Industry) Compliance https://www.pcicomplianceguide.org/faq/

Software Licenses

Software license guide https://choosealicense.com/licenses/

Team Management and Approaches

Agile Development

Scrum

Scrum.org Guide https://www.scrumguides.org/scrum-guide.html Scrum at Scale Guide https://www.scrumatscale.com/scrum-at-scale-guide-read-online/

Product Owner Role https://www.mountaingoatsoftware.com/agile/scrum/roles/product-owner Scrum Master Role https://www.mountaingoatsoftware.com/agile/scrum/roles/scrummaster Scrum Team Role https://www.mountaingoatsoftware.com/agile/scrum/roles/team

Kanban and LEAN Development

Principles of LEAN Development https://leankit.com/learn/lean/principles-of-lean-development/

Books

The LEAN Startup by Eric Ries http://theleanstartup.com/

https://www.blueprintsys.com/agile-development-101/agile-methodologies - A guide to Agile Methodologies

XP (Extreme Programming)

https://www.techopedia.com/definition/3791/extreme-programming-xp

Tools

ODATA (Open Data Protocol) API Development

OData Standard https://www.odata.org/documentation/

REST (Representational State Transfer) API Development

Funnily enough the Wikipedia page for REST provides a great overview https://en.wikipedia.org/wiki/Representational_state_transfer

REST API Design Best Practices

Standard HTTP Methods https://cloud.google.com/apis/design/standard_methods Resource oriented design https://cloud.google.com/apis/design/resources

Authentication

JSON Web Tokens (JWT) for Authentication

JSON Web Token (JWT) is an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object. This information can be verified and trusted because it is digitally signed. JWTs can be signed using a secret (with the HMAC algorithm) or a public/private key pair using RSA or ECDSA.

Cross-site Scripting (XSS) attacks: Storing tokens in either local or session storage can lead to XSS attacks. Because of this, it’s a good idea to store tokens in a cookie with httpOnly and secure flags.

https://jwt.io/

OAuth

OAuth 2.0 is the industry-standard protocol for authorization. OAuth 2.0 supersedes the work done on the original OAuth protocol created in 2006. OAuth 2.0 focuses on client developer simplicity while providing specific authorization flows for web applications, desktop applications, mobile phones, and living room devices. This specification and its extensions are being developed within the IETF OAuth Working Group.

https://oauth.net/2/

HATEOAS (Hypermedia as the Engine of Application State)

https://spring.io/understanding/HATEOAS

Documentation Tools

Swagger (Self documenting your API) https://swagger.io/

JAVA Development

JAVA Spring Framework https://spring.io/ Pivotal Cloud Foundry (Manage your project, develop, builkd test deploy) https://pivotal.io/platform , https://pivotal.io/learn

PHP Development

Awesome PHP https://github.com/ziadoz/awesome-php (Comprehensive .NET Development Bookmarks)

Official documentation http://php.net/manual/en/ Composer (The best PHP package manager for dependency management) https://getcomposer.org/ Laravel (One of the best PHP Frameworks) - https://laravel.com/ Eloquent ORM - https://laravel.com/docs/8.x/eloquent Artisan console https://laravel.com/docs/8.x/artisan Symfony (Also one of the best PHP Frameworks) - https://symfony.com/ Silex - Lightweight PHP Framework https://silex.symfony.com/ Wordpress Codex - Build your applications on top of wordpress with custom themes, plugins etc - https://codex.wordpress.org/

Ruby on rails development

Official documentation https://guides.rubyonrails.org/

C# Development

Awesome .NET https://github.com/quozd/awesome-dotnet (Comprehensive .NET Development Bookmarks)

NuGet package manager https://www.nuget.org/ .NET Entity Framework https://docs.microsoft.com/en-us/ef/ Azure devops (Will succeed Nuget private servers) https://azure.microsoft.com/en-gb/services/devops/

Blockchain Development

The blockchain is now used in most industries to improve trust and transparency as well as provide an immutable record of distributed data.

Many people have heard of bitcoin but distributed ledger technology provides potentially more valuable technologies.

  • Peer to peer cash (Everyone becomes their own bank)
  • Programmable money and universal cash without government inflation.
  • Automated brokering
  • Removal of third party intermediaries
  • Distributed organisations with no central control
  • Truly consensus based platform
  • Uncensorable data

Videos

How blockchain is alredy taking over (Coldfusion) https://www.youtube.com/watch?v=kP6EezXJKNM How big is Bitcoin? https://www.youtube.com/watch?v=zrY3i85W5tU

Bitcoin

Bitcoin whitepaper https://bitcoin.org/bitcoin.pdf

EOS

EOS Developers Portal https://developers.eos.io/

Ethereum

Web3 API https://github.com/ethereum/wiki/wiki/JavaScript-API

Popular Ethereum Tools and Libraries

Truffle https://github.com/trufflesuite/truffle Ganache https://truffleframework.com/ganache

Interesting Blockchain Projects

https://storj.io/ - Distributed data storage with clients earning tokens for the storage they provide.

Virtual Reality

Virtual reality on the web

https://aframe.io/ - Aframe WebVR Specification https://webvr.info/ WebVR Experiments https://experiments.withgoogle.com/collection/webvr

How to Close a Port in Windows

To close a port in Windows, you need to find the process ID of the application or service that opened the connection. Then, you can terminate the process or configure the application or service.

netstat -ano | find ":3306" TCP [::]:3306 [::]:0 LISTENING 33208

Use the process ID from the previous step to identify the application or service that owns that process.

tasklist /F /PID 33208 Image Name PID Services mysqld.exe 33208 N/A

If the application is not needed, use the process ID from the second step to terminate the process.

taskkill /F /PID 33208 SUCCESS: The process with PID 33208 has been terminated.

verify that the process is killed.

netstat -ano | find ":3306"

.NET

TBW

Unity

TBW

Python

https://github.com/vinta/awesome-python - A curated list of awesome Python frameworks, libraries, software and resources.

Pip - (The Python Package Installer)

https://pypi.org/project/pip/

User Guide https://pip.pypa.io/en/stable/user_guide/

DJANGO (A Python Framework)

https://docs.djangoproject.com/en/3.0/

Flask (A Python Framework - My preference)

https://flask.palletsprojects.com/en/1.1.x/

SQL Alchemy (Python ORM, Migration and Database Management Module)

https://www.sqlalchemy.org/

Other useful modules and packages

https://github.com/davedoesdev/python-jwt - Generate JSON Web Tokens for Authentication https://github.com/jazzband/django-oauth-toolkit - Handle authentication using OAuth 2

Useful CLI Commands

Windows

Check 32 or 64 bit using WMIC

You can get to know the OS architecture by running the below simple command.

wmic os get OSArchitecture

Get the windows installation date

systeminfo | findstr /C:"Install Date"

Kill a process by name

Stop-Process -Name ApplicationName
Stop-process -Name Chrome

Kill a process by ID

Stop-process -Id processId

Remove user account password from command prompt

wmic useraccount where name='loginId' set PasswordRequired=false

Remove user account password from all accounts

wmic useraccount set PasswordRequired=false

Re-enabling passwords for user accounts

wmic useraccount where name='test1' set PasswordRequired=true

Disable LAN Connection

netsh interface set interface name="Local Area Connection" admin=DISABLED

Change the CLI color

Do you want to change the look of Windows command prompt by changing background color? Windows command prompt has default color settings of black background and white foreground(text). This post explains you how to to change background/foreground colors using ‘color’ command.

‘Color’ command uses below codes to represent various colors.

 0 = Black       8 = Gray
    1 = Blue        9 = Light Blue
    2 = Green       A = Light Green
    3 = Aqua        B = Light Aqua
    4 = Red         C = Light Red
    5 = Purple      D = Light Purple
    6 = Yellow      E = Light Yellow
    7 = White       F = Bright White

Once you decide on which colors to use, just input the codes to the command. For example, if you want to use white background with Light blue for text, you should run the below command.

color 79

Check your Wifi connection

If you have a WiFi connection, how do you find out if your computer is connected to WiFi or not? The command netsh interface is used to find your WiFi connection status from command prompt.

Execute the below command to know WiFi connection status

netsh interface show interface | findstr /C:"Wireless" /C:"Name"

Example on my Windows 7 computer

C:\>netsh interface show interface | findstr /C:"Wireless" /C:"Name"
Admin State    State          Type             Interface Name
Enabled        Connected      Dedicated        Wireless Network Connection

As you can see, the command shows that Wifi connection is connected when we ran the command.

If the computer has WiFi enabled, but not connected to any network, then the command output would be like below.

C:\>netsh interface show interface | findstr /C:"Wireless" /C:"Name"
Admin State    State          Type             Interface Name
Enabled        Disconnected   Dedicated        Wireless Network Connection

If the Wifi is disabled, then the output would be

C:\>netsh interface show interface | findstr /C:"Wireless" /C:"Name"
Admin State    State          Type             Interface Name
Disabled       Disconnected   Dedicated        Wireless Network Connection

This command helps to know the status manually. However if we want to know the status in a script, we would need to refine the command further.

About

📑 Some of my own notes and bookmarks for software development

Resources

Readme
Activity

Stars

6 stars

Watchers

3 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published