Skip to content

Commit

Permalink
Fixed typos and and clarified awk/gawk sections
Browse files Browse the repository at this point in the history
  • Loading branch information
jhajek committed Mar 25, 2020
1 parent 515fbb0 commit 309b3c1
Show file tree
Hide file tree
Showing 8 changed files with 22 additions and 17 deletions.
2 changes: 1 addition & 1 deletion Chapter-05/chapter-05.md
Expand Up @@ -430,7 +430,7 @@ The improvements offered by bash over sh, csh, and ksh include:

#### Open Source sh Replacements

Unlike Bash which provided a *"free"* ```sh``` compatible shell for the GNU system, there was mcuh code written in ```sh``` but no opensource version available. In 1989 Kenneth Almquist released the Almquist Shell (ash) which was an updated and opensourc version of the Bourne shell. In 1995/96 Debian maintainers released another improved version of Ash Shell called the Debian Ash Shell (dash) and this is the standard replacement for the traditional ```/bin/sh```.
Unlike Bash which provided a *"free"* ```sh``` compatible shell for the GNU system, there was much code written in ```sh``` but no opensource version available. In 1989 Kenneth Almquist released the Almquist Shell (ash) which was an updated and opensourc version of the Bourne shell. In 1995/96 Debian maintainers released another improved version of Ash Shell called the Debian Ash Shell (dash) and this is the standard replacement for the traditional ```/bin/sh```.

## File Permissions and Ownership

Expand Down
2 changes: 1 addition & 1 deletion Chapter-07/chapter-07.md
Expand Up @@ -66,7 +66,7 @@ In 1991, Bram Moolenaar created a port of __vi__ called __vim__ *vi improved*. V

### vi has a Sharp Learning Curve

Many people will say that the __vi editor__ has a sharp learning curve and not to use it. I believe that is a spurious argument. The __vi editor__ is not a text editor comparable notepad, but vi was a specific tool developed to create complex interaction with text in as few key strokes as possible. Learning to play the guitar is difficult in the beginning but once your have the muscle memory to do it you can become an expert player that can make beautiful music that few others can. The power of __vi editor__ is in the ability to do line editing and visual editing all from the __vi editor__, the ability to search and find, execute internal commands, even use grep and regex for complex pattern matching and replacement from within vi. Keeping your fingers on the keyboard constantly moving keeps your fingers and mind occupied. Nothing takes more time then to change "contexts". Don't abandon it because it is hard! You will eventually be working on systems that have no GUI at all: FreeBSD or Ubuntu Server or RHEL or CentOS you will have to use vi.
Many people will say that the __vi editor__ has a sharp learning curve and not to use it. I believe that is a spurious argument. The __vi editor__ is not a text editor comparable to notepad, but vi was a specific tool developed to create complex interaction with text in as few key strokes as possible. Learning to play the guitar is difficult in the beginning but once your have the muscle memory to do it you can become an expert player that can make beautiful music that few others can. The power of __vi editor__ is in the ability to do line editing and visual editing all from the __vi editor__, the ability to search and find, execute internal commands, even use grep and regex for complex pattern matching and replacement from within vi. Keeping your fingers on the keyboard constantly moving keeps your fingers and mind occupied. Nothing takes more time than to change "contexts". Don't abandon it because it is hard! You will eventually be working on systems that have no GUI at all: FreeBSD or Ubuntu Server or RHEL or CentOS you will have to use vi.

#### vi and vim

Expand Down
16 changes: 8 additions & 8 deletions Chapter-08/chapter-08.md
Expand Up @@ -4,7 +4,7 @@

## Objectives

This portion of the book begins Part II. The first 7 chapters focused more on the *philosophy* and basic tenants of how Unix and Linux were created. The remaining 8 chapters now will focus on the application of what we have learned and focus on using the opensource technology of Linux. The Objectives of this chapter are as follows:
This portion of the book begins Part II. The first 7 chapters focused more on the *philosophy* and basic tenets of how Unix and Linux were created. The remaining 8 chapters now will focus on the application of what we have learned and focus on using the opensource technology of Linux. The Objectives of this chapter are as follows:

* Understand how to write and use basic shell scripts
* Understand how to use conditional statements in Bash scripts
Expand Down Expand Up @@ -480,11 +480,11 @@ Entry Description Equiva
* Bash Cookbook [http://shop.oreilly.com/product/0636920058304.do](http://shop.oreilly.com/product/0636920058304.do "Bash Cookbook")
* vi and vim [http://shop.oreilly.com/product/9780596529833.do](http://shop.oreilly.com/product/9780596529833.do "vi and vim")
## Awk and Sed
## Text Processing with Awk and Sed
### awk and gawk
### AWK
AWK is a programming language designed for text processing and typically used as a data extraction and reporting tool. It is a standard feature of most Unix-like operating systems[^91]. This original program was released in 1977, and was a powerful and focused lanugage. We take things like Perl, Python, and even databases such as MySQL and SQLite. The ```awk``` language was designed to do all these things because at the time text based files was the universal datatype. The initial program was developed by [Alfred Aho](https://en.wikipedia.org/wiki/Alfred_Aho "Alfred Aho"), [Peter Weinberger](https://en.wikipedia.org/wiki/Peter_J._Weinberger "peter J. Weinberger"), and [Brian Kerhnighan](https://en.wikipedia.org/wiki/Brian_Kernighan "Brian Kernighan") at Bell Labs.
AWK is a programming language designed for text processing and typically used as a data extraction and reporting tool. It is a standard feature of most Unix-like operating systems[^91]. This original program was released in 1977, and was a powerful and focused language. We take things like Perl, Python, and even databases such as MySQL and SQLite. The ```awk``` language was designed to do all these things because at the time text based files was the universal datatype. The initial program was developed by [Alfred Aho](https://en.wikipedia.org/wiki/Alfred_Aho "Alfred Aho"), [Peter Weinberger](https://en.wikipedia.org/wiki/Peter_J._Weinberger "Peter J. Weinberger"), and [Brian Kerhnighan](https://en.wikipedia.org/wiki/Brian_Kernighan "Brian Kernighan") at Bell Labs.
An AWK program consists of a sequence of optional directives, pattern-action statements, and optional function definitions.
Expand Down Expand Up @@ -517,13 +517,13 @@ sudo journalctl --no-pager | awk '$5 ~ /^sshd/'
Lets take a look at this statement:[^90]
* \$5 tells awk to look at the fifth "column".
* \~ tells awk to do a RegularExpression match \/....\/ is a Regular expression.
* \$5 tells AWK to look at the fifth "column".
* \~ tells AWK to do a RegularExpression match \/....\/ is a Regular expression.
* Within the RE is the string Linux and the special character \^.
* \^ causes the RE to match from the start (as opposed to matching anywhere in the line).
* Seen together: awk will match a regular expression with "Linux" at the start of the first column.
* Seen together: AWK will match a regular expression with "Linux" at the start of the first column.
You can find more information at this IBM tutorial: [Awk by example](https://www.ibm.com/developerworks/library/l-awk1/index.html "Awk by example")
You can find more information at this IBM tutorial: [AWK by example](https://www.ibm.com/developerworks/library/l-awk1/index.html "AWK by example"). The GNU version of AWK is called [gawk](http://www.gnu.org/software/gawk/ "GNU webpage for gawk") and it performs the same commands and options but is licensed under GPLv3+ as AWK was part of the proprietary UNIX tools and license. Linux systems symlink `awk` command to `gawk`.
### sed
Expand Down
6 changes: 3 additions & 3 deletions Chapter-09/chapter-09.md
Expand Up @@ -117,7 +117,7 @@ One of the most central functions of an operating system is logging. Without lo

### /var/log/\*

The default logging directory on all Linux systems is located in ```/var/log```. This is the place where the kernel, the operating system, and any default installed services will place its logs. For 30+ years this was the convention and all common knowledge. But with the recent adoption of systemd on all major Linux platforms, the logging facility that was once simple text, has now been moved into the ```journald``` and into a binary format. Note with the systemd, the logging convention has been changed to a binary format and placed under the ```journalctl``` command which we will cover in chapter 11. But for Fedora 28+ and any Ubuntu or Debian distro.
The default logging directory on all Linux systems is located in ```/var/log```. This is the place where the kernel, the operating system, and any default installed services will place its logs. For 30+ years this was the convention and all common knowledge. But with the recent adoption of systemd on all major Linux platforms, the logging facility that was once simple text, has now been moved into the ```journald``` and into a binary format. Note with the systemd, the logging convention has been changed to a binary format and placed under the ```journalctl``` command which we will cover in chapter 11.

When you install additional packages, those packages too will add a directory for its own logs. Note in the picture below there is a log called ```httpd``` that is created when you install the https (apache2 webserver package) to track the webserver error.log and access.log files. You will notice in these screenshots that there is a log entry for VBoxGuestAdditions--telling you that you are using VirtualBox.

Expand Down Expand Up @@ -503,9 +503,9 @@ One of the tools to alliviate this is called [fail2ban](https://www.fail2ban.org

#### OpenSSL

OpenSSL is an Opensource Library used for crytographic key generation by OpenSSH. In 2016, it suffered an exploit due to the quality of the library maintaining older code from non-existant systems as well as being woefully underfunded and understaffed. Take note that although Google built its entire business using opensource and OpenSSL, they contributred almost nothing to its development. After the exploit a huge infusion of cash and adoption by the Linux Foundation of this project as a core infrastructure project has increased the quality of its security and development.
OpenSSL is an Opensource Library used for crytographic key generation by OpenSSH. In 2016, it suffered an exploit due to the quality of the library maintaining older code from non-existent systems as well as being woefully underfunded and understaffed. Take note that although Google built its entire business using opensource and OpenSSL, they contributred almost nothing to its development. After the exploit a huge infusion of cash and adoption by the Linux Foundation of this project as a core infrastructure project has increased the quality of its security and development.

The heartbleed OpenSSL bug even has its own website to explain the deatils of it, located at [http://heartbleed.com](http://heartbleed.com "Heartbleed.com").
The heartbleed OpenSSL bug even has its own website to explain the details of it, located at [http://heartbleed.com](http://heartbleed.com "Heartbleed.com").

> "The Heartbleed Bug is a serious vulnerability in the popular OpenSSL cryptographic software library. This weakness allows stealing the information protected, under normal conditions, by the SSL/TLS encryption used to secure the Internet. SSL/TLS provides communication security and privacy over the Internet for applications such as web, email, instant messaging (IM) and some virtual private networks (VPNs)[^ch9f106]."
Expand Down
2 changes: 1 addition & 1 deletion Chapter-10/chapter-10.md
Expand Up @@ -195,7 +195,7 @@ Compared to the make up of a SysVinit service, systemd has a simple design for e

![*httpd.service*](images/Chapter-10/systemd/httpd-service.png "service")

You can see the dependencies the httpd service needs and will only start if the targets listed in \[Unit\] under *After* have started--which makes sense. The command ```sudo systemctl show --property "Wants" httpd.service``` will show only the property from the unit file that you identify with __--property__. The *ExecStart* and *ExecReload* are called when you start and restart the service--which point to the absolute path to the Apache2 binary. The final value is when the service is installed it has a dependency of being in a mutil-user target--it obviously be used on a single user system.
You can see the dependencies the httpd service needs and will only start if the targets listed in \[Unit\] under *After* have started--which makes sense. The command ```sudo systemctl show --property "Wants" httpd.service``` will show only the property from the unit file that you identify with __--property__. The *ExecStart* and *ExecReload* are called when you start and restart the service--which point to the absolute path to the Apache2 binary. The final value is when the service is installed it has a dependency of being in a mutli-user target--it obviously be used on a single user system.

#### Major systemd Components

Expand Down
2 changes: 1 addition & 1 deletion Chapter-11/chapter-11.md
Expand Up @@ -24,7 +24,7 @@ The original, cheapest, and densest storage type in dollars per megabit is still

![*Hard Drive*](images/Chapter-11/hdd/256px-Laptop-hard-drive-exposed.jpg "Hard Drive Internals")

These are made up of spinning platters where bits are stored via magnetic charge. These systems have down sides in that parts of the surface ware out over time as well as they have mechanical parts (servos) that can fail over time. They require constant amounts of power and if you scale this over large data centers, these costs can quickly add up. In addition as the size of storage density has increased, the need to develop new storage mediums has arisen. What was once a single magnetic platter, became three parallel platters, then became five spinning platters, which then became glass platters, which then became vertical electron based storage (somehow they got electrons to stand up instead of lay flat), then moving to the latest [helium filled 10 TB disks](https://www.anandtech.com/show/9955/seagate-unveils-10-tb-heliumfilled-hard-disk-drive "Seagate 10TB helium filled disks").
These are made up of spinning platters where bits are stored via magnetic charge. These systems have down sides in that parts of the surface wear out over time as well as they have mechanical parts (servos) that can fail over time. They require constant amounts of power and if you scale this over large data centers, these costs can quickly add up. In addition as the size of storage density has increased, the need to develop new storage mediums has arisen. What was once a single magnetic platter, became three parallel platters, then became five spinning platters, which then became glass platters, which then became vertical electron based storage (somehow they got electrons to stand up instead of lay flat), then moving to the latest [helium filled 10 TB disks](https://www.anandtech.com/show/9955/seagate-unveils-10-tb-heliumfilled-hard-disk-drive "Seagate 10TB helium filled disks").

As disks became bigger, new patterns for reading and writing data were created: SATA 1,2, and 3.x bus technology was introduced for transmitting data faster from the disk to the CPU. Also on-disk based cache memory was added which was essentially 16-64mb disk on the disk to cache frequently accessed data. On top of that we have [NCQ](https://sata-io.org/developers/sata-ecosystem/native-command-queuing "Native Command Queueing"). Native Command Queueing was introduces which looks at disk requests on the drive and reorders them to reduce round trips that a disk needs to make. All of this is happening at 5400-7200 rpms, revolutions per minute, at a fly height of 8 Pico meters.

Expand Down
7 changes: 6 additions & 1 deletion Chapter-12/chapter-12.md
Expand Up @@ -465,7 +465,12 @@ create table tutorials_tbl(
quit;
```

You can automate connections to MySQL by creating a file called `~/.my.cnf`. In this file you can create user options that will override default connections, including username and password. This increases security by not having to manually type a password on the commandline.
You can automate connections to MySQL by creating a file called `my.cnf`. In this file you can create user options that will override default connections, including username and password. This increases security by not having to manually type a password on the commandline. MySQL and MariaDB will look for a `my.cnf` file in four default locations in cascading order. Some of these locations are owned by root and require permission. We will use the last location, `~/.my.cnf` which stored in your home directory and requires no root access.

* `/etc/my.cnf`
* `/etc/mysql/my.cnf`
* `/usr/etc/my.cnf`
* `~/.my.cnf`

```bash

Expand Down
2 changes: 1 addition & 1 deletion Chapter-15/chapter-15.md
Expand Up @@ -22,7 +22,7 @@ At the conclusion of this chapter you will have a detailed overview of Service D

As applications move from single systems to distributed systems, then to multi-tiers, the issue of IP addresses comes into focus. What happens when you have services being launch and virtual systems being destroyed? How do you manage the IP addresses? Well on the internet we have DNS for name resolution and lookup. If you look up [https://youtube.com](https://youtube.com "YouTube.com") you do not need to know the IP address, as the DNS service will look it up for you. Now, think about the internals of YouTube itself. Extra servers need to come online all the time to handle surges in requests or QPS (queries per second). How do they handle IP allocation internally? They use some kind of **service discovery** or **service mesh**.

The concept of service discovery is essentially an internal or local DNS. Each system hard codes its own localized DNS entry then via a protocol called *gossip*, agents talk to each other and "spread" the local DNS entries. An application can simpley make a ```curl``` or http based request to the internal agent and resolve the IP internally.
The concept of service discovery is essentially an internal or local DNS. Each system hard codes its own localized DNS entry then via a protocol called *gossip*, agents talk to each other and "spread" the local DNS entries. An application can simply make a ```curl``` or http based request to the internal agent and resolve the IP internally.

HashiCorp created a software package called [Consul](https://www.consul.io "Consul") to do just this. With datacenters, Virtual Machines, even OS Containers able to be spun up and down with the touch of a button, manual configuration of servers is no longer an option.

Expand Down

0 comments on commit 309b3c1

Please sign in to comment.