03-file-management.Rmd

# File Management {#command-line-file-management}

In this chapter, we will explore commands for file management including:

- create new file/change timestamps
- copying files
- renaming/moving files
- deleting files
- comparing files

```{r table_file_manage, echo=FALSE}
cname   <- c("`touch`", "`cp`", "`mv`", "`rm`", "`diff`")
descrip <- c("Create empty file(s)/change timestamp",
             "Copy files & folders",
             "Rename/move file", 
             "Remove/delete file",
             "Compare files")
data.frame(Command = cname, Description = descrip) %>% 
  kable() %>% 
  kable_styling(
    bootstrap_options = c("striped", "hover", "condensed", "responsive")
  )
```

## Create new file

`touch` modifies file timestamps which is information associated with file modification. It can be any of the following:

- access time (the last time the file was read)
- modification time (the last time the contents of the file was changed)
- change time (the last time the file's metadata was changed)

If the file does not exist, it will create an empty file of the same name. Let us use `touch` to create a new file `myanalysis.R`. 

```{bash c17a, eval=FALSE}
touch myanalysis.R
ls
```

```{bash c17b, echo=FALSE}
cd cline
touch myanalysis.R
ls
```

## Copy Files/Folders

`cp` makes copies of files and directories. The general form of the command is 
**cp source destination**. By default, it will overwrite files without prompting for confirmation so be cautious while copying files or folders.

### Copy files in same folder

Let us create a copy of `release_names.txt` file and name it as `release_names_2.txt`.

```{bash c19a, eval=FALSE}
cp release_names.txt release_names_2.txt
ls
```

```{bash c19b, echo=FALSE}
cd cline
cp release_names.txt release_names_2.txt
ls
```

### Copy files into different folder

To copy a file into a different directory/folder, we need to specify the name of the destination folder. If the copied file should have a different name, then we need to specify the new name of the file as well. Let us copy the `release_names.txt` file into the `r_releases` folder (we will retain the same name for the file as we are copying it into a different folder).

```{bash c21a, eval=FALSE}
cp release_names.txt r_releases/release_names.txt
```

```{bash c117a, echo=FALSE}
cd cline
mkdir r_releases
cp release_names.txt release_names_3.txt 
```

```{bash c21b, echo=FALSE}
cd cline
cp release_names.txt r_releases/release_names.txt
```

Let us check if the file has been copied by listing the files in the `r_releases` folder using `ls`.

```{bash c20a, eval=FALSE}
ls r_releases
```

```{bash c20b, echo=FALSE}
cd cline
ls r_releases
```

### Copy folders

How about making copies of folders? Use the `-r` option to copy entire folders. Let us create a copy of the `r` folder and name it as `r2`. The `-r` option stands for `--recursive` i.e. copy directories recursively. 

```{bash c22a, eval=FALSE}
cp -r r r2
ls
```

```{bash c22b, echo=FALSE}
cd cline
cp -r r r2
ls
```

## Move/Rename Files

`mv` moves and renames files and directories. Using different options, we can ensure

- files are not overwritten
- user is prompted for confirmation before overwriting files
- details of files being moved is displayed

```{r table_mv, echo=FALSE}
cname   <- c("`mv`", "`mv -f`", "`mv -i`", "`mv -n`", "`mv -v`")
descrip <- c("Move or rename files/directories",
             "Do not prompt for confirmation before overwriting files",
             "Prompt for confirmation before overwriting files", 
             "Do not overwrite existing files",
             "Move files in verbose mode")
data.frame(Command = cname, Description = descrip) %>% 
  kable() %>% 
  kable_styling(
    bootstrap_options = c("striped", "hover", "condensed", "responsive")
  )
```

Let us move the `release_names_2.txt` file to the `r_releases` folder.

```{bash c23a, eval=FALSE}
mv release_names_2.txt r_releases
```

```{bash c23b, echo=FALSE}
cd cline
mv release_names_2.txt r_releases
```

Use `ls` to verfiy if the file has been moved. As you can see, `release_names_2.txt` is not present in the current working directory.

```{bash c23c, eval=FALSE}
ls
```

```{bash c23d, echo=FALSE}
cd cline
ls
```

Let us check if `release_names_2.txt` is present in the `r_releases` folder. Great! We have successfully moved the file into a different folder.

```{bash c23e, eval=FALSE}
ls r_releases
```

```{bash c23f, echo=FALSE}
cd cline
ls r_releases
``` 

### Move files in verbose mode

To view the details of the files being moved/renamed, use the `-v` option. In the below example, we move the `release_names_3.txt` file into the `r_releases` folder using `mv`. 

```{bash c24a, eval =FALSE}
mv -v release_names_3.txt r_releases
```

```{bash c24b, echo=FALSE}
cd cline
mv -v release_names_3.txt r_releases
```

### Do not overwrite existing files

How do we ensure that files are not overwritten without prompting the user first? In the below example, we will try to overwrite the `release_names_2.txt` in the `r_releases` folder using `mv` and see what happens. But first, let us look at the contents of the `release_names_2.txt` file using the `cat` command.

We will look into the `cat` command in more detail in the next chapter but for the time being it is sufficient to know that it prints contents of a file. The file contains release names of different R versions.

```{bash c77a, eval =FALSE}
cat r_releases/release_names_2.txt
```

```{bash c77b, echo=FALSE}
cd cline
cat r_releases/release_names_2.txt
```

In our current working directory, we will create another file of the same name i.e. `release_names_2.txt` but its contents are different from the file in the `r_releases` folder. It contains the string `release_names` and nothing else. We will now move this file into the `r_releases` folder but use the option `-n` to ensure that the file in the `r_releases` folder is not overwritten. We can confirm this by printing the contents of the file in the `r_releases` folder.

The `echo` command is used to print text to the terminal or to write to a file. We will explore it in more detail in the next chapter.

```{bash c78a, eval =FALSE}
echo "release_names" > release_names_2.txt 
mv -n release_names_2.txt r_releases
cat r_releases/release_names_2.txt
```

```{bash c78b, echo=FALSE}
cd cline
echo "release_names" > release_names_2.txt 
mv -n release_names_2.txt r_releases
cat r_releases/release_names_2.txt
```

As you can observe, the contents of the file in the `r_releases` folder has not changed. In the next section, we will learn to overwrite the contents using the `-f` option.

### Do not prompt for confirmation before overwriting files

What if we actually intend to overwrite a file and do not want to be prompted for confirming the same. In this case, we can use the `-f` option which stands for `--force` i.e. do not prompt before overwriting. Let us first print the contents of the `release_names_2.txt` file in the `r_releases` folder.

```{bash c79a, eval =FALSE}
cat r_releases/release_names_2.txt
```

```{bash c79b, echo=FALSE}
cd cline
cat r_releases/release_names_2.txt
```

Now we will create another file of the same name in the current working directory but with different content and use the `-f` option to overwrite the file in the `r_releases` folder. You can see that the contents of the file in the `r_releases` folder has changed.

```{bash c80a, eval =FALSE}
echo "release_names" > release_names_2.txt 
mv -f release_names_2.txt r_releases
cat r_releases/release_names_2.txt
```

```{bash c80b, echo=FALSE}
cd cline
echo "release_names" > release_names_2.txt 
mv -f release_names_2.txt r_releases
cat r_releases/release_names_2.txt
```

## Remove/Delete Files

The `rm` command is used to delete/remove files & folders. Using additional options, we can

- remove directories & sub-directories
- forcibly remove directories
- interactively remove multiple files
- display information about files removed/deleted

```{r table_rm, echo=FALSE}
cname   <- c("`rm`", "`rm -r`", "`rm -rf`", "`rm -i`", "`rm -v`")
descrip <- c("Remove files/directories",
             "Recursively remove a directory & all its subdirectories",
             "Forcibly remove directory without prompting for confirmation or showing error messages",
             "Interactively remove multiple files, with a prompt before every removal",
             "Remove files in verbose mode, printing a message for each removed file")
data.frame(Command = cname, Description = descrip) %>% 
  kable() %>% 
  kable_styling(
    bootstrap_options = c("striped", "hover", "condensed", "responsive")
  )
```

### Remove files

Let us use `rm` to  remove the file `myanalysis.R` (we created it earlier using the `touch` command). 

```{bash c25a, eval=FALSE}
rm myanalysis.R
ls
```

```{bash c25b, echo=FALSE}
cd cline
rm myanalysis.R
ls
```

### Recursive Deletion

How about folders or directories? We can remove a directory and all its contents including sub-directories using the option `-r` which stands for `--recursive` and removes directories and their contents recursively. Let us remove the `myproject1` folder and all its contents.

```{bash c26a, eval=FALSE}
rm -r myproject1
ls
```

```{bash c26b, echo=FALSE}
cd cline
rm -r myproject1
ls
```

### Force Removal

Use the `-f` option which stands for `--force` to forciby remove directory and all its contents without prompting for confirmation or showing error messages. Let us remove the `myproject2` folder and all its contents.

```{bash c81a, eval=FALSE}
rm -rf myproject2
ls
```

```{bash c81b, echo=FALSE}
cd cline
rm -rf myproject2
ls
```

### Verbose Mode

Remove files in verbose mode, printing a message for each removed file. This is useful when you want to see the details of the files being removed. In the below example, we will remove all files with `.txt` extension from the `myfiles` folder. Instead of specifying the name of each text file, we use the wildcard `*` along with `.txt` i.e. any file with the extension `.txt` will be removed.

```{bash c27a, eval=FALSE}
cd myfiles
rm -v *.txt
```

```{bash c27b, echo=FALSE}
cd cline
cd myfiles
rm -v *.txt
```

## Compare Files

`diff` stands for difference. It is used to compare files line by line and display differences. It also indicates which lines in one file must be changed to make the files identical. Using additional options, we can

- ignore white spaces while comparing files
- show differences sidy by side
- show differences in unified format
- compare directories recursively
- display names of files that differ

```{r table_diff, echo=FALSE}
cname   <- c("`diff`", "`diff -w`", "`diff -y`", "`diff -u`", "`diff -r`", "`diff -rq`")
descrip <- c("Compare files & directories",
             "Compare files; ignoring white spaces",
             "Compare files; showing differences side by side", 
             "Compare files; show differences in unified format",
             "Compare directories recursively",
             "Compare directories; show the names of files that differ")
data.frame(Command = cname, Description = descrip) %>% 
  kable() %>% 
  kable_styling(
    bootstrap_options = c("striped", "hover", "condensed", "responsive")
  )
```

### Compare Files

Let us compare the contents of the following files

- `imports_olsrr.txt`
- `imports_blorr.txt`

The files contain the names of R packages imported by the [olsrr](https://olsrr.rsquaredacademy.com/) and [blorr](https://blorr.rsquaredacademy.com/) packages respectively (**Full disclosure: both the above R pakages are developed by Rsquared Academy.**).

`diff` uses certain special symbols and gives instructions to make the files identical. The instructions are on how to change the first file to make it identical to the second file. We list the symbols below

- **a** for add
- **c** for change
- **d** for delete

We will use the `-w` option to ignore white spaces while comparing the files.

```{bash c30, eval=FALSE}
diff -w imports_olsrr.txt imports_blorr.txt
```

```{r r30, echo=FALSE, eval=FALSE}
suppressWarnings(cat(system2('diff', c("-w", "imports_olsrr.txt", "imports_blorr.txt"), TRUE), sep = "\n"))
```

Let us interpret the results. `4a5` indicates **after line 4 in file 1, add line 5 from file 2** to make both the files identical i.e. add `caret` which is line 5 in `imports_blorr.txt` after line 4 in `imports_olsrr.txt` which will make both the files identical.

Let us change the file order and see the instructions from `diff`.

```{bash c30a, eval=FALSE}
diff -w imports_blorr.txt imports_olsrr.txt
```

```{r r30a, echo=FALSE, eval=FALSE}
suppressWarnings(cat(system2('diff', c("-w", "imports_blorr.txt", "imports_olsrr.txt"), TRUE), sep = "\n"))
```

`5d4` indicates **delete line 5 from file 1 to match both the files at line4** i.e. delete `caret` which is line 5 in `imports_blorr.txt` to make both the files identical.

### Side By Side

To view the differences between the files side by side, use the `-y` option.

```{bash c31, eval=FALSE}
diff -y imports_olsrr.txt imports_blorr.txt
```

```{r r31, echo=FALSE, eval=FALSE}
suppressWarnings(cat(system2('diff', c("-y", "imports_olsrr.txt", "imports_blorr.txt"), TRUE), sep = "\n"))
```

### Unified Format

To view the differences between the files in a unified format, use the `-u` option.

```{bash c32, eval=FALSE}
diff -u imports_olsrr.txt imports_blorr.txt
```

```{r r32, echo=FALSE, eval=FALSE}
suppressWarnings(cat(system2('diff', c("-u", "imports_olsrr.txt", "imports_blorr.txt"), TRUE), sep = "\n"))
```

### Compare Recursively

To compare recursively, use the `-r ` option. Let us compare the `mypackage` and `myproject` folders.

```{bash c82, eval=FALSE}
diff -r mypackage myproject
```

```{r r82, echo=FALSE, eval=FALSE}
suppressWarnings(cat(system2('diff', c("-r", "mypackage", "myproject"), TRUE), sep = "\n"))
```

### File Details

To compare directories and view the names of files that differ, use the `-rq` option. In the below example, we look at the names of files that differ in `mypackage` and `myproject` folders.

```{bash c83, eval=FALSE}
diff -rq mypackage myproject
```

```{r r83, echo=FALSE, eval=FALSE}
suppressWarnings(cat(system2('diff', c("-rq", "mypackage", "myproject"), TRUE), sep = "\n"))
```

## R Functions

In R, file operations can be performed using functions from both base R and the [fs](https://fs.r-lib.org/index.html) package.

```{r r_file_manage, echo=FALSE}
cname   <- c("`touch`", "`cp`", "`mv`", "`rm`", "`diff`")
descrip <- c("`file.create()` / `fs::file_create()` / `fs::file_touch()`",
             "`file.copy()` / `fs::file_copy()` / `fs::dir_copy()`",
             "`file.rename()` / `fs::file_move()`", 
             "`file.remove()` / `fs::file_delete()`",
             "")
data.frame(Command = cname, R = descrip) %>% 
  kable() %>% 
  kable_styling(
    bootstrap_options = c("striped", "hover", "condensed", "responsive")
  )
```