diff --git a/ci/vale/dictionary.txt b/ci/vale/dictionary.txt index 5cc972d90a7..f5d1abe0dcc 100644 --- a/ci/vale/dictionary.txt +++ b/ci/vale/dictionary.txt @@ -27,6 +27,7 @@ antispam ap apache2 api +apiserver appimage appletalk appserver @@ -177,12 +178,14 @@ configtest configurability containerised convolutional +cookstyle coose copyfrom copyto coreutils corporateclean cpanminus +cpu cqlsh cqlshrc craftbukkit @@ -215,6 +218,7 @@ daemonizes daemonizing daemonset dahdi +databags datacenter datadir dataset @@ -384,6 +388,7 @@ flamegraph flamegraphs flatpress flyspray +foodcritic fortran fpm fragging @@ -703,6 +708,7 @@ luminus lxml lzma lzopio +macos magento maildir maildirs @@ -744,6 +750,7 @@ mesos metabase metacity metapackage +metaparameters metricbeat microblogs microservice @@ -1006,6 +1013,7 @@ pty pubkey publickey puppetd +puppetlabs puppetmaster puttygen pv diff --git a/docs/applications/configuration-management/automatically-configure-servers-with-ansible-and-playbooks/index.md b/docs/applications/configuration-management/automatically-configure-servers-with-ansible-and-playbooks/index.md index 10b6741634d..15de4e41029 100644 --- a/docs/applications/configuration-management/automatically-configure-servers-with-ansible-and-playbooks/index.md +++ b/docs/applications/configuration-management/automatically-configure-servers-with-ansible-and-playbooks/index.md @@ -279,9 +279,9 @@ This section demonstrates using a playbook to automate basic server configuratio ### Install the Stack -Create a basic server setup with Apache, PHP, and a test MySQL database to use. +Create a basic server setup with NGINX, PHP, and a test MySQL database to use. -1. The following playbook downloads the appropriate packages, turns on the Apache and MySQL services, and creates a basic database and user: +1. The following playbook downloads the appropriate packages, turns on the nginx and MySQL services, and creates a basic database and user: {{< file "setup_webserver.yml" yaml >}} --- diff --git a/docs/applications/configuration-management/beginners-guide-chef/index.md b/docs/applications/configuration-management/beginners-guide-chef/index.md index 305d0581b18..3903232cf98 100644 --- a/docs/applications/configuration-management/beginners-guide-chef/index.md +++ b/docs/applications/configuration-management/beginners-guide-chef/index.md @@ -6,9 +6,9 @@ description: 'A look into Chef''s primary components, features, and configuratio keywords: ["chef", "automation", "chefdk", "chef server", "chef development kit", "cookbooks", "beginners", "server automation", "configuration management"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' aliases: ['applications/chef/beginners-guide-chef/'] -modified: 2015-06-10 +modified: 2019-01-11 modified_by: - name: Elle Krout + name: Linode published: 2015-06-10 title: A Beginner's Guide to Chef external_resources: @@ -18,44 +18,66 @@ external_resources: --- -*Chef* is an automation platform that "turns infrastructure into code," allowing organizations or persons with large frameworks to generate a process that will save time and effort when making changes to part or all of their server fleet. +*Chef* is a declarative configuration management and automation platform used to translate infrastructure into code. This methodology enables organizations or individuals to generate a process with better testing, efficient and predictable deployments, centralized versioning, and reproducible environments across all your servers. ![Chef for beginners](a_beginners_guide_to_chef_smg.jpg) -Chef works with three core components: The Chef server, workstations, and nodes. The Chef server is the hub of Chef operations, where changes are stored for use. Workstations are static computers or virtual servers where all code is created or changed. There can be as many workstations as needed, whether this be one per person or otherwise. Finally, nodes are the servers that need to be managed by Chef -- these are the machines that changes are being pushed to, generally a fleet of multiple machines that require the benefits of an automation program. +Chef works with three core components, the Chef server, workstations, and nodes: + +- **Chef server:** As the center of operations, the Chef server stores, manages, and provides configuration data to all other Chef components. +- **Workstations:** Workstations are personal computers or virtual servers where all configuration code is created, tested, and changed. There can be as many workstations as needed, whether this be one per person or otherwise. +- **Nodes:** Nodes are the servers that are managed by Chef -- these are the machines that changes are being pushed to, generally a fleet of multiple machines that require the benefits of automation. Chef can manage nodes that are virtual servers, containers, network devices, and storage devices. A Chef client is installed on every node that is under management by Chef. [![Chef Workflow](chef_graph-small.png)](chef_graph.png) -These three components communicate in a mostly-linear fashion, with any changes being pushed from workstations to the Chef server, and then pulled from the server to the nodes. In turn, information about the node passes to the server to determine which files are different from the current settings and need to be updated. +These three components communicate in a mostly linear fashion, with any changes being pushed from workstations to the Chef server, and then pulled from the server to the nodes and implemented on each node via their Chef client. In turn, information about the node passes to the server to determine which files are different from the current settings and need to be updated. -If you wish to farther explore Chef please see the guides [Setting Up a Chef Server, Workstation, and Node on Ubuntu 18.04](/docs/applications/configuration-management/install-a-chef-server-workstation-on-ubuntu-18-04/) and [Creating Your First Chef Cookbook](/docs/applications/configuration-management/creating-your-first-chef-cookbook/). +If you wish to further explore Chef please see the guides [Setting Up a Chef Server, Workstation, and Node on Ubuntu 18.04](/docs/applications/configuration-management/install-a-chef-server-workstation-on-ubuntu-18-04/) and [Creating Your First Chef Cookbook](/docs/applications/configuration-management/creating-your-first-chef-cookbook/). ## The Chef Server -The Chef server is the primary mode of communication between the workstations where your infrastructure is coded, and the nodes where it is deployed. All configuration files, cookbooks, metadata, and other information are stored on the server. The Chef server also keeps information regarding the state of all nodes at the time of the last [chef-client](#chef-client) run. +The Chef server provides a communication pathway between the workstations where your infrastructure is coded, and the nodes where the configurations are deployed by the Chef client. All configuration files, cookbooks, metadata, and other information are created on workstations and stored on the Chef server. The Chef server also keeps information regarding the state of all nodes at the time of the last [chef-client](#chef-client) run. A workstation communicates with the Chef server using [*Knife*](/docs/applications/configuration-management/beginners-guide-chef/#knife) and Chef command line tools, while nodes communicate with the Chef server using the [Chef client](/docs/applications/configuration-management/beginners-guide-chef/#chef-client). -Any changes made must pass through the Chef server to be deployed. Prior to accepting or pushing changes, it verifies that the nodes and workstations are paired with the server through the use of authorization keys, and then allows for communication between the workstations and nodes. +Any changes made to your infrastructure code must pass through the Chef server in order to be applied to nodes. Prior to accepting or pushing changes, the Chef server authenticates all communication via its REST API using public key encryption. -### Bookshelf +### Components of a Chef Server +The Chef Server itself is made up of several components that help it to efficiently communicate with workstations and nodes. Each Chef server has an NGINX front-end load balancer to route all requests to the Chef server API. PostgreSQL is used to store data. An Apache Solr instance, wrapped by chef-solr, is used for indexing and search. A web interface, known as Chef manage, is available for common Chef server management tasks. All these components contribute to the Chef server's ability to handle requests for several thousands of nodes and make Chef server a resource heavy application. For this reason, a Chef server's minimum system requirements are a Linode plan of at minimum 8 GB of RAM and 4 CPU cores. For more specifications, see the [Chef System Requirements](https://docs.chef.io/chef_system_requirements.html) documentation page. -The Bookshelf is a versioned repository where cookbooks are stored on the Chef server (generally located at `/var/opt/opscode/bookshelf`; full root access is needed). When a cookbook is uploaded to the Chef server, the new version is compared to the one already stored; if there are changes, a new version is stored. The Chef server only stores one copy of a file or template at once, meaning if resources are shared between cookbooks and cookbook versions, they will not be stored multiple times. +### Bookshelf +To store cookbooks and related files and templates, the Chef server uses a *Bookshelf* that operates as a versioned repository (generally located at `/var/opt/opscode/bookshelf`; full root access is needed). When a cookbook is uploaded to the Chef server, the new version of the cookbook is compared to the one already stored; if there are changes, a new version is stored. The Chef server only stores one copy of a file or template, meaning if resources are shared between cookbooks and cookbook versions, they will not be stored multiple times. ## Workstations -Workstations are where users create, test, and maintain cookbooks and policies that will be pushed to nodes. Cookbooks created on workstations can be used privately by one organization, or uploaded to the Chef Supermarket for others to use. Similarly, workstations can be used to download cookbooks created by other Chef users and found in the Supermarket. +Workstations are where users create, test, and maintain cookbooks and policies that will be pushed to the Chef server and pulled by nodes. The workstation functionality is available by downloading the [Chef Workstation package](https://downloads.chef.io/chef-workstation/), which was formerly known as ChefDK. The Chef Workstation provides chef and knife command line tools, the testing tools Test Kitchen, ChefSpec, Cookstyle, and Foodcritic, and [InSpec](https://www.chef.io/inspec/) a tool that allows you to write automated tests for compliance, security and policy requirements. Additionally, the dependency manager for Chef cookbooks, *Berkshelf*, is installed. + +The Chef Workstation package can be installed on virtual servers or on personal computers. Workstations are set to interact with a single Chef server, and most work will be done in the `chef-repo` directory located on the workstation. -Workstations are set up to use the *Chef Development Kit* (ChefDK), and can be located on virtual servers or on physical workstation computers. Workstations are set to interact with only one Chef server, and most work will be done in the `chef-repo` directory located on the workstation. +Cookbooks created on workstations can be used privately by one organization, or uploaded to the [Chef Supermarket](https://supermarket.chef.io/) for others to use. Similarly, workstations can be used to download cookbooks created by other Chef users and found in the Supermarket. + +For a deeper dive into the various tools provided by the Chef Workstation package, see Chef's [official documentation](https://docs.chef.io/about_chefdk.html). ### chef-repo -The `chef-repo` directory is the specific area of the workstation where cookbooks are authored and maintained. The `chef-repo` is always version-controlled, most often through the use of Git, and stores information and history that will be used on nodes, such as cookbooks, environments, roles, and data bags. Chef is able to communicate with the server from the `chef-repo` and push any changes via the use of the `knife` command, which is included in the ChefDK. +The `chef-repo` directory is the area of the workstation where cookbooks are authored and maintained and any supporting resources like roles, data bags, and environments are stored. The `chef-repo` should be version-controlled with a remote version control system, like Git. Chef is able to communicate with the server from the `chef-repo` and push any changes via the use of `knife` commands. -Originally the `chef-repo` had to be pulled from GitHub using git commands, but that action is now integrated into Chef through the use of the `chef generate repo chef-repo` command. +You can easily generate a Chef repository using the following command: `chef generate repo repo-name`. ### Knife -The `knife` command communicates between the `chef-repo` located on a workstation and the Chef server. `knife` is configured with the `knife.rb` file, and is used from the workstation: +The Knife command line tool is the primary way that a workstation communicates the contents of its `chef-repo` directory with a Chef server. It also provides an interface for you to manage nodes, cookbooks, roles, environments, and databags. + +- A Knife command executed from your workstation uses the following format: + + knife subcommand [ARGUMENT] (options) + +- For example, to view the details of a Chef user, execute the following command: + + knife user show USER_NAME + +Knife allows for a variety of other useful operations to be executed on the Chef server and nodes. View Chef's [Knife](https://docs.chef.io/knife.html) documentation for a full list of all available commands. + +The Knife command line tool is configured with the `knife.rb` file: {{< file "~/chef-repo/.chef/knife.rb" ruby >}} log_level :info @@ -84,10 +106,13 @@ The default `knife.rb` file is defined with the following properties: - **syntax_check_cache_path:** The location in which `knife` stores information about files that have been checked for appropriate Ruby syntax. - **cookbook_path:** The path to the cookbook directory. +### Test Kitchen + +Test Kitchen provides you with a development environment on your workstation to create, test, and iterate on your cookbooks before distributing its contents to your production nodes. You can use the Kitchen command line tool to run integration tests against different platforms allowing you to test for the variety of nodes running on your production infrastructure. Refer to the [Kitchen CI](https://kitchen.ci/docs/getting-started/introduction/) documentation for information on getting started with Test Kitchen. ## Nodes -A *node* is a system configured to run the chef-client. This can be any system, as long as it is being maintained by Chef. +A *node* is any machine that is managed by a Chef server. Chef can manage nodes that are virtual servers, containers, network devices, and storage devices. Each node must have a corresponding Chef client installed in order to execute the steps needed to bring the node into the required stated defined by a cookbook. Nodes are validated through the `validator.pem` and `client.pem` certificates that are created on the node when it is bootstrapped. All nodes must be bootstrapped over SSH as either the root user or a user with elevated privileges. @@ -95,24 +120,23 @@ Nodes are kept up-to-date through the use of the chef-client, which runs a conve ### chef-client -The chef-client checks the current configuration of the node against the recipes and policies stored in the Chef server and brings the node up to match. The process begins with the chef-client checking the node's [run list](#run-lists), loading the cookbooks required, then checking and syncing the cookbooks with the current configuration of the node. +The chef-client checks the current configuration of the node against the recipes and policies stored on the Chef server and brings the node up to date. The process begins with the chef-client checking the node's [run list](#run-lists), loading the required cookbooks, then checking and syncing the cookbooks with the current configuration of the node. The chef-client must be run with elevated privileges in order to properly configure the node, and should be run periodically to ensure that the server is always up to date -- often this is achieved through a cron job or by setting up the chef-client to run as a service. ### Run Lists -Run lists define what cookbooks a node will use. The run list is an ordered list of all cookbooks and recipes that the chef-client needs to pull from the Chef server to run on a node. Run lists are also used to define [roles](http://docs.chef.io/server_manage_roles.html), which are used to define patterns and attributes across nodes. +Run lists define which [recipes](/docs/applications/configuration-management/beginners-guide-chef/#recipes) a node will use. The run list is an ordered list of all [*roles*](http://docs.chef.io/server_manage_roles.html) and recipes that the chef-client needs to pull from the Chef server to run on a node. Roles are used to define patterns and attributes across nodes. ### Ohai -Ohai collects information regarding nodes for the Chef server. It is required to be present on every node, and is installed as part of the bootstrap process. - -The information gathered includes network and memory usage, CPU data, kernel data, hostnames, FQDNs, and other automatic attributes that need to remain unchanged during the chef-client run. +Ohai collects information regarding nodes for the Chef client. It is required to be present on every node, and is installed as part of the bootstrap process. +The information gathered includes network and memory usage, CPU data, kernel data, hostnames, FQDNs, and other automatic attributes that helps the Chef client determine the state of the node prior to applying that nodes run list. ## Environments -Chef environments exist to mimic real-life workflow, allowing for nodes to be organized into different "groups" that define the role the node plays in the fleet. This allows for users to combine environments and versioned cookbooks to have different attributes for different nodes. For example, if testing a shopping cart, you may not want to test any changes on the live website, but with a "development" set of nodes. +Chef environments exist to mimic real-life workflow, allowing for nodes to be organized into different "groups" that define the role the node plays in the fleet. This allows for users to combine environments and versioned cookbooks to have different attributes for different nodes. For example, if testing a shopping cart, you may not want to test any changes on the live website, but instead, with a "development" set of nodes. Environments are defined in `chef-repo/environments` and saved as Ruby or JSON files. @@ -156,33 +180,49 @@ All nodes are automatically set to the "default" environment upon bootstrap. To ## Cookbooks -Cookbooks are the main component of configuring nodes on a Chef infrastructure. Cookbooks contain values and information about the *desired state* of a node, not how to get to that desired state -- Chef does all the work for that, through their extensive libraries. +Cookbooks are the basis for managing the configurations on any node. Cookbooks contain values and information about the *desired state* of a node and then the Chef server and Chef client ensure that the defined state is achieved. -Cookbooks are comprised of recipes, metadata, attributes, resources, templates, libraries, and anything else that assists in creating a functioning system, with attributes and recipes being the two core parts of creating a cookbook. Components of a cookbook should be modular, keeping recipes small and related. +Cookbooks are comprised of recipes, metadata, attributes, resources, templates, libraries, and anything else that assists in creating a functioning system, with attributes and recipes being the two core parts of a cookbook. Components of a cookbook should be modular, keeping recipes small and related. -Cookbooks can and should be version controlled. Versions can help when using environments and allow for the easier tracking of changes that have been made to the cookbook. +Cookbooks should be version controlled. Versions can help when using different Chef environments and allow you to distribute and collaborate on cookbooks with other team members. ### Recipes -Recipes are the fundamental part of cookbooks. Recipes are written in Ruby and contain information in regards to everything that needs to be run, changed, or created on a node. Recipes work as a collection of *resources* that determine the configuration or policy of a node, with resources being a configuration element of the recipe. For a node to run a recipe, it must be on that node's run list. +A Chef cookbook is comprised of recipes that a nodes desired state. Recipes are written in Ruby and contain information about everything that needs to be run, changed, or created on a node. Recipes work as a collection of *resources* that determine the configuration or policy of a node, with resources being a configuration element of the recipe. For a node to run a recipe, it must be on that node's run list. + +The example recipe is part of Chef's [Vim cookbook](https://github.com/chef-cookbooks/vim). It is in charge of installing the required Vim package based on a node's Linux distribution: + +{{< file "~/chef-repo/cookbooks/vim/packages.rb" >}} +... + +vim_base_pkgs = value_for_platform_family( + %w(debian arch) => ['vim'], + %w(rhel fedora) => ['vim-minimal', 'vim-enhanced'], + 'default' => ['vim'] +) + +package vim_base_pkgs + +package node['vim']['extra_packages'] unless node['vim']['extra_packages'].empty? +{{}} ### Attributes -Attributes define specific values about a node and its configuration. These values are used to override default settings, and are loaded in the order cookbooks are listed in the run list. Often attributes are used in conjunction with templates and recipes to define settings. +Attributes define specific values about a node and its configuration and are used by the Chef client to apply those attributes to nodes via its attribute list. The chef client can receive attributes from nodes, attribute files, recipes, environments, and roles. Often attributes are used in conjunction with templates and recipes to define settings. To learn more about attributes, refer to [Chef's documentation](https://docs.chef.io/attributes.html). ### Files -These are static files that can be uploaded to nodes. Files can be configuration and set-up files, scripts, website files -- anything that does not been to have different values on different nodes. +These are static files that can be uploaded to nodes. Files can be configuration and set-up files, scripts, website files. For example, you may have a recipe that uses an `index.php` file. You can use a `cookbook_file` resource block within a recipe to create the file on a node. All static files should be stored in a cookbook's `files` directory. ### Libraries -Although Chef comes with a number of libraries built in, additional libraries can be defined. Libraries are what bring recipes to life: If a recipe is the *desired state* of a node, than added libraries contain the behind-the-scenes information Chef needs for the nodes to reach this state. Libraries are written in Ruby, and can also be used to expand on any functionalities that Chef already contains. +Although Chef comes with a number of libraries built in, additional libraries can be defined. Libraries allow you to write Ruby code to be included in a cookbook. Libraries are a convenient way to include helper code for your existing recipes. Libraries provide a powerful way to extend the resources created by your recipes. -### Providers and Resources +### Resources -Providers and resources are also used to define new functionality to use in Chef recipes. A *resource* defines a set of actions and attributes, whereas *provider* informs the chef-client how to commit each action. +Resources are written in Ruby and defined in recipe files. Resources must contain a type, a name, one or more properties, and one or more actions. Resources are the key components that make up any single recipe. To learn Chef's, visit [Chef's official documentation](https://docs.chef.io/resource.html). ### Templates -Templates are embedded Ruby files (.erb) that allows for content based on the node itself and other variables generated when the chef-client is run and the template is used to create or update a file. +Templates are embedded Ruby files (`.erb`) that are used to dynamically create static text files. To use a template within a cookbook, you must declare a template resource in a recipe and include a corresponding `.erb` template file in a `template` subdirectory. Your template resource can contain variables that will then be used by the template to dynamically provide those values based on a nodes particular context. diff --git a/docs/applications/configuration-management/creating-your-first-chef-cookbook/index.md b/docs/applications/configuration-management/creating-your-first-chef-cookbook/index.md index 1c54ef795af..37d6f39de72 100644 --- a/docs/applications/configuration-management/creating-your-first-chef-cookbook/index.md +++ b/docs/applications/configuration-management/creating-your-first-chef-cookbook/index.md @@ -3,31 +3,46 @@ author: name: Elle Krout email: ekrout@linode.com description: 'Learn how to create Chef cookbooks by creating a LAMP stack in Chef' -keywords: ["chef", "automation", "cookbooks", "opscode", "lamp", "lamp stack", "beginner", "server automation"] +keywords: ["chef", "automation", "cookbooks", "configuration management", "DevOps"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' aliases: ['applications/chef/creating-your-first-chef-cookbook/'] -modified: 2018-08-06 +modified: 2019-01-17 modified_by: name: Linode published: 2015-06-10 title: Creating Your First Chef Cookbook external_resources: - '[Chef](http://www.chef.io)' + - '[About Cookbooks](https://docs.chef.io/cookbooks.html)' + - '[About Knife](https://docs.chef.io/knife.html)' + - '[About Nodes](https://docs.chef.io/nodes.html)' --- -Cookbooks are one of the key components in Chef. They describe the *desired state* of your nodes, and allow Chef to push out the changes needed to achieve this state. Creating a cookbook can seem like an arduous task at first, given the sheer number of options provided and areas to configure, so in this guide we will walk through the creation of one of the first things people often learn to configure: A LAMP stack. +Chef cookbooks describe the *desired state* of your nodes, and allow Chef to push out the changes needed to achieve this state. In this guide you will learn how to create a cookbook that configures A LAMP stack on a Linode. ![Creating Your First Chef Cookbook](creating-your-first-chef-cookbook.png "Creating Your First Chef Cookbook") -Prior to using this guide, set up Chef with the [Setting Up a Chef Server, Workstation, and Node](/docs/applications/configuration-management/install-a-chef-server-workstation-on-ubuntu-18-04/) guide. When following that guide, **choose Ubuntu 16.04 as your Linux image for the Chef node**. This is required because the [MySQL Chef cookbook](https://supermarket.chef.io/cookbooks/mysql/) that will be used is not yet compatible with Ubuntu 18.04. +## Before You Begin -If needed, review the [Beginner's Guide to Chef](/docs/applications/configuration-management/beginners-guide-chef/). +1. Set up Chef with the [Setting Up a Chef Server, Workstation, and Node](/docs/applications/configuration-management/install-a-chef-server-workstation-on-ubuntu-18-04/) guide. When following that guide, **choose Ubuntu 16.04 as your Linux image for the Chef node you will bootstrap and manage**. This guide will use the [MySQL Chef cookbook](https://supermarket.chef.io/cookbooks/mysql/), which does not yet support Ubuntu 18.04. -The examples in this tutorial require a root user account. Readers who choose to use a limited user account will need to prefix commands with sudo where required when working on the Chef client node. If you have yet to create a limited user account, follow the steps in the [Securing Your Server](/docs/security/securing-your-server/#add-a-limited-user-account) guide. +1. Once your node is bootstrapped, you can use a Chef cookbook to secure your node. Consider using the [Users](https://supermarket.chef.io/cookbooks/users) cookbook and the [Firewall](https://supermarket.chef.io/cookbooks/firewall) cookbook for this work. While this is not required to complete this guide, it is recommended. + +1. You can also review [A Beginner's Guide to Chef](/docs/applications/configuration-management/beginners-guide-chef/)to receive an overview on Chef concepts. + +1. The examples in this tutorial require a user account with sudo privileges. Readers who use a limited user account will need to prefix commands with sudo when issuing commands to the Chef client node and replace `-x root` with `-x username` where `username` is your limited user account. + +1. Ensure that your workstation's `/etc/hosts` file contains its own IP address and hostname and the IP address and hostname for any nodes you will interact with from the workstation. For example: + + {{< file "/etc/hosts">}} + 127.0.0.1 localhost + 192.0.2.0 workstation + 198.51.100.0 node-hostname + {{}} ## Create the Cookbook -1. From your workstation, move to your `cookbooks` directory in `chef-repo`: +1. From your workstation, move to your `chef-repo/cookbooks` directory: cd chef-repo/cookbooks @@ -39,27 +54,25 @@ The examples in this tutorial require a root user account. Readers who choose to cd lamp_stack -1. List the files located in the newly-created cookbook to see that a number of directories and files have been created: - - ls + If you issue the `ls` command, you should see the following files and directories: - {{< output >}} -Berksfile CHANGELOG.md chefignore LICENSE metadata.rb README.md recipes spec test -{{}} + {{< output >}} + Berksfile CHANGELOG.md chefignore LICENSE metadata.rb README.md recipes spec test + {{}} - For more information about these directories see the [Beginner's Guide to Chef](/docs/applications/configuration-management/beginners-guide-chef/). +### default.rb -## default.rb +Attributes are pieces of data that help the chef-client determine the current state of a node and any changes that have taken place on the node from one chef-client run to another. Attributes are gathered from the state of the node, cookbooks, roles and environments. Using these sources, an attribute list is created for each chef-client run and is applied to the node. If a `default.rb` file exists within a cookbook, it will be loaded first, but has the lowest attribute precedence. The `default.rb` file in `recipes` contains the "default" recipe resources. -Because each section of the LAMP stack (Apache, MySQL, and PHP) will have its own recipe, the `default.rb` file is used to prepare your servers. +In this example, the `lamp_stack` cookbook's `default.rb` file is used to update the node's distribution software. -1. From within your `lamp_stack` directory, navigate to the `recipes` folder: +1. From within the `lamp_stack` directory, navigate to the `recipes` folder: cd recipes -1. Open `default.rb` and add the Ruby command below, which will run system updates: +1. Open the `default.rb` file and add the following code: {{< file "~/chef-repo/cookbooks/lamp_stack/recipe/default.rb" ruby >}} # @@ -83,21 +96,25 @@ end knife cookbook upload lamp_stack -1. Test that the recipe has been added to the chef server: +1. Verify that the recipe has been added to the Chef server: knife cookbook list + You should see a similar output: + + {{< output >}} + lamp_stack 0.1.0 + {{}} + 1. Add the recipe to your chosen node's *run list*, replacing `nodename` with your node's name: knife node run_list add nodename "recipe[lamp_stack]" - Because this is the default recipe, the recipe name does not need to be defined after `lamp_stack` cookbook in the code above. - -1. Access your chosen node and run the *chef-client*: +1. From your workstation, apply the configurations defined in the cookbook by running the chef-client on your node. Replace `nodename` with the name of your node: - chef-client + knife ssh 'name:nodename' 'sudo chef-client' -x root - It should output a successful Chef run. If not, review your code for any errors, usually defined in the output of the `chef-client` run. + Your output should display a successful Chef run. If not, review your code for any errors, usually defined in the output of the `chef-client` run. ## Apache @@ -139,20 +156,26 @@ end Because this is not the `default.rb` recipe, the recipe name, *apache*, must be appended to the recipe value. -1. From that **node**, run `chef-client`: + {{< note >}} + To view a list of all nodes managed by your, Chef server issue the following command from your workstation: + + knife node list + {{}} + +1. From your workstation, apply the configurations defined in the cookbook by running the chef-client on your node. Replace `nodename` with the name of your node: - chef-client + knife ssh 'name:nodename' 'sudo chef-client' -x root If the recipe fails due to a syntax error, Chef will note it during the output. 1. After a successful `chef-client` run, check to see if Apache is running: - systemctl status apache2 - - It should say that `apache2` is running. + knife ssh 'name:nodename' 'systemctl status apache2' -x root {{< note >}} -Repeat Steps 5-7 to upload the cookbook and run chef-client as needed through the rest of this guide to ensure your recipes are working properly and contain no errors. Remember to replace the recipe name in the run list code when adding a new recipe. +Repeat steps 4-7 to upload each recipe to your Chef server, as you create it. Run `chef-client` on your node, as needed, throughout the rest of this guide to ensure your recipes are working properly and contain no errors. When adding a new recipe, ensure you are using its correct name in the run list. + +This is not the recommended workflow for a production environment. You might consider creating different [Chef environments](https://docs.chef.io/environments.html) for testing, staging, and production. {{< /note >}} ### Configure Virtual Hosts @@ -163,7 +186,7 @@ This configuration is based off of the [How to Install a LAMP Stack on Ubuntu 16 chef generate attribute ~/chef-repo/cookbooks/lamp_stack default -1. Within the new `default.rb`, create the default values of the cookbook: +1. Within the new `default.rb`, create the default values for the cookbook: {{< file "~/chef-repo/cookbooks/lamp_stack/attributes/default.rb" ruby >}} default["lamp_stack"]["sites"]["example.com"] = { "port" => 80, "servername" => "example.com", "serveradmin" => "webmaster@example.com" } @@ -210,7 +233,7 @@ node["lamp_stack"]["sites"].each do |sitename, data| end {{< /file >}} -1. However, this does not create the directory itself. To do so, the `directory` resource should be used, with a `true` recursive value so all directories leading up to the `sitename` will be created. A permissions value of `0755` allows for the file owner to have full access to the directory, while group and regular users will have read and execute privileges: +1. Create the `document_root` directory. Declare a `directory` resource with a `true` recursive value so all directories leading up to the `sitename` will be created. A permissions value of `0755` allows for the file owner to have full access to the directory, while group and regular users will have read and execute privileges: {{< file "~/chef-repo/cookbooks/lamp_stack/apache.rb" ruby >}} node["lamp_stack"]["sites"].each do |sitename, data| @@ -275,7 +298,7 @@ end The name of the template resource should be the location where the virtual host file is placed on the nodes. The `source` is the name of the template file. Mode `0644` gives the file owner read and write privileges, and everyone else read privileges. The values defined in the `variables` section are taken from the attributes file, and they are the same values that are called upon in the template. -1. The sites now need to be enabled in Apache, and the server restarted. This should *only* occur if there are changes to the virtual hosts, so the `notifies` value should be added to the `template` resource. `notifies` tells Chef when things have changed, and **only then** runs the commands: +1. The sites need to be enabled in Apache, and the server restarted. This should *only* occur if there are changes to the virtual hosts, so the `notifies` value should be added to the `template` resource. `notifies` tells Chef when things have changed, and **only then** runs the commands: {{< file "~/chef-repo/cookbooks/lamp_stack/recipes/apache.rb" ruby >}} template "/etc/apache2/sites-available/#{sitename}.conf" do @@ -294,7 +317,7 @@ end The `notifies` command names the `:action` to be committed, then the resource, and resource name in square brackets. -1. `notifies` can also call on `execute` commands, which will run `a2ensite`and enable the sites we've made virtual hosts files for. Add the following `execute` command **above** the `template` resource code to create the `a2ensite` script: +1. `notifies` can also call on `execute` commands, which will run `a2ensite`and enable the sites that have corresponding virtual hosts files. Add the following `execute` command **above** the `template` resource code to create the `a2ensite` script: {{< file "~/chef-repo/cookbooks/lamp_stack/recipes/apache.rb" ruby >}} # [...] @@ -458,7 +481,7 @@ Chef contains a feature known as *data bags*. Data bags store information, and c knife data bag create mysql rtpass.json --secret-file ~/chef-repo/.chef/encrypted_data_bag_secret {{< note >}} -Some knife commands require that information be edited as JSON data using a text editor. Your `knife.rb` file should contain a configuration for the text editor to use for such commands. If your `knife.rb` file does not already contain this configuration, add `knife[:editor] = "/usr/bin/vim"` to the bottom of the file to set vim as the default text editor. +Some knife commands require that information be edited as JSON data using a text editor. Your `config.rb` file should contain a configuration for the text editor to use for such commands. If your `config.rb` file does not already contain this configuration, add `knife[:editor] = "/usr/bin/vim"` to the bottom of the file to set vim as the default text editor. {{}} You will be asked to edit the `rtpass.json` file: @@ -527,12 +550,6 @@ end `mysqldefault` is the name of the MySQL service for this container. The `inital_root_password` calls to the value defined in the text above, while the action creates the database and starts the MySQL service. - {{< note >}} -When running MySQL from your nodes you will need to define the socket: - - mysql -S /var/run/mysql-mysqldefault/mysqld.sock -p -{{< /note >}} - ## PHP 1. Under the recipes directory, create a new `php.rb` file. The commands below install PHP and all the required packages for working with Apache and MySQL: @@ -599,4 +616,4 @@ end knife cookbook upload lamp_stack knife node run_list add nodename "recipe[lamp_stack],recipe[lamp_stack::apache],recipe[lamp_stack::mysql],recipe[lamp_stack::php]" -You have just created a LAMP Stack cookbook. Through this guide, you should have learned to use the execute, package, service, node, directory, template, cookbook_file, and mysql_service resources within a recipe, as well as download and use LWRPs, create encrypted data bags, upload/update your cookbooks to the server, and use attributes, templates, and cookbook files, giving you a strong basis in Chef and cookbook creation for future projects. +You have just created a LAMP Stack cookbook. Through this guide, you should have learned to use the execute, package, service, node, directory, template, cookbook_file, and mysql_service resources within a recipe, as well as download and use LWRPs, create encrypted data bags, upload/update your cookbooks to the server, and use attributes, templates, and cookbook files. This gives you a strong basis in Chef and cookbook creation for future projects. diff --git a/docs/applications/configuration-management/getting-started-with-puppet-6-1-basic-installation-and-setup/index.md b/docs/applications/configuration-management/getting-started-with-puppet-6-1-basic-installation-and-setup/index.md new file mode 100644 index 00000000000..bc0f06e710c --- /dev/null +++ b/docs/applications/configuration-management/getting-started-with-puppet-6-1-basic-installation-and-setup/index.md @@ -0,0 +1,874 @@ +--- +author: + name: Linode + email: docs@linode.com +description: 'Basic instructions to set up and configure a Puppet master and agents using Ubuntu and CentOS servers.' +keywords: ["puppet installation", "configuration change management", "server automation"] +license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' +modified: 2019-01-15 +modified_by: + name: Linode +published: 2019-01-15 +title: Getting Started with Puppet - Basic Installation and Setup +external_resources: + - '[Puppet Labs](https://puppetlabs.com/)' + - '[Puppet Open Source Documentation](https://docs.puppetlabs.com/puppet/)' +--- + +[Puppet](https://puppetlabs.com/) is a configuration management tool that simplifies system administration. Puppet uses a client/server model in which your managed nodes, running a process called the Puppet *agent*, talk to and pull down configuration profiles from a Puppet *master*. + + + +Puppet deployments can range from small groups of servers up to enterprise-level operations. This guide will demonstrate how to install Puppet 6.1 on three servers: + +- A Puppet master running Ubuntu 18.04 +- A managed Puppet node running Ubuntu 18.04 +- A managed Puppet node running CentOS 7 + +After installation, the next section will show you how to secure these servers via Puppet. This section will demonstrate core features of the Puppet language. + +{{< note >}} +Most guides will instruct you to follow the [How to Secure your Server](/docs/security/securing-your-server/) guide before proceeding. Because Puppet will be used to perform this task, you should begin this guide as the `root` user. A limited user with administrative privileges will be configured via Puppet in later steps. +{{< /note >}} + +## Before You Begin + +The following table displays example system information for the servers that will be deployed in this guide: + +| Description | OS | Hostname | FQDN | IP | +| ----------------- | ----------- | ----- | ----- | ---- | +| Puppet master | Ubuntu 18.04 | puppet | puppet.example.com | 192.0.2.2 +| Node 1 (Ubuntu) | Ubuntu 18.04 | puppet-agent-ubuntu |puppet-agent-ubuntu.example.com | 192.0.2.3 | +| Node 2 (CentOS) | CentOS 7 | puppet-agent-centos | puppet-agent-centos.example.com | 192.0.2.4 | + +You can choose different hostnames and fully qualified domain names (FQDN) for each of your servers, and the IP addresses for your servers will be different from the example addresses listed. You will need to have a registered domain name in order to specify FQDNs for your servers. + +Throughout this guide, commands and code snippets will reference the values displayed in this table. Wherever such a value appears, replace it with your own value. + +### Create your Linodes + +1. Create three Linodes corresponding to the servers listed in the table above. Your Puppet master Linode should have at least four CPU cores; the [Linode 8GB](https://www.linode.com/pricing) plan is recommended. The two other nodes can be of any plan size, depending on how you intend to use them after Puppet is installed and configured. + +1. [Configure your timezone](/docs/getting-started/#set-the-timezone) on your master and agent nodes so that they all have the same time data. + +1. [Set the hostname](/docs/getting-started/#set-the-hostname) for each server. + +1. [Set the FQDN](/docs/networking/dns/using-your-systems-hosts-file/) for each Linode by editing the servers' `/etc/hosts` files. + + {{< disclosure-note "Example content for the hosts file" >}} +You can model the contents of your `/etc/hosts` files on these snippets: + +{{< file "Master" >}} +127.0.0.1 localhost +192.0.2.2 puppet.example.com puppet + +# The following lines are desirable for IPv6 capable hosts +::1 localhost ip6-localhost ip6-loopback +ff02::1 ip6-allnodes +ff02::2 ip6-allrouters +{{< /file >}} + +{{< file "Node 1 (Ubuntu)" >}} +127.0.0.1 localhost +192.0.2.3 puppet-agent-ubuntu.example.com puppet-agent-ubuntu + +# The following lines are desirable for IPv6 capable hosts +::1 localhost ip6-localhost ip6-loopback +ff02::1 ip6-allnodes +ff02::2 ip6-allrouters +{{< /file >}} + +{{< file "Node 2 (CentOS)" >}} +127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 +::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 +192.0.2.4 puppet-agent-centos.example.com puppet-agent-centos +{{< /file >}} + +{{< /disclosure-note >}} + +1. [Set up DNS records](/docs/platform/manager/dns-manager/#add-records) for your Linodes' FQDNs. For each Linode, create a new *A record* with the name specified by its FQDN and assign it to that Linode's IP address. + + If you don't use Linode's name servers for your domain, consult your name server authority's website for instructions on how to edit your DNS records. + + {{< content "update-dns-at-common-name-server-authorities" >}} + +## Puppet Master + +### Install the Puppet Server Software + +The Puppet master runs the `puppetserver` service, which is responsible for compiling and supplying configuration profiles to your managed nodes. + +The `puppetserver` service has the Puppet agent service as a dependency (which is just called `puppet` when running on your system). This means that the agent software will also be installed and can be run on your master. Because your master can run the agent service, you can configure your master via Puppet just as you can configure your other managed nodes. + +1. Log in to your Puppet master via SSH (as root): + + ssh root@puppet.example.com + +1. Download the Puppet repository, update your system packages, and install `puppetserver`: + + wget https://apt.puppetlabs.com/puppet-release-bionic.deb + dpkg -i puppet-release-bionic.deb + apt update + apt install puppetserver + +### Configure the Server Software + +1. Use the `puppet config` command to set values for the `dns_alt_names` setting: + + /opt/puppetlabs/bin/puppet config set dns_alt_names 'puppet,puppet.example.com' --section main + + If you inspect the configuration file, you'll see that the setting has been added: + + cat /etc/puppetlabs/puppet/puppet.conf + + {{< output >}} +[main] +dns_alt_names = puppet,puppet.example.com +# ... +{{< /output >}} + + {{< note >}} +The `puppet` command by default is not added to your PATH. Using Puppet's interactive commands requires a full file path. To avoid this, update your PATH for your existing shell session: + + export PATH=/opt/puppetlabs/bin:$PATH + +A more permanent solution would be to add this to your `.profile` or `.bashrc` files. +{{}} + +1. Update your Puppet master's `/etc/hosts` to resolve your managed nodes' IP addresses. For example, your `/etc/hosts` file might look like the following: + + {{< file "/etc/hosts" >}} +127.0.0.1 localhost +192.0.2.2 puppet.example.com puppet + +192.0.2.3 puppet-agent-ubuntu.example.com puppet-agent-ubuntu +192.0.2.4 puppet-agent-centos.example.com puppet-agent-centos + +# The following lines are desirable for IPv6 capable hosts +::1 localhost ip6-localhost ip6-loopback +ff02::1 ip6-allnodes +ff02::2 ip6-allrouters +{{< /file >}} + + {{< note >}} +This snippet incorporates the FQDN declaration described in the [Create your Linodes](#create-your-linodes) section. +{{< /note >}} + +1. Start and enable the `puppetserver` service: + + systemctl start puppetserver + systemctl enable puppetserver + + By default, the Puppet master listens for client connections on port 8140. If the `puppetserver` service fails to start, check that the port is not already in use: + + netstat -anpl | grep 8140 + +## Puppet Agents + +### Install Puppet Agent + +1. On your managed node running **Ubuntu 18.04**, install the `puppet-agent` package: + + wget https://apt.puppetlabs.com/puppet-release-bionic.deb + dpkg -i puppet-release-bionic.deb + apt update + apt install puppet-agent + +1. On your managed node running **CentOS 7**, enter: + + rpm -Uvh https://yum.puppet.com/puppet/puppet-release-el-7.noarch.rpm + yum install puppet-agent + +### Configure Puppet Agent + +1. Modify your managed nodes' hosts files to resolve the Puppet master's IP. To do so, add a line like: + + {{< file "/etc/hosts" >}} +192.0.2.2 puppet.example.com puppet +{{< /file >}} + + {{< disclosure-note "Example content for the hosts file" >}} +You can model the contents of your managed nodes' `/etc/hosts` files on the following snippets. These incorporate the FQDN declarations described in the [Create your Linodes](#create-your-linodes) section: + +{{< file "Node 1 (Ubuntu)" >}} +127.0.0.1 localhost +192.0.2.3 puppet-agent-ubuntu.example.com puppet-agent-ubuntu + +192.0.2.2 puppet.example.com puppet + +# The following lines are desirable for IPv6 capable hosts +::1 localhost ip6-localhost ip6-loopback +ff02::1 ip6-allnodes +ff02::2 ip6-allrouters +{{< /file >}} + +{{< file "Node 2 (CentOS)" >}} +127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 +::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 +192.0.2.4 puppet-agent-centos.example.com puppet-agent-centos + +192.0.2.2 puppet.example.com puppet +{{< /file >}} + +{{< /disclosure-note >}} + +1. On each managed node, use the `puppet config` command to set the value for your `server` setting to the FQDN of the master: + + /opt/puppetlabs/bin/puppet config set server 'puppet.example.com' --section main + + If you inspect the configuration file on the nodes, you'll see that the setting has been added: + + cat /etc/puppetlabs/puppet/puppet.conf + + {{< output >}} +[main] +server = puppet.example.com +# ... +{{< /output >}} + +1. Use the `puppet resource` command to start and enable the Puppet agent service: + + /opt/puppetlabs/bin/puppet resource service puppet ensure=running enable=true + + {{< note >}} +On systemd systems, the above command is equivalent to using these two `systemctl` commands: + + systemctl start puppet + systemctl enable puppet +{{< /note >}} + +### Generate and Sign Certificates + +Before your managed nodes can receive configurations from the master, they first need to be authenticated: + +1. On your **Puppet agents**, generate a certificate for the Puppet master to sign: + + /opt/puppetlabs/bin/puppet agent -t + + This command will output an error, stating that no certificate has been found. This error is because the generated certificate needs to be approved by the Puppet master. + +1. Log in to your **Puppet master** and list the certificates that need approval: + + /opt/puppetlabs/bin/puppetserver ca list + + It should output a list with your agent nodes' hostnames. + +1. Approve the certificates: + + /opt/puppetlabs/bin/puppetserver ca sign --certname puppet-agent-ubuntu.example.com,puppet-agent-centos.example.com + +1. Return to the **Puppet agent** nodes and run the Puppet agent again: + + /opt/puppetlabs/bin/puppet agent -t + + You should see something like the following: + + {{< output >}} +Info: Downloaded certificate for hostname.example.com from puppet +Info: Using configured environment 'production' +Info: Retrieving pluginfacts +Info: Retrieving plugin +Info: Retrieving locales +Info: Caching catalog for hostname.example.com +Info: Applying configuration version '1547066428' +Info: Creating state file /opt/puppetlabs/puppet/cache/state/state.yaml +Notice: Applied catalog in 0.02 seconds +{{< /output >}} + +## Add Modules to Configure Agent Nodes + +The Puppet master and agent nodes are now functional, but they are not secure. Based on concepts from the [How to Secure your Server](/docs/security/securing-your-server/) guide, a limited user and a firewall should be configured. This can be done on all nodes through the creation of basic Puppet modules, shown below. + +{{< note >}} +This is not meant to provide a basis for a fully-hardened server, and is intended only as a starting point. Alter and add firewall rules and other configuration options, depending on your specific needs. +{{< /note >}} + +Puppet modules are Puppet's prescribed way of organizing configuration code to serve specific purposes, like installing and configuration an application. You can create custom modules, or you can download and use modules published on [Puppet Forge](https://forge.puppet.com/). + +### Add a Limited User + +To create a new limited user on your nodes, you will create and apply a new module called `accounts`. This module will employ the [`user` resource](https://puppet.com/docs/puppet/6.1/types/user.html). + +1. From the **Puppet master**, navigate to the `/etc/puppetlabs/code/environments/production/modules` directory. When a managed node requests its configuration from the master, the Puppet server process will look in this location for your modules: + + cd /etc/puppetlabs/code/environments/production/modules/ + +1. Create the directory for a new `accounts` module: + + mkdir accounts + cd accounts + +1. Create the following directories inside the `accounts` module: + + mkdir {examples,files,manifests,templates} + + | Directory | Description | + | --------- | ----------- | + | `manifests` | The Puppet code which powers the module | + | `files` | Static files to be copied to managed nodes | + | `templates` | Template files to be copied to managed nodes that can e customized with variables | + | `examples` | Example code which shows how to use the module | + + {{< note >}} +Review Puppet's [Module fundamentals](https://puppet.com/docs/puppet/6.1/modules_fundamentals.html#module-structure) article for more information on how a module is structured. +{{< /note >}} + +1. Navigate to the `manifests` directory: + + cd manifests + +1. Any file which contains Puppet code is called a *manifest*, and each manifest file ends in `.pp`. When located inside a module, a manifest should only define one class. If a module's manifests directory has an `init.pp` file, the class definition it contains is considered the main class for the module. The class definition inside `init.pp` should have the same name as the module. + + Create an `init.pp` file with the contents of the following snippet. Replace all instances of `username` with a username of your choosing: + + {{< file "accounts/manifests/init.pp" puppet >}} +class accounts { + + user { 'username': + ensure => present, + home => '/home/username', + shell => '/bin/bash', + managehome => true, + gid => 'username', + } + +} +{{< /file >}} + + | Option | Description | + | --------- | ----------- | + | `ensure` | Ensures that the user exists if set to `present`, or does not exist if set to `absent` | + | `home` | The path for the user's home directory | + | `managehome` | Controls whether a home directory should be created when creating the user | + | `shell` | The path to the shell for the user | + | `gid` | The user's primary group | + +1. Although the class declares what the user's primary group should be, it will not create the group itself. Create a new file called `groups.pp` inside the `manifests` directory with the following contents. Replace `username` with your chosen username: + + {{< file "accounts/manifests/groups.pp" puppet >}} +class accounts::groups { + + group { 'username': + ensure => present, + } + +} +{{< /file >}} + +1. Your `accounts` class can [declare](https://puppet.com/docs/puppet/6.1/lang_classes.html#declaring-classes) your new `accounts::groups` class for use within the `accounts` class scope. Open your `init.pp` in your editor and enter a new `include` declaration at the beginning of the class: + + {{< file "accounts/manifests/init.pp" puppet >}} +class accounts { + + include accounts::groups + + # ... + +} + +{{< /file >}} + +1. The new user should have administrative privileges. Because we have agent nodes on both Debian- and Red Hat-based systems, the new user needs to be in the `sudo` group on Debian systems, and the `wheel` group on Red Hat systems. + + This value can be set dynamically through the use of [Puppet *facts*](https://puppet.com/docs/puppet/6.1/lang_facts_and_builtin_vars.html). The facts system collects system information about your nodes and makes it available in your manifests. + + Add a selector statement to the top of your `accounts` class: + + {{< file "accounts/manifests/init.pp" puppet >}} +class accounts { + + $rootgroup = $osfamily ? { + 'Debian' => 'sudo', + 'RedHat' => 'wheel', + default => warning('This distribution is not supported by the Accounts module'), + } + + include accounts::groups + + # ... + +} + +{{< /file >}} + + This code defines the value for the `$rootgroup` variable by checking the value of `$osfamily`, which is one of Puppet's [core facts](https://puppet.com/docs/facter/6.1/core_facts.html). If the value for `$osfamily` does not match Debian or Red Hat, the `default` value will output a warning that the distribution selected is not supported by this module. + + {{< note >}} +The Puppet Configuration Language executes code from top to bottom. Because the `user` resource declaration will reference the `$rootgroup` variable, you must define `$rootgroup` *before* the `user` declaration. +{{< /note >}} + +1. Update the user resource to include the `groups` option as follows: + + {{< file "accounts/manifests/init.pp" puppet >}} +# ... + +user { 'username': + ensure => present, + home => '/home/username', + shell => '/bin/bash', + managehome => true, + gid => 'username', + groups => "$rootgroup", +} + +# ... +{{< /file >}} + + The value `"$rootgroup"` is enclosed in double quotes `" "` instead of single quotes `' '` because it is a variable which needs to be interpolated in your code. + +1. The final value that needs to be added is the user's password. Since we do not want to use plain text, the password should be supplied to Puppet as a SHA1 digest, which is supported by default. Generate a digest with the `openssl` command: + + openssl passwd -1 + + You will be prompted to enter your password. A hashed password will be output. Copy this value to your clipboard. + +1. Update the user resource to include the `password` option as follows; insert your copied password hash as the value for the option: + + {{< file "accounts/manifests/init.pp" puppet >}} +# ... + +user { 'username': + ensure => present, + home => '/home/username', + shell => '/bin/bash', + managehome => true, + gid => 'username', + groups => "$rootgroup", + password => 'your_password_hash', +} + +# ... +{{< /file >}} + + {{< caution >}} +The hashed password **must** be included in single quotes `' '`. +{{< /caution >}} + +1. After saving your changes, use the Puppet parser to ensure that the code is correct: + + /opt/puppetlabs/bin/puppet parser validate init.pp + + Any errors that need to be addressed will be logged to standard output. If nothing is returned, your code is valid. + +1. Navigate to the `examples` directory and create another `init.pp` file: + + cd ../examples + + {{< file "accounts/examples/init.pp" puppet >}} +include accounts +{{< /file >}} + +1. While still in the `examples` directory, test the module: + + /opt/puppetlabs/bin/puppet apply --noop init.pp + + {{< note >}} +The `--noop` parameter prevents Puppet from actually applying the module to your system and making any changes. +{{< /note >}} + + It should return: + + {{< output >}} +Notice: Compiled catalog for puppet.example.com in environment production in 0.26 seconds +Notice: /Stage[main]/Accounts::Groups/Group[username]/ensure: current_value absent, should be present (noop) +Notice: Class[Accounts::Groups]: Would have triggered 'refresh' from 1 events +Notice: /Stage[main]/Accounts/User[username]/ensure: current_value absent, should be present (noop) +Notice: Class[Accounts]: Would have triggered 'refresh' from 1 events +Notice: Stage[main]: Would have triggered 'refresh' from 2 events +Notice: Finished catalog run in 0.02 seconds +{{< /output >}} + +1. Again from the `examples` directory, run `puppet apply` to make these changes to the Puppet master server: + + /opt/puppetlabs/bin/puppet apply init.pp + + Puppet will create your limited Linux user on your master. + +1. Log out as `root` and log in to the Puppet master as your new user. + +### Edit SSH Settings + +Although a new limited user has successfully been added to the Puppet master, it is still possible to login to the system as root. To properly secure your system, root access should be disabled. + +{{< note >}} +Because you are now logged in to the Puppet master as a limited user, you will need to execute commands and edit files with the user's sudo privileges. +{{< /note >}} + +1. Navigate to the `files` directory within the `accounts` module: + + cd /etc/puppetlabs/code/environments/production/modules/accounts/files + +1. Copy your system's existing `sshd_config` file to this directory: + + sudo cp /etc/ssh/sshd_config . + +1. Open the file in your editor (making sure that you open it with `sudo` privileges) and set the `PermitRootLogin` value to `no`: + + {{< file "accounts/files/sshd_config" aconf >}} +PermitRootLogin no + +{{< /file >}} + +1. Navigate back to the `manifests` directory: + + cd ../manifests + +1. Create a new manifest called `ssh.pp`. Use the `file` resource to replace the default SSH configuration file with one managed by Puppet: + + {{< file "accounts/manifests/ssh.pp" puppet >}} +class accounts::ssh { + + file { '/etc/ssh/sshd_config': + ensure => present, + source => 'puppet:///modules/accounts/sshd_config', + } + +} +{{< /file >}} + + {{< note >}} +The `files` directory is omitted from the `source` line because the `files` folder is the default location of files within a module. For more information on the format used to access resources in a module, refer to the [official Puppet module documentation](https://docs.puppet.com/puppet/6.1/modules_fundamentals.html#module-layout). +{{< /note >}} + +1. Create a second resource to restart the SSH service and set it to run whenever `sshd_config` is changed. This will also require a selector statement because the SSH service is named `ssh` on Debian systems and `sshd` on Red Hat systems: + + {{< file "accounts/manifests/ssh.pp" puppet >}} +class accounts::ssh { + + $sshname = $osfamily ? { + 'Debian' => 'ssh', + 'RedHat' => 'sshd', + default => warning('This distribution is not supported by the Accounts module'), + } + + file { '/etc/ssh/sshd_config': + ensure => present, + source => 'puppet:///modules/accounts/sshd_config', + notify => Service["$sshname"], + } + + service { "$sshname": + hasrestart => true, + } + +} +{{< /file >}} + + {{< note >}} +`notify` is one of Puppet's [relationship metaparameters](https://puppet.com/docs/puppet/6.1/lang_relationships.html). +{{< /note >}} + +1. Include the `accounts::ssh` class within the `accounts` class in `init.pp`: + + {{< file "accounts/manifests/init.pp" puppet >}} +class accounts { + + # ... + + include accounts::groups + include accounts::ssh + + # ... + +} +{{< /file >}} + + {{< disclosure-note "The complete init.pp" >}} +The contents of your `init.pp` should now look like the following snippet: + +{{< file "accounts/manifests/init.pp" puppet >}} +class accounts { + + $rootgroup = $osfamily ? { + 'Debian' => 'sudo', + 'RedHat' => 'wheel', + default => warning('This distro not supported by Accounts module'), + } + + include accounts::groups + include accounts::ssh + + user { 'example': + ensure => present, + home => '/home/username', + shell => '/bin/bash', + managehome => true, + gid => 'username', + groups => "$rootgroup", + password => 'your_password_hash' + } + +} +{{< /file >}} +{{< /disclosure-note >}} + +1. Run the Puppet parser to test the syntax of the new class, then navigate to the `examples` directory to test and run the update to your `accounts` class: + + sudo /opt/puppetlabs/bin/puppet parser validate ssh.pp + cd ../examples + sudo /opt/puppetlabs/bin/puppet apply --noop init.pp + sudo /opt/puppetlabs/bin/puppet apply init.pp + + {{< note >}} +You may see the following line in your output when validating: + +{{< output >}} +Error: Removing mount "files": /etc/puppet/files does not exist or is not a directory +{{< /output >}} + +This refers to a Puppet configuration file, not the module resource you're trying to copy. If this is the only error in your output, the operation should still succeed. +{{< /note >}} + +1. To ensure that the `ssh` class is working properly, log out of the Puppet master and then try to log in as `root`. You should not be able to do so. + +### Add and Configure IPtables + +To complete this guide's security settings, the firewall needs to be configure on your Puppet master and nodes. The `iptables` firewall software will be used. + +1. By default, changes to your `iptables` rules will not persist across reboots. To avoid this, install the appropriate package on your Puppet master and nodes: + + **Ubuntu/Debian**: + + sudo apt install iptables-persistent + + **CentOS 7**: + + CentOS 7 uses firewalld by default as a controller for iptables. Be sure firewalld is stopped and disabled before starting to work directly with iptables: + + sudo systemctl stop firewalld && sudo systemctl disable firewalld + sudo yum install iptables-services + +1. On your Puppet master, install [Puppet Lab's firewall module](https://forge.puppet.com/puppetlabs/firewall) from the Puppet Forge: + + sudo /opt/puppetlabs/bin/puppet module install puppetlabs-firewall + + The module will be installed in your `/etc/puppetlabs/code/environments/production/modules` directory. + +1. Navigate to the `manifests` directory inside the new `firewall` module: + + cd /etc/puppetlabs/code/environments/production/modules/firewall/manifests/ + +2. Create a file titled `pre.pp`, which will contain all basic networking rules that should be run first: + + {{< file "firewall/manifests/pre.pp" puppet >}} +class firewall::pre { + + Firewall { + require => undef, + } + + # Accept all loopback traffic + firewall { '000 lo traffic': + proto => 'all', + iniface => 'lo', + action => 'accept', + }-> + + #Drop non-loopback traffic + firewall { '001 reject non-lo': + proto => 'all', + iniface => '! lo', + destination => '127.0.0.0/8', + action => 'reject', + }-> + + #Accept established inbound connections + firewall { '002 accept established': + proto => 'all', + state => ['RELATED', 'ESTABLISHED'], + action => 'accept', + }-> + + #Allow all outbound traffic + firewall { '003 allow outbound': + chain => 'OUTPUT', + action => 'accept', + }-> + + #Allow ICMP/ping + firewall { '004 allow icmp': + proto => 'icmp', + action => 'accept', + } + + #Allow SSH connections + firewall { '005 Allow SSH': + dport => '22', + proto => 'tcp', + action => 'accept', + }-> + + #Allow HTTP/HTTPS connections + firewall { '006 HTTP/HTTPS connections': + dport => ['80', '443'], + proto => 'tcp', + action => 'accept', + } + +} + +{{< /file >}} + +1. In the same directory, create `post.pp`, which will run any firewall rules that need to be input last: + + {{< file "firewall/manifests/post.pp" puppet >}} +class firewall::post { + + firewall { '999 drop all': + proto => 'all', + action => 'drop', + before => undef, + } + +} + +{{< /file >}} + + These rules will direct the system to drop all inbound traffic that is not already permitted in the firewall. + +1. Run the Puppet parser on both files to check their syntax for errors: + + sudo /opt/puppetlabs/bin/puppet parser validate pre.pp + sudo /opt/puppetlabs/bin/puppet parser validate post.pp + +1. Navigate to the main `manifests` directory: + + cd /etc/puppetlabs/code/environments/production/manifests + +1. Create a file named `site.pp` inside `/etc/puppetlabs/code/environments/production/manifests`. This file is the [main manifest](https://puppet.com/docs/puppet/6.1/dirs_manifest.html) for the Puppet server service. It is used to map modules, classes, and resources to the nodes that they should be applied to. + + {{< file "site.pp" puppet >}} +node default { + +} + +node 'puppet.example.com' { + + include accounts + + resources { 'firewall': + purge => true, + } + + Firewall { + before => Class['firewall::post'], + require => Class['firewall::pre'], + } + + class { ['firewall::pre', 'firewall::post']: } + + firewall { '200 Allow Puppet Master': + dport => '8140', + proto => 'tcp', + action => 'accept', + } + +} +{{< /file >}} + +1. Run the `site.pp` file through the Puppet parser to check its syntax for errors. Then, test the file with the `--noop` option to see if it will run: + + sudo /opt/puppetlabs/bin/puppet parser validate site.pp + sudo /opt/puppetlabs/bin/puppet apply --noop site.pp + + If successful, run `puppet apply` without the `--noop` option: + + sudo /opt/puppetlabs/bin/puppet apply site.pp + +1. Once Puppet has finished applying the changes, check the Puppet master's iptables rules: + + sudo iptables -L + + It should return: + + Chain INPUT (policy ACCEPT) + target prot opt source destination + ACCEPT all -- anywhere anywhere /* 000 lo traffic */ + REJECT all -- anywhere 127.0.0.0/8 /* 001 reject non-lo */ reject-with icmp-port-unreachable + ACCEPT all -- anywhere anywhere /* 002 accept established */ state RELATED,ESTABLISHED + ACCEPT icmp -- anywhere anywhere /* 004 allow icmp */ + ACCEPT tcp -- anywhere anywhere multiport ports ssh /* 005 Allow SSH */ + ACCEPT tcp -- anywhere anywhere multiport ports http,https /* 006 HTTP/HTTPS connections */ + ACCEPT tcp -- anywhere anywhere multiport ports 8140 /* 200 Allow Puppet Master */ + DROP all -- anywhere anywhere /* 999 drop all */ + + Chain FORWARD (policy ACCEPT) + target prot opt source destination + + Chain OUTPUT (policy ACCEPT) + target prot opt source destination + ACCEPT tcp -- anywhere anywhere /* 003 allow outbound */ + +### Apply Modules to the Agent Nodes + +Now that the `accounts` and `firewall` modules have been created, tested, and run on the Puppet master, it is time to apply them to your managed nodes. + +1. On the **Puppet master**, navigate to `/etc/puppetlabs/code/environments/production/manifests`: + + cd /etc/puppetlabs/code/environments/production/manifests + +1. Update `site.pp` to declare the modules, classes, and resources that should be applied to each managed node: + + {{< file "site.pp" puppet >}} +node default { + +} + +node 'puppet.example.com' { + # ... +} + +node 'puppet-agent-ubuntu.example.com' { + + include accounts + + resources { 'firewall': + purge => true, + } + + Firewall { + before => Class['firewall::post'], + require => Class['firewall::pre'], + } + + class { ['firewall::pre', 'firewall::post']: } + +} + +node 'puppet-agent-centos.example.com' { + + include accounts + + resources { 'firewall': + purge => true, + } + + Firewall { + before => Class['firewall::post'], + require => Class['firewall::pre'], + } + + class { ['firewall::pre', 'firewall::post']: } + +} +{{< /file >}} + +1. By default, the Puppet agent service on your managed nodes will automatically check with the master once every 30 minutes and apply any new configurations from the master. You can also manually invoke the Puppet agent process in-between automatic agent runs. + + Log in to each managed node (as root) and run the Puppet agent: + + /opt/puppetlabs/bin/puppet agent -t + +1. To ensure the Puppet agent worked: + + - Log out from your root SSH session and log back in as the limited user that was created. + + - Check the node's firewall rules: + + sudo iptables -L + +Congratulations! You've successfully installed Puppet on a master and two managed nodes. Now that you've confirmed everything is working, you can create additional modules to automate configuration management on your nodes. For more information, review Puppet's [open source documentation](https://docs.puppet.com/puppet/latest/reference/modules_fundamentals.html). You can also install and use modules others have created on the [Puppet Forge](https://docs.puppet.com/puppet/latest/reference/modules_installing.html). diff --git a/docs/applications/configuration-management/getting-started-with-puppet-6-1-basic-installation-and-setup/install-puppet-title.png b/docs/applications/configuration-management/getting-started-with-puppet-6-1-basic-installation-and-setup/install-puppet-title.png new file mode 100644 index 00000000000..ffe2096351d Binary files /dev/null and b/docs/applications/configuration-management/getting-started-with-puppet-6-1-basic-installation-and-setup/install-puppet-title.png differ diff --git a/docs/applications/configuration-management/install-a-chef-server-workstation-on-ubuntu-18-04/index.md b/docs/applications/configuration-management/install-a-chef-server-workstation-on-ubuntu-18-04/index.md index 8814bddf332..0a2da35024a 100644 --- a/docs/applications/configuration-management/install-a-chef-server-workstation-on-ubuntu-18-04/index.md +++ b/docs/applications/configuration-management/install-a-chef-server-workstation-on-ubuntu-18-04/index.md @@ -3,22 +3,22 @@ author: name: Linode email: docs@linode.com description: 'Instructions on how to configure a Chef server and a virtual workstation, and how to bootstrap a client node on Ubuntu 18.04.' -keywords: ["chef", "chef installation", "configuration change management", "server automation", "chef server", "chef workstation", "chef-client", "knife.rb", "version control"] +keywords: ["chef", "configuration management", "server automation", "chef server", "chef workstation", "chef-client"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' published: 2018-08-06 -modified: 2018-08-06 +modified: 2019-01-17 modified_by: name: Linode title: 'Install a Chef Server Workstation on Ubuntu 18.04' --- -[Chef](http://www.chef.io) is an automation platform that "turns infrastructure into code," allowing users to manage and deploy resources across multiple servers, or *nodes*. Chef allows users to create and download recipes (stored in *cookbooks*) to automate content and policies on these nodes. - -Chef is comprised of a Chef server, one or more workstations, and a number of nodes that are managed by the chef-client installed on each node. +[Chef](http://www.chef.io) is a Ruby based configuration management tool used to define infrastructure as code. This enables users to automate the management of many *nodes* and maintain consistency across those nodes. *Recipes* declare the desired state for managed nodes and are created on a user's *workstation* using the *Chef Workstation* package. Your recipes are distributed across nodes via a *Chef server*. A *Chef client*, installed on each node, is in charge of applying the recipe to its corresponding node. [![chef_graph-small.png](chef-graph-small.png)](chef-graph.png "Illustration showing that workstations talk to Nodes via the Chef Server") -This guide shows how to create and configure a Chef server, a virtual workstation, and how to bootstrap a node to run the chef-client, all on individual Linodes. +This guide will show you how to create and configure a Chef server and workstation. You will also bootstrap a node to manage with Chef. This work will require three individual Linodes. + +See [A Beginner's Guide to Chef](/docs/applications/configuration-management/beginners-guide-chef/) for an introduction to Chef concepts. {{< note >}} This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you're not familiar with the `sudo` command, you can check our [Users and Groups](/docs/tools-reference/linux-users-and-groups/#understanding-sudo) guide. @@ -26,23 +26,24 @@ This guide is written for a non-root user. Commands that require elevated privil ## Prerequisites -- One 4GB Linode, running Ubuntu 18.04, to host the Chef server -- Two Linodes of any size, each running Ubuntu 18.04, to host a workstation and a node -- Each Linode should be configured per the [Getting Started](/docs/getting-started/) and [Securing Your Sever](/docs/security/securing-your-server/) guides -- Each Linode needs to be configured with a valid FQDN +- One 8GB Linode running Ubuntu 18.04. This Linode will host the Chef server. + - Assign a Domain to the Chef server. Ensure your domain has a corresponding domain zone, NS record, and A/AAA record. See the [DNS Manager guide](/docs/platform/manager/dns-manager-new-manager/#add-a-domain-zone) for details. + - Ensure your Chef server's hostname is the same as its Domain name. Your Chef server will automatically create SSL certificates based on the Linode's hostname. +- Two 2 GB Linodes, each running Ubuntu 18.04. One Linode will host a workstation and the other a node to be managed by Chef. +- The workstation and Chef server should be configured per the [Getting Started](/docs/getting-started/) and [Securing Your Sever](/docs/security/securing-your-server/) guides. Once your node is [bootstrapped](/docs/applications/configuration-management/install-a-chef-server-workstation-on-ubuntu-18-04/#bootstrap-a-node), you can use a Chef cookbook to secure your node. Consider using the [Users](https://supermarket.chef.io/cookbooks/users) cookbook and the [Firewall](https://supermarket.chef.io/cookbooks/firewall) cookbook for this work. - Ensure that all servers are up-to-date: sudo apt update && sudo apt upgrade ## The Chef Server -The Chef server is the hub of interaction between all workstations and nodes using Chef. Changes made through workstations are uploaded to the Chef server, which is then accessed by `chef-client` and used to configure each individual node. +The Chef server is the hub of interaction between all workstations and nodes under Chef management. Changes made to configuration code on workstations are pushed to the Chef server, and then pulled by a node's `chef-client` to apply the configurations. ### Install the Chef Server 1. Download the [latest Chef server core](https://downloads.chef.io/chef-server/#ubuntu): - wget https://packages.chef.io/files/current/chef-server/12.17.54+20180531095715/ubuntu/18.04/chef-server-core_12.17.54+20180531095715-1_amd64.deb + wget https://packages.chef.io/files/stable/chef-server/12.18.14/ubuntu/18.04/chef-server-core_12.18.14-1_amd64.deb 1. Install the server: @@ -68,50 +69,78 @@ In order to link workstations and nodes to the Chef server, create an administra sudo chef-server-ctl user-create USER_NAME FIRST_NAME LAST_NAME EMAIL 'PASSWORD' --filename ~/.chef/USER_NAME.pem + To view a list of all users on your Chef server issue the following command: + + sudo chef-server-ctl user-list + 1. Create an organization and add the user created in the previous step to the admins and billing admins security groups. Replace `ORG_NAME` with a short identifier for the organization, `ORG_FULL_NAME` with the organizations' complete name, `USER_NAME` with the username created in the step above and `ORG_NAME.pem` with organization's short identifier followed by `.pem`: sudo chef-server-ctl org-create ORG_NAME "ORG_FULL_NAME" --association_user USER_NAME --filename ~/.chef/ORG_NAME.pem - With the Chef server installed and the RSA keys generated, you can begin configuring your workstation, where all major work will be performed for your Chef's nodes. + To view a list of all organizations on your Chef server, use the following command: + + sudo chef-server-ctl org-list + +With the Chef server installed and the RSA keys generated, you can begin configuring your workstation. The workstation is where all major configurations will be created for your nodes. ## Chef Workstations -Your Chef workstation is where you create and configure any recipes, cookbooks, attributes, and other changes made to your Chef configurations. Although this can be a local machine running any OS, there is some benefit to keeping a remote server as your workstation so you can access it from anywhere. +The Chef workstation is where you create and configure any recipes, cookbooks, attributes, and other changes necessary to manage your nodes. Although this can be a local machine running any OS, there is some benefit to keeping a remote server as your workstation so you can access it from anywhere. + +In this section, you will download and install the Chef Workstation package, which provides all tools also included with the [ChefDK](https://docs.chef.io/about_chefdk.html), Chef's development kit. ### Setting Up a Workstation -1. [Download the latest Chef Development Kit](https://downloads.chef.io/chef-dk/ubuntu/): +1. [Download the latest Chef Workstation](https://downloads.chef.io/chef-workstation/0.2.43#ubuntu): - wget https://packages.chef.io/files/stable/chefdk/3.1.0/ubuntu/18.04/chefdk_3.1.0-1_amd64.deb + wget https://packages.chef.io/files/stable/chef-workstation/0.2.43/ubuntu/18.04/chef-workstation_0.2.43-1_amd64.deb -1. Install ChefDK: +1. Install Chef Workstation: - sudo dpkg -i chefdk_*.deb + dpkg -i chef-workstation_*.deb 1. Remove the installation file: - rm chefdk_*.deb + rm chef-workstation_*.deb -1. Generate the `chef-repo` and move into the newly-created directory: +1. Create your Chef repository. The `chef-repo` directory will store your Chef cookbooks and other related files. - chef generate app chef-repo - cd chef-repo + chef generate repo chef-repo -1. Make the `.chef` directory to store the `knife.rb`, `ORGANIZATION-validator.pem`, and `USER.pem` files later: +1. Ensure that your workstation's `/etc/hosts` file maps its IP address to your workstation's hostname. For example: - mkdir .chef + {{< file "/etc/hosts">}} +127.0.0.1 localhost +192.0.2.0 workstation +... + {{}} + +1. Create a `.chef` subdirectory. The `.chef` subdirectory will store your [Knife](/docs/applications/configuration-management/beginners-guide-chef/#knife) configuration file and your `.pem` files that are used for RSA key pair authentication with the Chef server. Move into the `chef-repo` directory: + + mkdir ~/chef-repo/.chef + cd chef-repo ### Add the RSA Private Keys -1. The RSA private keys generated when setting up the Chef server now need to be placed on the workstation. The process behind this will vary depending on whether or not you are using SSH key pair authentication to log into your Linodes. +Authentication between the Chef server and workstation and/or nodes is completed with public key encryption. This ensures that the Chef server only communicates with trusted machines. In this section, the RSA private keys, generated when setting up the Chef server, will be copied to the workstation to allow communicate between the Chef server and workstation. + +1. If you do not already have an RSA key-pair on your workstation, generate one. This key-pair will be used to gain access to the Chef server and then transfer their `.pem` files: - - **If you are not using key pair authentication**, copy the file directly from the Chef Server. Replace `user` with your username on the server, and `192.0.2.0` with the URL or IP of your Chef Server: + ssh-keygen -b 4096 - scp user@192.0.2.0:~/.chef/*.pem ~/chef-repo/.chef/ + Press **Enter** to use the default names `id_rsa` and `id_rsa.pub` in `/home/your_username/.ssh` before entering your passphrase. - - **If you are using key pair authentication**, then from your **local terminal** copy the `.pem` files from your server to your workstation using the `scp` command. Replace `user` with the appropriate username, and `192.0.2.0` with the URL or IP for your Chef Server and `203.0.113.0` with the URL or IP for your workstation: + {{< note >}} + If you have disabled SSH password authentication on your Chef server's Linode, as recommended by the [How to Secure Your Server](/docs/security/securing-your-server/#ssh-daemon-options) guide, re-enable SSH password authentication prior to performing these steps. Be sure to disable it again once you have added your workstation's public ssh key to the Chef server's Linode. + {{}} - scp -3 user@192.0.2.0:~/.chef/*.pem user@203.0.113.0:~/chef-repo/.chef/ + Upload your workstation's public key to the Linode hosting the Chef server. Ensure you replace `example_user` with the Chef server's user account and `192.0.2.0` with its IP address: + + ssh-copy-id example_user@192.0.2.0 + +1. Copy the `.pem` files from your Chef server to your workstation using the `scp` command. Replace `user` with the appropriate username, and `192.0.2.0` with your Chef server's IP: + + scp example_user@192.0.2.0:~/.chef/*.pem ~/chef-repo/.chef/ 1. Confirm that the files have been copied successfully by listing the contents of the `.chef` directory: @@ -121,23 +150,24 @@ Your Chef workstation is where you create and configure any recipes, cookbooks, ### Add Version Control -The workstation is used to add and edit cookbooks and other configuration files. It is beneficial to implement some form of version control. The ChefDK adds the Git component to your workstation and initializes a Git repository in the directory used to generate the Chef repo. Configure Git to add your username and email, and to add and commit any new files generated in the steps above. +The workstation is used to create, download, and edit cookbooks and other related files. You should track any changes made to these files with version control software, like Git. The Chef Workstation adds the Git component to your workstation and initializes a Git repository in the directory where the `chef-repo` was generated. Configure Git by adding your username and email, and add and commit any new files created in the steps above. 1. Configure Git by adding your username and email, replacing the needed values: git config --global user.name yourname git config --global user.email user@email.com -1. Add the `.chef` directory to the `.gitignore` file: +2. Add the `.chef` directory to the `.gitignore` file: - echo ".chef" > .gitignore + echo ".chef" > ~/chef-repo/.gitignore -1. Add and commit all existing files: +3. Move into the `~/chef-repo` directory, if you are not already there and add and commit all existing files: + cd ~/chef-repo git add . git commit -m "initial commit" -1. Make sure the directory is clean: +4. Make sure the directory is clean: git status @@ -148,13 +178,23 @@ The workstation is used to add and edit cookbooks and other configuration files. nothing to commit, working directory clean {{}} -### Generate knife.rb +## Generate your First Cookbook + +1. Generate a new Chef cookbook: -1. Create a knife configuration file by navigating to your `~/chef-repo/.chef` folder and creating a file named `knife.rb` in your chosen text editor. + chef generate cookbook my_cookbook +1. Generate the `chef-repo` and move into the newly-created directory: + + chef generate app chef-repo + cd chef-repo -1. Copy the following configuration into the `knife.rb` file: +### Configure Knife - {{< file "~/chef-repo/.chef/knife.rb" ruby >}} +1. Create a knife configuration file by navigating to your `~/chef-repo/.chef` directory and creating a file named `config.rb` using your preferred text editor. + +1. Copy the following configuration into the `config.rb` file: + + {{< file "~/chef-repo/.chef/config.rb" ruby >}} current_dir = File.dirname(__FILE__) log_level :info log_location STDOUT @@ -170,11 +210,11 @@ cookbook_path ["#{current_dir}/../cookbooks"] 1. Change the following: - - The value for `node_name` should be the username that was created on the chef server. + - The value for `node_name` should be the username that was created on the Chef server. - Change `USER.pem` under `client_key` to reflect your `.pem` file for your user. - The `validation_client_name` should be your organization's `ORG_NAME` followed by `-validator`. - `ORGANIZATION-validator.pem` in the `validation_key` path should be set to the `ORG_NAME` followed by `-validator.pem`. - - Finally the `chef_server_url` needs to contain the IP address or FQDN of your Chef server, with your `ORG_NAME`. + - Finally the `chef_server_url` should be the Chef server's domain with `/organizations/ORG_NAME` appended. Be sure to replace `ORG_NAME` with your own organization's name. 1. Move to the `chef-repo` directory and copy the needed SSL certificates from the server: @@ -185,17 +225,21 @@ cookbook_path ["#{current_dir}/../cookbooks"] The SSL certificates are generated during the installation of the Chef server. These certificates are self-signed, which means there isn’t a signing certificate authority (CA) to verify. The Chef server's hostname and FQDN should be the same so that the workstation can fetch and verify the SSL certificates. You can verify the Chef server's hostname and FQDN by running `hostname` and `hostname -f`, respectively. Consult the [Chef documentation](https://docs.chef.io/server_security.html#regenerate-certificates) for details on regenerating SSL certificates. {{}} -1. Confirm that `knife.rb` is set up correctly by running the client list: +1. Confirm that `config.rb` is set up correctly by running the client list: knife client list This command should output the validator name. -With both the server and a workstation configured, it is possible to bootstrap your first node. +Now that your Chef server and workstation are configured, you can bootstrap your first node. ## Bootstrap a Node -Bootstrapping a client node installs the Chef client and validates the node, allowing it to read from the Chef server and make any needed configuration changes picked up by the chef-client in the future. +Bootstrapping a node installs the Chef client on the node and validates the node. This allows the node to read from the Chef server and pull down and apply any needed configuration updates detected by the chef-client. + +{{< note >}} +If you encounter any `401 Unauthorized` errors ensure that your `ORGANIZATION.pem` file has `700` permissions. See [Chef's troubleshooting](https://docs.chef.io/errors.html) guide for further information on diagnosing authentication errors. +{{}} 1. From your *workstation*, navigate to your `~/chef-repo/.chef` directory: @@ -215,17 +259,26 @@ Bootstrapping a client node installs the Chef client and validates the node, all knife bootstrap 192.0.2.0 --ssh-user username --sudo --identity-file ~/.ssh/id_rsa.pub --node-name hostname -2. Confirm that the node has been bootstrapped by listing the client nodes: +1. Confirm that the node has been bootstrapped by listing the client nodes: knife client list Your new client node should be included in the list. +1. Add the bootstrapped node to your workstation's `/etc/hosts` file. Replace `node-hostname` with the hostname you just assigned to the node when it was bootstrapped: + + {{< file "/etc/hosts">}} +127.0.0.1 localhost +192.0.2.0 workstation +198.51.100.0 node-hostname +... + {{}} + ## Download a Cookbook (Optional) -When using Chef you will want the Chef client to periodically run on your nodes and pull in any changes pushed to the Chef server. You will also want the `validation.pem` file that is uploaded to your node upon bootstrap to be deleted for security purposes. While these steps can be performed manually, it is often easier and more efficient to set it up as a cookbook. +When using Chef, the Chef client should periodically run on your nodes and pull down any changes pushed to the Chef server from your workstation. You will also want the `validation.pem` file that is uploaded to your node upon bootstrap to be deleted for security purposes. While these steps can be performed manually, it is often easier and more efficient to set them up as a cookbook. -This section is optional, but provides instructions on downloading a cookbook to your workstation, pushing it to a server, and includes the skeleton of a basic cookbook to expand and experiment with. +This section is optional, but provides instructions on downloading a cookbook to your workstation and pushing it to a server, and includes the skeleton of a basic cookbook to expand on and experiment with. 1. From your *workstation*, navigate to your `~/chef-repo/.chef` directory: @@ -260,7 +313,7 @@ end `file "/etc/chef/validation.pem" do` calls to the `validation.pem` file. The `action` defines that the file should be removed (`:delete`). - These are two very basic sets of code in Ruby, and provide an example of the code structure that will be used when creating Chef cookbooks. These examples can be edited and expanded as needed. + These are two very basic sets of code written in Ruby that provide an example of the code structure that will be used when creating Chef cookbooks. These examples can be edited and expanded as needed. 1. Add the recipe to your node's run list, replacing `nodename` with your node's name: @@ -272,10 +325,10 @@ end This command is also used when updating cookbooks. -1. Switch to your *bootstrapped* node(s) and run the initial `chef-client` command: +1. Use `knife-ssh` to run the `chef-client` command on your node. Replace `nodename` with your node's name. If you have set up your node with a limited user account, replace `-x root` with the correct username, i.e. `-x username`. - chef-client + knife ssh 'name:nodename' 'sudo chef-client' -x root - If running the node as a non-root user, append the above command with `sudo`. + The recipes in the run list will be pulled from the server and run on the node. In this instance, it will be the `cron-delvalidate` recipe. This recipe ensures that any cookbooks pushed to the Chef Server, and added to the node's run list will be pulled down to bootstrapped nodes once an hour. This automated step eliminates connecting to the node in the future to pull down changes. - The recipes in the run list will be pulled from the server and run. In this instance, it will be the `cron-delvalidate` recipe. This recipe ensures that any cookbooks made, pushed to the Chef Server, and added to the node's run list will be pulled down to bootstrapped nodes once an hour. This automated step eliminates connecting to the node in the future to pull down changes. +Now that you know how to download Chef cookbooks, add the cookbook to a node's run list, and apply the configurations on a node, consider securing your Chef managed node with the following cookbooks: [Users](https://supermarket.chef.io/cookbooks/users) and [Firewall](https://supermarket.chef.io/cookbooks/firewall). diff --git a/docs/applications/configuration-management/install-and-configure-puppet/index.md b/docs/applications/configuration-management/install-and-configure-puppet/index.md index 1b812dd26dc..11d243cdd68 100644 --- a/docs/applications/configuration-management/install-and-configure-puppet/index.md +++ b/docs/applications/configuration-management/install-and-configure-puppet/index.md @@ -6,9 +6,11 @@ description: 'Basic instructions to set up and configure a Puppet master and age keywords: ["puppet installation", "configuration change management", "server automation"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' aliases: ['websites/puppet/basic-puppet-setup-and-configuration/','websites/puppet/manage-and-automate-systems-configuration-with-puppet/','applications/puppet/set-up-puppet-master-agent/','applications/puppet/install-and-configure-puppet/'] -modified: 2017-03-08 +deprecated: true +deprecated_link: 'applications/configuration-management/getting-started-with-puppet-6-1-basic-installation-and-setup/' +modified: 2019-01-15 modified_by: - name: Phil Zona + name: Linode published: 2015-09-17 title: Install and Configure Puppet external_resources: diff --git a/docs/applications/configuration-management/install-and-manage-mysql-databases-with-puppet-hiera-on-ubuntu-16-04/index.md b/docs/applications/configuration-management/install-and-manage-mysql-databases-with-puppet-hiera-on-ubuntu-16-04/index.md index bb81d4f4ad4..1898a3cc83f 100644 --- a/docs/applications/configuration-management/install-and-manage-mysql-databases-with-puppet-hiera-on-ubuntu-16-04/index.md +++ b/docs/applications/configuration-management/install-and-manage-mysql-databases-with-puppet-hiera-on-ubuntu-16-04/index.md @@ -19,6 +19,8 @@ external_resources: - '[The Puppet Forge](https://forge.puppet.com/)' - '[Hiera documentation](https://docs.puppet.com/hiera/)' - '[Facter](https://docs.puppet.com/facter/)' +deprecated: true +deprecated_link: 'applications/configuration-management/install-and-manage-mysql-databases-with-puppet-hiera-on-ubuntu-18-04/' --- diff --git a/docs/applications/configuration-management/install-and-manage-mysql-databases-with-puppet-hiera-on-ubuntu-18-04/index.md b/docs/applications/configuration-management/install-and-manage-mysql-databases-with-puppet-hiera-on-ubuntu-18-04/index.md new file mode 100644 index 00000000000..75389da8325 --- /dev/null +++ b/docs/applications/configuration-management/install-and-manage-mysql-databases-with-puppet-hiera-on-ubuntu-18-04/index.md @@ -0,0 +1,245 @@ +--- +author: + name: Linode Community + email: docs@linode.com +description: 'Use this guide to install Puppet with MySQL modules and Puppet Hiera configuration manifests to manage MySQL in a variety of environments.' +keywords: ["puppet installation", "configuration change management", "server automation", "mysql", "database", "hiera"] +license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' +published: 2019-01-15 +modified: 2019-01-15 +modified_by: + name: Linode +title: Install and Manage MySQL Databases with Puppet Hiera on Ubuntu 18.04 +contributor: + name: Linode +external_resources: + - '[Puppet Labs](https://puppetlabs.com/)' + - '[Puppet Open Source Documentation](https://docs.puppetlabs.com/puppet/)' + - '[The Puppet Forge](https://forge.puppet.com/)' + - '[Hiera documentation](https://docs.puppet.com/hiera/)' + - '[Facter](https://docs.puppet.com/facter/)' +--- + +[Puppet](https://puppetlabs.com/) is a configuration management system that helps simplify the use and deployment of different types of software, making system administration more reliable and replicable. In this guide, we use Puppet to manage an installation of [MySQL](https://www.mysql.com/), a popular relational database used for applications such as WordPress, Ruby on Rails, and others. [Hiera](https://docs.puppet.com/hiera/) is a method of defining configuration values that Puppet will use to simplify MySQL configuration. + +In this guide, you'll use Puppet to deploy [modules](https://docs.puppet.com/puppet/latest/modules_fundamentals.html) on your server. At the end, you will have MySQL installed, configured, and ready to use for a variety of applications that require a database backend. + +{{< note >}} +This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you're not familiar with the `sudo` command, see the [Users and Groups](/docs/tools-reference/linux-users-and-groups/) guide. +{{< /note >}} + +## Before You Begin + +1. A [Linode 1GB](https://www.linode.com/pricing) plan should be sufficient to run MySQL. Consider using a larger plan if you plan to use MySQL heavily, or for more than just a simple personal website. + +2. Familiarize yourself with our [Getting Started](/docs/getting-started/) guide and complete the steps for setting your Linode's hostname and timezone. + +3. This guide will use `sudo` wherever possible. Complete the sections of our [Securing Your Server](/docs/security/securing-your-server/) to create a standard user account, harden SSH access and remove unnecessary network services. + +4. Update your system: + + sudo apt-get update && sudo apt-get upgrade + +## Install and Configure Puppet + +Follow these steps to set up Puppet for single-host, local-only deployment. If you need to configure more than one server or to deploy a Puppet master, follow our [multi-server Puppet guide](/docs/applications/configuration-management/install-and-configure-puppet/). + +### Install the Puppet Package + +1. Install the `puppetlabs-release-bionic` repository to add the Puppet packages: + + wget https://apt.puppetlabs.com/puppet-release-bionic.deb + sudo dpkg -i puppet-release-bionic.deb + +2. Update the apt package index to make the Puppet Labs repository packages available, then install Puppet. This will install the `puppet-agent` package, which provides the `puppet` executable within in a compatible Ruby environment: + + sudo apt update && sudo apt install puppet-agent + +3. Confirm the version of Puppet installed: + + puppet --version + + At the time of writing, the Puppet version is `6.1.0`. + +### Install the Puppet MySQL Module + +[Puppet Forge](https://forge.puppet.com/) is a collection of _modules_ that aid in the installation of different types of software. The [MySQL module](https://forge.puppet.com/puppetlabs/mysql) handles the installation and configuration of MySQL without you needing to manage various configuration files and services by hand. + +1. Install the MySQL module: + + sudo puppet module install puppetlabs-mysql --version 7.0.0 + + This will install the `mysql` module into the default path: `/etc/puppetlabs/code/environments/production/modules/`. + +### Puppet MySQL Manifest + +This guide uses a Puppet *manifest* to provide Puppet with installation and configuration instructions. Alternatively, you can configure [a Puppet master](/docs/applications/configuration-management/install-and-configure-puppet/). + +While the entirety of a Puppet *manifest* can contain the desired configuration for a host, values for Puppet *classes* or *types* can also be defined in a Hiera configuration file to simplify writing Puppet manifests in most cases. In this example, the `mysql::server` class parameters will be defined in Hiera, but the class must first be applied to the host. + +To apply the `mysql::server` class to all hosts by default, create the following Puppet manifest: + +{{< file "/etc/puppetlabs/code/environments/production/manifests/site.pp" puppet >}} +include ::mysql::server +{{< /file >}} + + +Note that `site.pp` is the default manifest file. Without a qualifying `node { .. }` line, this applies the class to any host applying the manifest. Puppet now knows to apply the `mysql::server` class, but still needs values for resources like databases, users, and other settings. Configure Hiera to provide these values in the next section. + +## Install and Configure Puppet Hiera + +To understand how Hiera works, consider this excerpt from the default `hiera.yaml` file: + +{{< file "/etc/puppetlabs/code/environments/production/hiera.yaml" yaml >}} +--- +version: 5 +hierarchy: + - name: "Per-node data" + path: "nodes/%{::trusted.certname}.yaml" + - name: "Common data" + path: "common.yaml" +{{< /file >}} + + +This Hiera configuration instructs Puppet to accept variable values from `nodes/%{::trusted.certname}.yaml`. If your Linode's hostname is `examplehostname`, define a file called `nodes/examplehostname.yaml`). Any variables found in YAML files higher in the hierarchy are preferred, while any variable names that do not exist in those files will fall-through to files lower in the hierarchy (in this example, `common.yaml`). + +The following configuration will define Puppet variables in `common.yaml` to inject variables into the `mysql::server` class. + +### Initial Hiera Configuration + +Hiera configuration files are formatted as yaml, with keys defining the Puppet parameters to inject their associated values. To get started, set the MySQL root password. The following example of a Puppet manifest is one way to control this password: + +{{< file "example.pp" >}} +class { '::mysql::server': + root_password => 'examplepassword', +} +{{< /file >}} + +We can also define the root password with the following Hiera configuration file. Create the following YAML file and note how the `root_password` parameter is defined as Hiera yaml: + +{{< file "/etc/puppetlabs/code/environments/production/data/common.yaml" >}} +mysql::server::root_password: examplepassword +{{< /file >}} + +Replace `examplepassword` with the secure password of your choice. Run Puppet to set up MySQL with default settings and the chosen root password: + + sudo -i puppet apply /etc/puppetlabs/code/environments/production/manifests/site.pp + +Puppet will output its progress before completing. To confirm MySQL has been configured properly, run a command: + + mysql -u root -p -e 'select version();' + +Enter the password and MySQL returns its version: + + +-------------------------+ + | version() | + +-------------------------+ + | 5.7.24-0ubuntu0.18.04.1 | + +-------------------------+ + +### Define MySQL Resources + +Using Hiera, we can define the rest of the MySQL configuration entirely in yaml. The following steps will create a database and user for use in a WordPress installation. + +1. Create a pre-hashed MySQL password. Replace the password `wordpresspassword` in this example, and when prompted for a the root MySQL password, use the first root password chosen in the previous section to authenticate. Note the string starting with a `*` that the command returns for Step 2: + + mysql -u root -p -NBe 'select password("wordpresspassword")' + *E62D3F829F44A91CC231C76347712772B3B9DABC + +2. With the MySQL password hash ready, we can define Hiera values. The following YAML defines parameters to create a database called `wordpress` and a user named `wpuser` that has permission to connect from `localhost`. The YAML also defines a `GRANT` allowing `wpuser` to operate on the `wordpress` database with `ALL` permissions: + + {{< file "/etc/puppetlabs/code/environments/production/data/common.yaml" yaml >}} +mysql::server::root_password: examplepassword +mysql::server::databases: + wordpress: + ensure: present +mysql::server::users: + wpuser@localhost: + ensure: present + password_hash: '*E62D3F829F44A91CC231C76347712772B3B9DABC' +mysql::server::grants: + wpuser@localhost/wordpress.*: + ensure: present + privileges: ALL + table: wordpress.* + user: wpuser@localhost + +{{< /file >}} + + +1. Re-run Puppet: + + sudo -i puppet apply /etc/puppetlabs/code/environments/production/manifests/site.pp + +2. The `wpuser` should now be able to connect to the `wordpress` database. To verify, connect to the MySQL daemon as the user `wpuser` to the `wordpress` database: + + mysql -u wpuser -p wordpress + + After you enter the password for `wpuser`, exit the MySQL prompt: + + exit + +### Add Hierarchies for Specific Environments + +Additional configurations can be added that will only be applied to specific environments. For example, backup jobs may only be applied for hosts in a certain region, or specific databases can be created in a particular deployment. + +In the following example, Puppet will configure the MySQL server with one additional database, but only if that server's distribution is Debian-based. + +1. Modify `hiera.yaml` to contain the following: + + {{< file "/etc/puppetlabs/code/environments/production/hiera.yaml" yaml >}} +--- +version: 5 +hierarchy: + - name: "Per OS Family" + path: "os/%{facts.os.family}.yaml" + - name: "Other YAML hierarchy levels" + paths: + - "common.yaml" +{{< /file >}} + + + This change instructs Hiera to look for Puppet parameters first in `"os/%{facts.os.family}.yaml"` and then in `common.yaml`. The first, fact-based element of the hierarchy is dynamic, and dependent upon the host that Puppet and Hiera control. In this Ubuntu-based example, Hiera will look for `Debian.yaml` in the `os` folder, while on a distribution such as CentOS, the file `RedHat.yaml` will automatically be referenced instead. + +1. Create the following YAML file: + + {{< file "/etc/puppetlabs/code/environments/production/data/os/Debian.yaml" yaml >}} +lookup_options: + mysql::server::databases: + merge: deep + +mysql::server::databases: + ubuntu-backup: + ensure: present + +{{< /file >}} + + + Though similar to the `common.yaml` file defined in previous steps, this file will add the `ubuntu-backup` database *only* on Debian-based hosts (like Ubuntu). In addition, the `lookup_options` setting ensures that the `mysql::server:databases` parameter is *merged* between `Debian.yaml` and `common.yaml` so that all databases are managed. Without `lookup_options` set to deeply merge these hashes, only the most specific hierarchy file will be applied to the host, in this case, `Debian.yaml`. + + * Alternatively, because our Puppet manifest is short, we can test the same command using the `-e` flag to apply an inline manifest: + + sudo -i puppet apply -e 'include ::mysql::server' + +3. Run Puppet and observe the changes: + + sudo -i puppet apply /etc/puppetlabs/code/environments/production/manifests/site.pp + +4. Verify that the new database exists: + + mysql -u root -p -e 'show databases;' + + This includes the new `ubuntu-backup` database: + + +---------------------+ + | Database | + +---------------------+ + | information_schema | + | mysql | + | performance_schema | + | sys | + | ubuntu-backup | + | wordpress | + +---------------------+ + +Congratulations! You can now control your Puppet configuration via highly configurable Hiera definitions. diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/Laravel_forge_deploy.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/Laravel_forge_deploy.png index f61feb65e5c..4bb7e91d7b7 100644 Binary files a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/Laravel_forge_deploy.png and b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/Laravel_forge_deploy.png differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/active_servers.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/active_servers.png index 92d19afabe3..4e09dafe98b 100644 Binary files a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/active_servers.png and b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/active_servers.png differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/active_sites.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/active_sites.png new file mode 100644 index 00000000000..9b412683525 Binary files /dev/null and b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/active_sites.png differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/add_new_site.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/add_new_site.png new file mode 100644 index 00000000000..734b732ba68 Binary files /dev/null and b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/add_new_site.png differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/adding_api_key_to_forge.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/adding_api_key_to_forge.png index 997c4b05a0f..1c707cea4de 100644 Binary files a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/adding_api_key_to_forge.png and b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/adding_api_key_to_forge.png differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/adding_git_repository.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/adding_git_repository.png index 8fb91aa0c35..28fda6fa1d7 100644 Binary files a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/adding_git_repository.png and b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/adding_git_repository.png differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/adding_new_domain.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/adding_new_domain.png deleted file mode 100644 index 2d2fbadf4b7..00000000000 Binary files a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/adding_new_domain.png and /dev/null differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/adding_repository.gif b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/adding_repository.gif deleted file mode 100644 index 2fd410d49b9..00000000000 Binary files a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/adding_repository.gif and /dev/null differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/apps_tab.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/apps_tab.png new file mode 100644 index 00000000000..0601a4a2716 Binary files /dev/null and b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/apps_tab.png differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/create_a_server.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/create_a_server.png index a334dae2767..87cfb5216ae 100644 Binary files a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/create_a_server.png and b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/create_a_server.png differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/git_repository_name.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/git_repository_name.png deleted file mode 100644 index 4a58bf7e981..00000000000 Binary files a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/git_repository_name.png and /dev/null differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/github_authorization.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/github_authorization.png new file mode 100644 index 00000000000..bb750778f03 Binary files /dev/null and b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/github_authorization.png differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/index.md b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/index.md index 0fe4db4d345..f4d724f89d4 100644 --- a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/index.md +++ b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/index.md @@ -5,9 +5,9 @@ author: description: 'This guide shows how to use Laravel Forge to automate deployment of your PHP projects on a Linode.' keywords: ["content management", "web-server automation", "laravel", "php", "wordpress", "drupal", "cms", "joomla", "Laravel Forge"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -modified: 2017-09-20 +modified: 2019-01-18 modified_by: - name: Onwuka Gideon + name: Linode published: 2017-09-18 title: 'Use Laravel Forge to Automate Web-Server Creation on a Linode' external_resources: @@ -18,147 +18,165 @@ external_resources: ## What is Laravel Forge -[Laravel Forge](https://forge.laravel.com) is a tool for deploying and configuring web applications. It was developed by the makers of the [Laravel framework](https://laravel.com), but can be used to automate the deployment of any web application that uses a PHP server. - -Creating a fully-functioning web server is normally a complicated and time-consuming process, involving the installation of multiple components such as Nginx, MySQL, and PHP. Laravel Forge automates all of the necessary installation and configuration, allowing you to get your website up and running quickly. +[Laravel Forge](https://forge.laravel.com) is a tool for deploying and configuring web applications. It was developed by the makers of the [Laravel framework](https://laravel.com), but it can be used to automate the deployment of any web application that uses a PHP server. -Once your server has been created, deploying updates becomes as simple and painless as pushing to your repository on Github. Also, you can easily manage the configuration of your website though a web interface. Finally, Forge automatically provides advanced security features, such as free SSL certificates (through [Let's Encrypt](https://letsencrypt.org/)) and automatic firewall configuration. +Creating a fully-functioning web server normally involves the installation of multiple components such as NGINX, MySQL, and PHP. Laravel Forge automates all of the necessary installation and configuration steps, allowing you to get your website up and running quickly. +Once your server has been created, deploying updates becomes as clear and painless as pushing to your repository on GitHub. Also, you can easily manage the configuration of your website though a web interface. Finally, Forge automatically provides advanced security features, such as free SSL certificates (through [Let's Encrypt](https://letsencrypt.org/)) and automatic firewall configuration. ## Before You Begin -1. Sign up for a [Laravel Forge](https://forge.laravel.com/auth/register) account if you do not have one. +1. Sign up for a [Laravel Forge](https://forge.laravel.com/auth/register) account if you don't have one. -2. This guide will require a [Linode](https://www.linode.com/) account and an API key. For more detailed instructions, refer to [this guide](/docs/platform/api/api-key/). +1. Create a Linode API key, which Laravel Forge will use to interface with your account. [Forge uses Linode's new APIv4](https://blog.laravel.com/forge-switching-to-the-new-linode-manager), and APIv4 tokens are created in the [Linode Cloud Manager](https://cloud.linode.com). Refer to the [Getting Started with the Linode API](/docs/platform/api/getting-started-with-the-linode-api-new-manager/#get-an-access-token) to learn how to create your key. -## Link to a Source Control Service +1. If you don't have a registered domain name for your website, purchase one from a domain name registrar. -If you want to be able to quickly deploy from Github, Gitlab or Bitbucket, you will need to link these sites to your Forge account. From your Forge dashboard, select the **Your Profile** tab. + {{< note >}} +You will be able to set up a site without a domain name (by visiting your Linode's IP address directly), but you will only be able to use SSL with a domain. +{{< /note >}} -1. Choose your preferred source control by clicking on the **Source Control** link on the Left. +## Set Up your Forge Account - ![Linode API](linking_source_control.png) +### Link to a Source Control Service -2. Click on the source control service you would like to connect to. +If you want to be able to quickly deploy from GitHub, GitLab, or Bitbucket, you will need to link these sites to your Forge account. - Once you link to a source control service, you will be prompted to authorize Laravel Forge. You will then be redirected back to Forge’s website. +1. From the top navigation menu of the Laravel Forge website, click on your username and then choose the **My Account** option: - ![Linode API](source_control_authorized.png) + ![My Account Menu](my_account_menu.png "The My Account menu") -## Adding Your Linode API Key to Forge +1. Navigate to the **Source Control** section: -1. Open your Laravel Forge Dashboard. + ![Linking to Source Control](linking_source_control.png "Linking to source control") -2. Click on **Service Providers**. +1. Choose your source control provider. Your browser will navigate to the source control provider's website, where an authorization prompt will appear. -3. Select **Linode** and input your API key. + ![GitHub Authorization](github_authorization.png "GitHub's authorization form") - ![Linode API](adding_api_key_to_forge.png) +1. Confirm this authorization request. You will be returned to the Laravel Forge website. -## Create a Server +### Adding your Linode API Key to Forge -1. Click on **Linode**. +1. From the **My Account** page, navigate to the **Server Providers** section. Select **Linode Cloud**: - ![Linode API](create_a_server.png) + ![Linode Cloud Provider](adding_api_key_to_forge.png "Adding your Linode Cloud APIv4 token") - You will then be provided with some options: +1. Enter a profile name. This can be your Linode username or it can be any other string that will help you identify the profile. Then, enter your APIv4 key and click the **Add Credential** button. - **Credentials:** Any cloud providers that you have linked to your account. +## Create a Server - **Name:** A name for your server. You can choose any name you like. +1. Click on the **Forge** icon in the top left navigation menu. Then click on the **Linode** provider button. - **Server Size:** The size of the server. +1. Fill out the form that appears: - **Region:** The data center where you want your server hosted. Choose a location close to where you expect the majority of users to be. + ![Create a Server](create_a_server.png "The form for creating a server in Forge") - **PHP Version:** The installed PHP version. + | **Option**                       | **Description** | + | ------ | ----------- | + | Credentials | Any of the Linode accounts that you have linked to your Laravel Forge account. | + | Name | A name for your server. Laravel Forge auto-generates a random name, but you can edit it. | + | Region | The data center where you want your server hosted. Choose a location close to where you expect the majority of users to be. | + | Server Size | The hardware resource plan for your server. Plans with more CPU and memory can serve more connections to your sites, and a larger storage capacity can hold bigger databases. | + | PHP Version | The PHP version to install. | + | Post-Provision Recipe | [Actions](https://medium.com/@taylorotwell/post-provision-recipes-on-forge-634ccb189847) that should be taken after the server is provisioned. | + | Database | The database package to install. | + | Database Name | Your application's database name. By default it'll be named `forge`. | - **Database Name:** If your application needs a database, then you can name it here. By default it will be named `forge`. +1. Once you have finished selecting options, click **Create Server**. A pop-up dialog will show you the sudo and database passwords that have been automatically generated for you. Be sure to copy these values and store them in a secure place. -2. Once you have finished selecting options, click **Create Server**. A pop up will show you the sudo and database passwords that have been automatically generated for you. Be sure to copy these values and store them in a secure place. + ![Server Credential Pop-up Dialog](server_credential.png "The pop-up dialog which displays your new server's credentials") - ![Linode API](server_credential.png) +1. Forge will now perform the steps necessary to create and configure a Linode based on the settings you provided. The new server will appear in the **Active Servers** section, and a list of recent events representing the server's configuration will appear below it. - Forge will now perform the steps necessary to create and configure a Linode based on the settings you chose earlier. + ![Active Servers](active_servers.png "The active servers panel") -3. A green check in the **Status** column means the server is active. This will take a few seconds. +1. When the server has been fully provisioned the **Status** of the server will be **Active**. Navigate to the public IP address of your new Linode in a browser, and you will see the PHP settings active for the server: - ![Linode API](active_servers.png) + ![PHP Settings Page](server_set_up_with_php.png "The PHP settings page of your new server") -4. Navigate to the public IP address of your Linode in a browser. +1. When the setup process has completed you will also receive an email containing details about your new server: - ![Linode API](server_set_up_with_php_7_1.png) + ![Email Confirmation](mail_from_forge_showing_details_created.png "An email confirmation for your new Forge-created server") -5. When the setup process has completed, you will receive an email containing details about your new server. +## Set Up your Site - ![Linode API](mail_from_linode_showing_details_created.png) +Your server has been created, but no sites have been set up on it, apart from the default site which displays your PHP settings. -## Add a Domain Name +{{< note >}} +If you do not want to use a domain with your website, you can configure the *default* site on your server. Select the default site from the **Active Sites** panel of your server's dashboard in Forge, then skip to the [Add a repository](#add-a-repository) instructions. +{{< /note >}} -If you want to use Let's Encrypt to obtain a free SSL certificate for your website, you will need to buy a Fully Qualified Domain Name (FQDN) and set it to point to your new server. +### Add a New Site -1. Head back to Laravel Forge account then click on the active server. +1. Set up your DNS records for your domain. [Create a Domain Zone and an *A record*](/docs/platform/manager/dns-manager-new-manager/#add-a-domain-zone) assigned to your Linode's IP address. If you use Linode's name servers, review the [DNS Manager](/docs/platform/manager/dns-manager-new-manager/) guide for instructions. -2. Go to your domain name provider, add an ANAME record for the domain or subdomain, and point it to the public IP address of your server. + If you use another DNS provider, check their documentation for instructions. + {{< content "update-dns-at-common-name-server-authorities" >}} - ![Linode API](adding_new_domain.png) +1. From the **Servers** menu in the top navigation bar, choose your new server. If you don't see this menu yet, refresh your browser window: - - Root Domain: This is your domain name. - - Project Type: This is the type of project you are using. If you are building regular PHP, you can choose `General PHP/Laravel`. Other options include Static, HTML, Symfony, and Symfony (Dev). - - Web Directory: This is the directory from which public files will be served. + ![Servers Menu](navigation-menu-server-drop-down.png "The servers drop-down menu") -3. Submit the form by clicking **ADD SITE**. +1. Fill out the **New Site** form and then click the **Add Site** button: - {{< note >}} -If you do not add a domain name, you can still access your server through its public IP address. -{{< /note >}} + ![New Site Form](add_new_site.png "The form for creating a new site on your server") -## Add a Repository + | **Option**               | **Description** | + | ------ | ----------- | + | Root Domain | Your website's domain. | + | Project Type | Your project type. If you are building regular PHP, you can choose `General PHP/Laravel`. Other options include Static, HTML, Symfony, and Symfony (Dev). | + | Web Directory | The directory from which public files will be served. You will need to make sure your website's files are in this directory in your source code repository. | -1. Click on your active servers. Under the active site, choose the domain name you added in the previous step. +1. Your new site will appear below the form in the **Active Sites** panel: - ![Linode API](adding_repository.gif) + ![Active Sites](active_sites.png "The active sites panel") -2. Click **Apps** on the left-hand side and then select your Git repository. +### Add a Repository - ![Linode API](adding_git_repository.png) +1. From the **Active Sites** panel, click on your site. The **Apps** section will appear: -3. Fill in the repository field with your username and repository. Then select the branch you want to use. Click on **INSTALL REPOSITORY.** Observe that the repository field is `gitlab_username/repository_name_without_.git`. The sample image below shows `dongido/transaction-manager`: + ![Apps Tab](apps_tab.png "The apps section for your site") - ![Linode API](git_repository_name.png) +1. Select the **Git Repository** option and fill out the form that appears. The format for the **Repository** field should be `source_control_username/repository_name`. Laravel Forge will pull your code from the branch that you enter. -4. If you visit the website again, the repository is now copied to your Linode server and everything is set up. + ![Git Repository Form](adding_git_repository.png "The form for adding a Git repository to your site") - ![Linode API](site_up_and_running.png) + {{< note >}} +If you do not use [Composer](https://getcomposer.org), leave the **Install Composer Dependencies** option disabled, as it will cause errors for your deployment if enabled. +{{< /note >}} -## Quick Deploy -These steps will show how to immediately deploy changes from the source directory to the live site. +1. Click on the **Install Repository** button. Forge will copy your source code to the server. When this finishes, visit your domain name in your browser, and you should see your site's contents. -1. From the **INSTALL REPOSITORY** menu, toggle **Quick Deploy** to "on". +1. If your site deployment runs into any errors, a **Server Alerts** panel will appear. Click on the information button in this panel to review the errors in detail. - ![Linode API](Laravel_forge_deploy.png) + ![Server Alerts](server-alerts.png "The server alerts panel") -2. Make changes to your project on Github or Gitlab and commit them to the master branch. Your live site will be updated to reflect the changes. +1. Verify that your site displays your repository's latest changes by navigating to your site's URL. If you did not register a Domain, then navigate to your site's IP address. - ![Linode API](laravel_forge_sites.png) +### Quick Deploy -## Adding SSL to Your Domain Name -SSL (Secure Sockets Layer) is the standard security technology for establishing an encrypted link between a web server and a browser. To add SSL: +Forge can automatically deploy updates to your server whenever you push new code to your application's repository. -1. Choose your server on Forge. +1. From the **Apps** section of your site's dashboard in Forge, click **Enable Quick Deploy**: -2. Select the domain name of your server. + ![Quick Deploy](Laravel_forge_deploy.png "The quick deploy option") -3. Click on **SSL** from the menu on the left. +1. Make changes to your project's code and push them to your repository service. Your live site will be updated to reflect the changes. - ![Linode API](laravel_forge_adding_ssl.png) +### Adding SSL to your Domain Name - Laravel Forge comes with Let's Encrypt which help us generate a free SSL certificate. +SSL (Secure Sockets Layer) is the standard security technology for establishing an encrypted link between a web server and a browser. To add SSL: + +1. Navigate to the **SSL** section of your site's dashboard in Forge. -4. If you already have an SSL certificate, click on the Install Existing Certificate. Otherwise, select **LetsEncrypt (Beta)**. +1. If you already have an SSL certificate, click on **Install Existing Certificate**. Otherwise, select the **Let's Encrypt** option: -5. If you chose to use LetsEncrypt, you will be presented with a button to obtain a certificate. Click on it and allow it to install. + ![Let's Encrypt Form](laravel_forge_adding_ssl.png "The form for adding Let's Encrypt to your site") -6. Activate your new certificate by clicking on the **activate** icon. + {{< note >}} +[Let's Encrypt](https://letsencrypt.org) is a free and widely-trusted SSL certificate authority. +{{< /note >}} +1. If you choose to use Let's Encrypt, verify that the domains you want to secure are listed correctly and click **Obtain Certificate**. diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/laravel_forge_adding_ssl.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/laravel_forge_adding_ssl.png index 882270ea306..41827a5956c 100644 Binary files a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/laravel_forge_adding_ssl.png and b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/laravel_forge_adding_ssl.png differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/laravel_forge_sites.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/laravel_forge_sites.png deleted file mode 100644 index 3804b0738fe..00000000000 Binary files a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/laravel_forge_sites.png and /dev/null differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/linking_source_control.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/linking_source_control.png index 35ad36e9ccc..46016ff87fb 100644 Binary files a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/linking_source_control.png and b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/linking_source_control.png differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/mail_from_forge_showing_details_created.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/mail_from_forge_showing_details_created.png new file mode 100644 index 00000000000..24d2b644b23 Binary files /dev/null and b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/mail_from_forge_showing_details_created.png differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/mail_from_linode_showing_details_created.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/mail_from_linode_showing_details_created.png deleted file mode 100644 index 843d5a01d04..00000000000 Binary files a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/mail_from_linode_showing_details_created.png and /dev/null differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/my_account_menu.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/my_account_menu.png new file mode 100644 index 00000000000..95d0fc85864 Binary files /dev/null and b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/my_account_menu.png differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/navigation-menu-server-drop-down.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/navigation-menu-server-drop-down.png new file mode 100644 index 00000000000..43bfdac49a9 Binary files /dev/null and b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/navigation-menu-server-drop-down.png differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/server-alerts.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/server-alerts.png new file mode 100644 index 00000000000..bc138fbbf60 Binary files /dev/null and b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/server-alerts.png differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/server_credential.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/server_credential.png index a725996bd9a..c6663a08bdc 100644 Binary files a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/server_credential.png and b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/server_credential.png differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/server_set_up_with_php.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/server_set_up_with_php.png new file mode 100644 index 00000000000..a151c08ca0d Binary files /dev/null and b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/server_set_up_with_php.png differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/server_set_up_with_php_7_1.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/server_set_up_with_php_7_1.png deleted file mode 100644 index 23da5884a4d..00000000000 Binary files a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/server_set_up_with_php_7_1.png and /dev/null differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/site_up_and_running.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/site_up_and_running.png deleted file mode 100644 index fa8105976c0..00000000000 Binary files a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/site_up_and_running.png and /dev/null differ diff --git a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/source_control_authorized.png b/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/source_control_authorized.png deleted file mode 100644 index c9df3420412..00000000000 Binary files a/docs/applications/configuration-management/use-laravel-forge-to-automate-web-server-creation-on-a-linode/source_control_authorized.png and /dev/null differ diff --git a/docs/applications/containers/how-to-deploy-nginx-on-a-kubernetes-cluster/index.md b/docs/applications/containers/how-to-deploy-nginx-on-a-kubernetes-cluster/index.md index 3c33e0c3e71..3e54acbc57b 100644 --- a/docs/applications/containers/how-to-deploy-nginx-on-a-kubernetes-cluster/index.md +++ b/docs/applications/containers/how-to-deploy-nginx-on-a-kubernetes-cluster/index.md @@ -4,7 +4,7 @@ author: description: 'This guide shows how to install Kubernetes on a Linode with CentOS or Ubuntu. Includes a section on how to deploy nginx to the example cluster.' keywords: ["kubernetes","docker","container","deployment","nginx"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -modified: 2018-01-08 +modified: 2019-01-21 modified_by: name: Linode published: 2017-11-27 @@ -24,7 +24,11 @@ external_resources: ## What is Kubernetes? -[Kubernetes](https://kubernetes.io/) is an open-source container management system that is based on [Google Borg](https://research.google.com/pubs/pub43438.html). It can be configured to provide highly available, horizontally autoscaling, automated deployments. This guide shows you how to set up a Kubernetes cluster on a Linode and manage the lifecycle of an NGINX service. +[Kubernetes](https://kubernetes.io/) is an open-source container management system that is based on [Google Borg](https://research.google.com/pubs/pub43438.html). It can be configured to provide highly available, horizontally autoscaling, automated deployments. This guide shows you how to manually set up a Kubernetes cluster on a Linode and manage the lifecycle of an NGINX service. + +{{< note >}} +You can now create a Kubernetes cluster with one command using the Linode CLI. To provision Kubernetes on Linodes, this tool uses the [Linode Kubernetes Terraform module](https://github.com/linode/terraform-linode-k8s), the [Linode Cloud Controller Manager (CCM)](https://github.com/linode/linode-cloud-controller-manager), and the [Container Storage Interface (CSI) Driver](https://github.com/linode/linode-blockstorage-csi-driver) for Linode Block Storage. See the [Kubernetes Tools](https://developers.linode.com/kubernetes/) page for installation steps. For an in-depth dive into the the Linode Kubernetes Terraform module, see its related [Community Site post](https://www.linode.com/community/questions/17611/the-linode-kubernetes-module-for-terraform). +{{}} ## Before You Begin @@ -46,9 +50,43 @@ The steps in this guide create a two-node cluster. Evaluate your own resource re 1. Create two Linodes with at least 2GB memory within the same data center. -2. For each node, go into the Remote Access tab of your Linode Manager and add a [private IP](/docs/networking/remote-access#adding-private-ip-addresses). It is possible to build a Kubernetes cluster using public IPs between data centers, but performance and security may suffer. +1. For each node, go into the Remote Access tab of your Linode Manager and add a [private IP](/docs/networking/remote-access#adding-private-ip-addresses). It is possible to build a Kubernetes cluster using public IPs between data centers, but performance and security may suffer. + +1. Configure a firewall with [UFW](/docs/security/firewalls/configure-firewall-with-ufw/) or [iptables](/docs/security/firewalls/control-network-traffic-with-iptables/) to ensure only the two nodes can communicate with each other. + + When configuring your firewall, a good place to start is to create rules for the ports Kubernetes requires to function. This includes any inbound traffic on Master nodes and their required ports. If you have changed any custom ports, you should ensure those ports are also open. Master Nodes will have a public IP address or `192.168.0.0/16`. See the chart below for more details. + + On Worker nodes, you should allow inbound Kubelet traffic. For NodePort traffic you should allow a large range from the world or if you are using the [Linode NodeBalancers service](https://github.com/linode/linode-cloud-controller-manager) exclusively for ingress, `192.168.255.0/24`. See the chart below for more details. + + The table below provides a list of the required ports for Master nodes and Worker nodes. You should also include port `22`. + + **Master nodes** + + | Protocol | Direction | Port Range | Purpose | Used By | + | -------- | --------- | ---------- | ------- | ------- | + | TCP | Inbound | 6443* | Kubernetes API server | All | + | TCP | Inbound | 2379-2380 | etcd server client API | kube-apiserver, etcd | + | TCP | Inbound | 10250 | Kubelet API | Self, Control plane | + | TCP | Inbound | 10251 | kube-scheduler | Self | + | TCP | Inbound | 10253 | kube-controller-manager | Self | + + **Worker nodes** + + | Protocol | Direction | Port Range | Purpose | Used By | + | -------- | --------- | ---------- | ------- | ------- | + | TCP | Inbound | 10250 | Kubelet API | Self, Control plane | + | TCP | Inbound | 30000-32767 | NodePort Services** | All | + + + {{< note >}} + By design, kube-proxy will always place its iptables chains first. It inserts 2 rules, KUBE-EXTERNAL-SERVICES and KUBE-FIREWALL at the top of the INPUT chain. See the [Kubernetes discussion forum](https://discuss.kubernetes.io/t/custom-iptables-rules-for-input-chain/3509) for more details. + {{}} + +1. You should consider using the Linode NodeBalancer service with the [Linode Cloud Controller Manager (CCM)](https://github.com/linode/linode-cloud-controller-manager). + + - When using Linode NodeBalancers ensure you add iptables rules to allow the NodeBalancer traffic: `192.168.255.0/24`. -3. Configure a firewall with [UFW](/docs/security/firewalls/configure-firewall-with-ufw/) or [iptables](/docs/security/firewalls/control-network-traffic-with-iptables/) to ensure only the two nodes can communicate with each other. +1. To obtain persistent storage capabilities, you can use the [Container Storage Interface (CSI) Driver](https://github.com/linode/linode-blockstorage-csi-driver) for Linode Block Storage. ### Disable Swap Memory @@ -71,7 +109,7 @@ The `/etc/fstab` should look something like this: 1. Delete the line describing the swap partition. In this example, Line 10 with `/dev/sdb`. -2. Disable swap memory usage: +1. Disable swap memory usage: swapoff -a @@ -81,13 +119,13 @@ To make the commands in this guide easier to understand, set up your hostname an 1. Choose a node to designate as your Kubernetes master node and SSH into it. -2. Edit `/etc/hostname`, and add: +1. Edit `/etc/hostname`, and add: {{< file "/etc/hostname" >}} kube-master {{< /file >}} -3. Add the following lines to `/etc/hosts`: +1. Add the following lines to `/etc/hosts`: {{< file "/etc/hosts" >}} kube-master @@ -98,9 +136,9 @@ kube-master To make it easier to understand output and debug issues later, consider naming each hostname according to its role (`kube-worker-1`, `kube-worker-2`, etc.). -4. Perform Steps 2 and 3 on each worker node, changing the values accordingly. +1. Perform Steps 2 and 3 on each worker node, changing the values accordingly. -5. For the changes to take effect, restart your Linodes. +1. For the changes to take effect, restart your Linodes. ### Confirm Hostnames @@ -116,7 +154,7 @@ If you are unable to ping any of your hosts by their hostnames or private IPs: 1. SSH into the host that isn't responding. -2. Enter `ifconfig`. You should see an entry for `eth0:1` that lists your private IP. If `eth0:1` isn't listed, it's possible that you deployed your Linode image before adding a private IP to the underlying host. Recreate the image and return to the beginning of the guide. +1. Enter `ifconfig`. You should see an entry for `eth0:1` that lists your private IP. If `eth0:1` isn't listed, it's possible that you deployed your Linode image before adding a private IP to the underlying host. Recreate the image and return to the beginning of the guide. ## Install Docker and Kubernetes on Linode @@ -186,13 +224,13 @@ as root: kubeadm join --token 921e92.d4582205da623812 :6443 --discovery-token-ca-cert-hash sha256:bd85666b6a97072709b210ddf677245b4d79dab88d61b4a521fc00b0fbcc710c {{< /output >}} -2. On the master node, configure the `kubectl` tool: +1. On the master node, configure the `kubectl` tool: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config -3. Check on the status of the nodes with `kubectl get nodes`. Output will resemble: +1. Check on the status of the nodes with `kubectl get nodes`. Output will resemble: root@kube-master:~# kubectl get nodes name status roles age version @@ -202,11 +240,11 @@ as root: The `--pod-network-cidr` argument used in the [Configure the Kubernetes Master Node](#configure-the-kubernetes-master-node) section defines the network range for the CNI. -5. While still on the master node run the following command to deploy the CNI to your cluster: +1. While still on the master node run the following command to deploy the CNI to your cluster: kubectl apply -f https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml -6. To ensure Calico was set up correctly, use `kubectl get pods --all-namespaces` to view the pods created in the `kube-system` namespace: +1. To ensure Calico was set up correctly, use `kubectl get pods --all-namespaces` to view the pods created in the `kube-system` namespace: root@kube-master:~# kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE @@ -230,7 +268,7 @@ as root: kube-public Active 4h kube-system Active 4h -7. Run `kubectl get nodes` again to see that the master node is now running properly: +1. Run `kubectl get nodes` again to see that the master node is now running properly: root@kube-master:~# kubectl get nodes name status roles age version @@ -242,7 +280,7 @@ as root: kubeadm join --token kube-master:6443 --discovery-token-ca-cert-hash sha256: -2. On the master node, use `kubectl` to see that the slave node is now ready: +1. On the master node, use `kubectl` to see that the slave node is now ready: {{< output >}} root@kube-master:~# kubectl get nodes @@ -259,11 +297,11 @@ A *deployment* is a logical reference to a pod or pods and their configurations. kubectl create deployment nginx --image=nginx -2. This creates a deployment called `nginx`. `kubectl get deployments` lists all available deployments: +1. This creates a deployment called `nginx`. `kubectl get deployments` lists all available deployments: kubectl get deployments -3. Use `kubectl describe deployment nginx` to view more information: +1. Use `kubectl describe deployment nginx` to view more information: {{< output >}} Name: nginx @@ -299,7 +337,7 @@ Events: The `describe` command allows you to interrogate different kubernetes resources such as pods, deployments, and services at a deeper level. The output above indicates that there is a deployment called `nginx` within the default namespace. This deployment has a single replicate, and is running the docker image `nginx`. The ports, mounts, volumes and environmental variable are all unset. -4. Make the NGINX container accessible via the internet: +1. Make the NGINX container accessible via the internet: kubectl create service nodeport nginx --tcp=80:80 @@ -312,15 +350,15 @@ Events: kubernetes ClusterIP 10.96.0.1 443/TCP 5h nginx NodePort 10.98.24.29 80:32555/TCP 52s -5. Verify that the NGINX deployment is successful by using `curl` on the slave node: +1. Verify that the NGINX deployment is successful by using `curl` on the slave node: root@kube-master:~# curl kube-worker-1:32555 The output will show the unrendered "Welcome to nginx!" page HTML. -6. To remove the deployment, use `kubectl delete deployment`: +1. To remove the deployment, use `kubectl delete deployment`: root@kube-master:~# kubectl delete deployment nginx deployment "nginx" deleted root@kube-master:~# kubectl get deployments - No resources found. + No resources found. \ No newline at end of file diff --git a/docs/applications/containers/install-docker-ce/index.md b/docs/applications/containers/install-docker-ce/index.md index 4f50871c033..c19b701de78 100644 --- a/docs/applications/containers/install-docker-ce/index.md +++ b/docs/applications/containers/install-docker-ce/index.md @@ -36,10 +36,12 @@ These steps install Docker Community Edition (CE) using the official Ubuntu repo You should see output similar to the following: - pub 4096R/0EBFCD88 2017-02-22 - Key fingerprint = 9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88 - uid Docker Release (CE deb) - sub 4096R/F273FCD8 2017-02-22 + {{< output >}} +pub 4096R/0EBFCD88 2017-02-22 + Key fingerprint = 9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88 +uid Docker Release (CE deb) +sub 4096R/F273FCD8 2017-02-22 +{{< /output >}} 5. Add the `stable` Docker repository: @@ -52,9 +54,11 @@ These steps install Docker Community Edition (CE) using the official Ubuntu repo 7. Add your limited Linux user account to the `docker` group: - sudo usermod -aG docker exampleuser + sudo usermod -aG docker $USER - You will need to restart your shell session for this change to take effect. + {{< note >}} +After entering the `usermod` command, you will need to close your SSH session and open a new one for this change to take effect. +{{< /note >}} 8. Check that the installation was successful by running the built-in "Hello World" program: diff --git a/docs/applications/containers/manage-a-docker-cluster-with-kubernetes/index.md b/docs/applications/containers/manage-a-docker-cluster-with-kubernetes/index.md index 8dfa2354cc8..5f70a7b4d35 100644 --- a/docs/applications/containers/manage-a-docker-cluster-with-kubernetes/index.md +++ b/docs/applications/containers/manage-a-docker-cluster-with-kubernetes/index.md @@ -27,6 +27,10 @@ external_resources: A Kubernetes cluster consists of at least one master node and several worker nodes. The master node runs the API server, the scheduler and the controller manager, and the actual application is deployed dynamically across the cluster. +{{< note >}} +You can now create a Kubernetes cluster with one command using the Linode CLI. To provision Kubernetes on Linodes, this tool uses the [Linode Kubernetes Terraform module](https://github.com/linode/terraform-linode-k8s), the [Linode Cloud Controller Manager (CCM)](https://github.com/linode/linode-cloud-controller-manager), and the [Container Storage Interface (CSI) Driver](https://github.com/linode/linode-blockstorage-csi-driver) for Linode Block Storage. See the [Kubernetes Tools](https://developers.linode.com/kubernetes/) page for installation steps. For an in-depth dive into the the Linode Kubernetes Terraform module, see its related [Community Site post](https://www.linode.com/community/questions/17611/the-linode-kubernetes-module-for-terraform). +{{}} + ## System Requirements To complete this guide you will need three Linodes running Ubuntu 16.04 LTS, each with at least 4GB of RAM. Before beginning this guide, you should also use the Linode Manager to generate a [private IP address](https://linode.com/docs/networking/remote-access#adding-private-ip-addresses) for each Linode. diff --git a/docs/applications/media-servers/install-plex-media-server-on-centos-7/index.md b/docs/applications/media-servers/install-plex-media-server-on-centos-7/index.md index c38ed73836b..9643ead59cc 100644 --- a/docs/applications/media-servers/install-plex-media-server-on-centos-7/index.md +++ b/docs/applications/media-servers/install-plex-media-server-on-centos-7/index.md @@ -3,114 +3,109 @@ author: name: Nick Brewer email: nbrewer@linode.com description: View and organize your media library with Plex on CentOS 7 -keywords: ["plex media server", "install plex", "plex centos"] +keywords: ["plex", "media", "centos"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -modified: 2017-06-21 +modified: 2019-01-24 modified_by: - name: Nick Brewer + name: Linode published: 2017-05-03 title: Install Plex Media Server on CentOS 7 external_resources: - '[Plex Media Server Documentation](https://support.plex.tv/hc/en-us/categories/200007567-Plex-Media-Server)' --- -[Plex](https://www.plex.tv/) is a feature-rich media library platform that allows you to organize and stream your digital video and audio from virtually anywhere. This guide will show you how to set up the **Plex Media Server** on your Linode running CentOS 7, as well as how to connect to your media server from a Plex [client](https://support.plex.tv/hc/en-us/categories/200006953-Plex-Apps) application. A Plex media server could benefit from large amounts of disk space, so consider using our [Block Storage](/docs/platform/how-to-use-block-storage-with-your-linode) service with this setup. +[Plex](https://www.plex.tv/) is a feature-rich media library platform that allows you to organize and stream your digital video and audio from virtually anywhere. Basic Plex features are [free](https://support.plex.tv/articles/202526943-plex-free-vs-paid/), while the paid Plex Pass adds additional features. + +This guide demonstrates how to set up **Plex Media Server** on a Linode running CentOS 7, and how to connect client devices. A Plex media server could benefit from large amounts of disk space, so consider using Linode's [Block Storage](/docs/platform/how-to-use-block-storage-with-your-linode) service with this setup. ![Install Plex Media Server on CentOS 7](install-plex-media-server-on-centos-7.png) -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Users and Groups](/docs/tools-reference/linux-users-and-groups/) guide. -{{< /note >}} ## Before you Begin -1. Familiarize yourself with our [Getting Started](/docs/getting-started) guide and complete the steps for setting your Linode's hostname and timezone. +- You will need root access to your Linode, or a [limited user account](/docs/security/securing-your-server/#add-a-limited-user-account) with `sudo` privilege. -2. Complete the sections of our [Securing Your Server](/docs/security/securing-your-server/) to create a standard user account with `sudo` privileges. +- Set your system's [hostname](/docs/getting-started/#set-the-hostname) and [time zone](/docs/getting-started/#set-the-timezone). -3. Ensure that your system is up to date, and install `wget`: +- Plex requires that you create an [account](https://www.plex.tv/features/) to make use of the service, and provides additional features such as DVR capability and offline viewing if you pay for their premium [Plex Pass](https://www.plex.tv/features/plex-pass/) offering. To complete this guide, you will need a Plex account. - sudo yum update && sudo yum install wget -4. Plex requires that you create an [account](https://www.plex.tv/features/) to make use of the service, and provides additional features such as DVR capability and offline viewing if you pay for their premium [Plex Pass](https://www.plex.tv/features/plex-pass/) offering. To complete this guide, you will need a Plex account. +## Install and Configure Plex -## Install Plex +1. Go to [Plex's download page](https://www.plex.tv/media-server-downloads/). Select *Linux*, and then click *Choose Distribution*. -1. Head to the Plex [Downloads](https://www.plex.tv/downloads/) page and copy the installation link for CentOS. Use `wget` to download the installer via the copied link: + ![Plex download button](plex-download-button.png) - wget https://downloads.plex.tv/plex-media-server/1.12.3.4973-215c28d86/plexmediaserver-1.12.3.4973-215c28d86.x86_64.rpm +1. In the menu that appears, right-click on *CentOS 64-bit (RPM for CentOS 7 or newer)* and copy the download link. Use `cURL` to download the .rpm package directly to your Linode. The example below uses the current link for CentOS at the time of writing. Be sure you install the most recent version of Plex. - This example uses the current link for CentOS, at the time of writing. Be sure to use the up-to-date version supplied on the Plex website. + curl -O https://downloads.plex.tv/plex-media-server/1.14.1.5488-cc260c476/plexmediaserver-1.14.1.5488-cc260c476.x86_64.rpm -2. Use `yum` to install the Plex server: +1. Update your system and install Plex: + sudo yum update sudo yum install plexmediaserver*.rpm -3. Enable Plex Media Server to start on reboot, and then start the server: - - sudo systemctl enable plexmediaserver.service - sudo systemctl start plexmediaserver.service +1. Enable Plex Media Server to start on reboot and then start the server: + sudo systemctl enable plexmediaserver + sudo systemctl start plexmediaserver -## Configure Plex +1. Finally, you'll create the directories that will store your Plex media. In this example we'll create library directories for `movies` and `television` within a `plex-media` directory. These will be located within your user's `/home`: -This section will show you how to complete your server setup and start adding media libraries. - -### Initial Setup + cd ~/ + mkdir -p plex-media/movies && mkdir plex-media/television -1. Administration of the Plex server is performed via its web interface. Before you can connect to the web interface from your workstation, you will first need to create an SSH tunnel to your Linode. Substitute `user` with the `sudo` user on your Linode, and `192.0.2.1` with the IP address of the Plex server: +1. Administration of the Plex server is performed via its web interface. Before you can connect to the web interface from your workstation, you will first need to create an SSH tunnel to your Linode. Substitute `user` with the `sudo` user on your Linode, and `192.0.2.0` with its IP address. - ssh user@192.0.2.1 -L 8888:localhost:32400 + ssh user@192.0.2.0 -L 8888:localhost:32400 -2. Enter `http://localhost:8888/web` into your browser to view the Plex web interface, as shown below. Input your Plex account username and password to proceed with the setup process: +1. Go to `http://localhost:8888/web` in a web browser and log in to Plex. [![Plex web interface.](plex-browser-view-small.png)](plex-browser-view.png) -3. Give your Plex server a name. Be sure to leave the **Allow me to access my media outside my home** box checked, and click **Next**: +1. Give your Plex server a name. Be sure to leave the **Allow me to access my media outside my home** box checked, and click **Next**: [![Plex web interface - Server Name.](plex-server-name-small.png)](plex-server-name.png) -4. Finally, you'll create the directories that will store your Plex media. In this example we'll create library directories for `movies` and `television` within a `plex-media` directory. These will be located within your user's `/home`: - cd ~/ - mkdir -p plex-media/movies && mkdir plex-media/television +## Add and Organize Media -### Add and Organize Media - -1. Now that you've signed into Plex, you should see the following page. Click the **Add Library** button to start setting up your media libraries: +1. Now that you've signed into Plex, you should see the following page. Click the **Add Library** button to start setting up your media libraries. [![Plex web interface - Add Library](plex-add-library-small.png)](plex-add-library.png) -2. Select your library type, and click **Next**: +1. Select your library type, and click **Next**. [![Plex web interface - Library type](plex-library-type-small.png)](plex-library-type.png) -3. Navigate to the corresponding media directory that you created previously, then click **Add**: +1. Navigate to the corresponding media directory that you created previously, then click **Add**. [![Plex web interface - Library location](plex-library-location-small.png)](plex-library-location.png) -4. You can add additional libraries by clicking the **+** symbol next to the **Libraries** list on the Plex side bar: +1. You can add additional libraries by clicking the **+** symbol next to the **Libraries** list on the Plex side bar: ![Plex web interface - additional Library](plex-additional-library.png) -5. Add your media to the appropriate directories. Be sure to review Plex's [naming conventions](https://support.plex.tv/hc/en-us/categories/200028098-Media-Preparation) for media files, to ensure that your files are identified correctly. +1. Add your media to the appropriate directories. Be sure to review Plex's [naming conventions](https://support.plex.tv/hc/en-us/categories/200028098-Media-Preparation) for media files to ensure that your files are identified correctly. + +## Disable DLNA (Recommended) -### Disable DLNA (Recommended) +[DLNA](https://en.wikipedia.org/wiki/Digital_Living_Network_Alliance) is a protocol that incorporates [Universal Plug and Play](https://en.wikipedia.org/wiki/Universal_Plug_and_Play) (or UPnP) standards for digital media sharing across devices. Any DLNA device or application can have full unrestricted access to your Plex content, so disable DLNA if you won't be using it. -[DLNA](https://en.wikipedia.org/wiki/Digital_Living_Network_Alliance) is a protocol that incorporates [Universal Plug and Play](https://en.wikipedia.org/wiki/Universal_Plug_and_Play) (or UPnP) standards for digital media sharing across devices. If you do not wish to make use of it, it's recommended that you disable this feature, as it is openly connectable on port `1900`. From the Plex web interface, click the wrench icon in the upper right corner, select **Server**, and navigate to the **DLNA** section. Uncheck **Enable the DLNA server**, and click **Save Changes**: +From the Plex web interface, click the wrench icon in the upper right corner and then click **DLNA** in the left sidebar. Uncheck **Enable the DLNA server** and click **Save Changes**. [![Plex media client](plex-dlna-disable-small.png)](plex-dlna-disable.png) ## Connect to your Plex Server -Now that your server is set up, you're ready to connect to it from your Plex client application. Plex is supported by a number of different platforms, and you can find a full list of client applications [here](https://support.plex.tv/hc/en-us/categories/200006953-Plex-Apps). +Now that your server is set up, you're ready to connect to it from a Plex client. Plex is supported by a number of different platforms and you can find a full list of client applications [here](https://support.plex.tv/hc/en-us/categories/200006953-Plex-Apps). -The examples provided here will use **Plex Media Player** for MacOS. +The examples provided here will use Plex Media Player for macOS. -1. [Download](https://www.plex.tv/downloads/) the appropriate media player application, or install it via your device's application store. +1. [Download](https://www.plex.tv/downloads/) the appropriate media player application or install it via your device's app store. -2. Sign in to the Plex client application using the same Plex account as your server. +1. Sign in to the Plex client app using the same Plex account as your server. -3. Your Plex client will have a drop down menu where you can select your server. Once it's selected, you can navigate to the library with the content that you wish to view: +1. Your Plex client will have a dropdown menu where you can select your server. Once it's selected, you can navigate to the library with the content that you wish to view: [![Plex media client](plex-media-client-small.png)](plex-media-client.png) diff --git a/docs/applications/media-servers/install-plex-media-server-on-centos-7/plex-download-button.png b/docs/applications/media-servers/install-plex-media-server-on-centos-7/plex-download-button.png new file mode 100644 index 00000000000..cbab747231d Binary files /dev/null and b/docs/applications/media-servers/install-plex-media-server-on-centos-7/plex-download-button.png differ diff --git a/docs/databases/mysql/mysqldump-database-backup-short/index.md b/docs/databases/mysql/mysqldump-database-backup-short/index.md deleted file mode 100644 index 2347ccd3e58..00000000000 --- a/docs/databases/mysql/mysqldump-database-backup-short/index.md +++ /dev/null @@ -1,45 +0,0 @@ ---- -author: - name: Linode - email: docs@linode.com -description: "Shortguide for backing up a MySQL database with mysqldump." -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -keywords: [] -modified: 2018-08-13 -modified_by: - name: Linode -title: "Back up a MySQL database with mysqldump - Shortguide" -published: 2018-08-13 -headless: true ---- - - - -The `mysqldump` command’s general syntax is: - - mysqldump -u [username] -p [databaseName] > [filename]-$(date +%F).sql - -* `mysqldump` prompts for a password before it starts the backup process. -* Depending on the size of the database, it could take a while to complete. -* The database backup will be created in the directory the command is run. -* `-$(date +%F)` adds a timestamp to the filename. - -Example use cases include: - -* Create a backup of an entire Database Management System (DBMS): - - mysqldump --all-databases --single-transaction --quick --lock-tables=false > full-backup-$(date +%F).sql -u root -p - -* Back up a specific database. Replace `db1` with the name of the database you want to back up: - - mysqldump -u username -p db1 --single-transaction --quick --lock-tables=false > db1-backup-$(date +%F).sql - -* Back up a single table from any database. In the example below, `table1` is exported from the database `db1`: - - mysqldump -u username -p --single-transaction --quick --lock-tables=false db1 table1 > db1-table1-$(date +%F).sql - -Here's a breakdown of the `mysqldump` command options used above: - -- `--single-transaction`: Issue a BEGIN SQL statement before dumping data from the server. -- `--quick`: Enforce dumping tables row by row. This provides added safety for systems with little RAM and/or large databases where storing tables in memory could become problematic. -- `--lock-tables=false`: Do not lock tables for the backup session. diff --git a/docs/databases/mysql/mysqldump-database-restore-short/index.md b/docs/databases/mysql/mysqldump-database-restore-short/index.md deleted file mode 100644 index 88b12ef04f7..00000000000 --- a/docs/databases/mysql/mysqldump-database-restore-short/index.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -author: - name: Linode - email: docs@linode.com -description: "Shortguide for restoring a mysqldump database with MySQL." -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -keywords: [] -modified: 2018-08-13 -modified_by: - name: Linode -title: "Restore a MySQL database from mysqldump - Shortguide" -published: 2018-08-13 -headless: true ---- - - - -The restoration command's general syntax is: - - mysql -u [username] -p [databaseName] < [filename].sql - -* Restore an entire DBMS backup. You will be prompted for the MySQL root user's password:\ - **This will overwrite all current data in the MySQL database system** - - mysql -u root -p < full-backup.sql - -* Restore a single database dump. An empty or old destination database must already exist to import the data into, and the MySQL user you're running the command as must have write access to that database: - - mysql -u [username] -p db1 < db1-backup.sql - -* Restore a single table, you must have a destination database ready to receive the data: - - mysql -u dbadmin -p db1 < db1-table1.sql diff --git a/docs/databases/mysql/use-mysqldump-to-back-up-mysql-or-mariadb/index.md b/docs/databases/mysql/use-mysqldump-to-back-up-mysql-or-mariadb/index.md index 3b3ae0ab3f0..e4968a74ada 100644 --- a/docs/databases/mysql/use-mysqldump-to-back-up-mysql-or-mariadb/index.md +++ b/docs/databases/mysql/use-mysqldump-to-back-up-mysql-or-mariadb/index.md @@ -19,7 +19,9 @@ external_resources: ![Use mysqldump to Back Up MySQL or MariaDB](mysqldump-backup-title.jpg "Use mysqldump to Back Up MySQL or MariaDB") -[MySQL](http://www.mysql.com/) and [MariaDB](https://mariadb.com/) include the [mysqldump](https://dev.mysql.com/doc/refman/5.7/en/mysqldump.html) utility to simplify the process to create a backup of a database or system of databases. Using `mysqldump` is a form of *logical backup*, as opposed to a [*physical backup*](/docs/databases/mysql/create-physical-backups-of-your-mariadb-or-mysql-databases/), which is a copy of the filesystem structure which contains your data. +[MySQL](http://www.mysql.com/) and [MariaDB](https://mariadb.com/) include the [mysqldump](https://dev.mysql.com/doc/refman/5.7/en/mysqldump.html) utility to simplify the process to create a backup of a database or system of databases. Using `mysqldump` creates a *logical backup*. **You can only use this tool if your database process is accessible and running.** + +If your database isn't accessible for any reason, you can instead create a [*physical backup*](/docs/databases/mysql/create-physical-backups-of-your-mariadb-or-mysql-databases/), which is a copy of the filesystem structure which contains your data. The instructions in this guide apply to both MySQL and MariaDB. For simplification, the name MySQL will be used to apply to either. @@ -31,7 +33,34 @@ The instructions in this guide apply to both MySQL and MariaDB. For simplificati ## Back up a Database -{{< content "mysqldump-database-backup-short" >}} +The `mysqldump` command’s general syntax is: + + mysqldump -u [username] -p [databaseName] > [filename]-$(date +%F).sql + +* `mysqldump` prompts for a password before it starts the backup process. +* Depending on the size of the database, it could take a while to complete. +* The database backup will be created in the directory the command is run. +* `-$(date +%F)` adds a timestamp to the filename. + +Example use cases include: + +* Create a backup of an entire Database Management System (DBMS): + + mysqldump --all-databases --single-transaction --quick --lock-tables=false > full-backup-$(date +%F).sql -u root -p + +* Back up a specific database. Replace `db1` with the name of the database you want to back up: + + mysqldump -u username -p db1 --single-transaction --quick --lock-tables=false > db1-backup-$(date +%F).sql + +* Back up a single table from any database. In the example below, `table1` is exported from the database `db1`: + + mysqldump -u username -p --single-transaction --quick --lock-tables=false db1 table1 > db1-table1-$(date +%F).sql + +Here's a breakdown of the `mysqldump` command options used above: + +- `--single-transaction`: Issue a BEGIN SQL statement before dumping data from the server. +- `--quick`: Enforce dumping tables row by row. This provides added safety for systems with little RAM and/or large databases where storing tables in memory could become problematic. +- `--lock-tables=false`: Do not lock tables for the backup session. ## Automate Backups with cron @@ -59,4 +88,19 @@ password = MySQL root user's password ## Restore a Backup -{{< content "mysqldump-database-restore-short" >}} +The restoration command's general syntax is: + + mysql -u [username] -p [databaseName] < [filename].sql + +* Restore an entire DBMS backup. You will be prompted for the MySQL root user's password:\ + **This will overwrite all current data in the MySQL database system** + + mysql -u root -p < full-backup.sql + +* Restore a single database dump. An empty or old destination database must already exist to import the data into, and the MySQL user you're running the command as must have write access to that database: + + mysql -u [username] -p db1 < db1-backup.sql + +* Restore a single table, you must have a destination database ready to receive the data: + + mysql -u dbadmin -p db1 < db1-table1.sql diff --git a/docs/databases/postgresql/how-to-back-up-your-postgresql-database/index.md b/docs/databases/postgresql/how-to-back-up-your-postgresql-database/index.md index b2e24d8f47e..490b2b3bf04 100644 --- a/docs/databases/postgresql/how-to-back-up-your-postgresql-database/index.md +++ b/docs/databases/postgresql/how-to-back-up-your-postgresql-database/index.md @@ -58,11 +58,6 @@ PostgreSQL provides the `pg_dump` utility to simplify backing up a single databa - `*.sql`: plaintext dump - `*.tar`: tarball - {{< note >}} -By default, PostgreSQL will ignore any errors that occur during the backup process. This can result in an incomplete backup. To prevent this, you can run the `pg_dump` command with the `-1` option. This will treat the entire backup procedure as a single transaction, which will prevent partial backups in the event of an error: - - pg_dump dbname > dbname.bak -{{< /note >}} ### Remote Database diff --git a/docs/development/java/install-java-on-ubuntu-18-04/index.md b/docs/development/java/install-java-on-ubuntu-18-04/index.md index 5521a7688b6..74089c81b33 100644 --- a/docs/development/java/install-java-on-ubuntu-18-04/index.md +++ b/docs/development/java/install-java-on-ubuntu-18-04/index.md @@ -2,138 +2,86 @@ author: name: Linode Community email: docs@linode.com -description: Install the Oracle Java development kit -keywords: - - oracle - - java - - JDK - - install java - - ubuntu +description: Install the Oracle Java Development Kit on Ubuntu Bionic Beaver. +keywords: ["java", "oracle", "openjdk", "jdk"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: - - applications/development/install-java-on-ubuntu-18-04/ - - development/install-java-on-ubuntu-18-04/ -modified: 2018-08-06 +aliases: ['applications/development/install-java-on-ubuntu-18-04/','development/install-java-on-ubuntu-18-04/'] +modified: 2019-01-28 modified_by: - name: Phil Zona -published: 2017-05-30T00:00:00.000Z -title: Install Java 10 on Ubuntu 18.04 + name: Linode +published: 2017-05-30 +title: Install Oracle Java SE Development Kit 11 on Ubuntu contributor: name: Phil Zona link: 'https://github.com/pbzona' -external_resources: - - '[Oracle Java](https://www.oracle.com/java/index.html)' audiences: ["beginner"] languages: ["java"] --- -[Java](https://www.oracle.com/java/index.html) is one of the world's most popular programming languages. Java can be used to create anything from software to basic web applications. - -In this guide you will install the Oracle Java development kit (JDK) for building Java applications. This guide will also cover OpenJDK, an open-source alternative to the Oracle Java development kit. - -## Before You Begin - -1. Familiarize yourself with our [Getting Started](/docs/getting-started) guide and complete the steps for setting your Linode's hostname and timezone. - -2. This guide will use `sudo` wherever possible. Complete the sections of our [Securing Your Server](/docs/security/securing-your-server) to create a standard user account, harden SSH access, and remove unnecessary network services. +[Java](https://www.oracle.com/java/index.html) is one of the world's most popular programming languages and can be used to create anything from software to basic web applications. Java was originally developed at Sun Microsystems, which was bought by Oracle. For this reason, Oracle's Java platform is the most well known. However, there are other such as [IBM Runtimes](https://www.ibm.com/us-en/marketplace/support-for-runtimes), [Red Hat's Open JDK](https://developers.redhat.com/products/openjdk/overview/), and [Zulu](https://www.azul.com/products/zulu-enterprise/). -3. Update your system: +With the release of version 11, Oracle has updated the JDK license to be significantly different from previous versions. Oracle JDK is now a paid commercial license, but free for development use. - ``` - sudo apt update -y && sudo apt upgrade -y - ``` - -## Install Oracle JDK +Oracle's [OpenJDK](https://jdk.java.net/11) is GPL licensed. With the exception of [cosmetic and packaging differences](https://blogs.oracle.com/java-platform-group/oracle-jdk-releases-for-java-11-and-later), the OpenJDK and Java SE (Standard Edition) development kits are identical. See Oracle's Java [download page and license](https://www.oracle.com/technetwork/java/javase/downloads/jdk11-downloads-5066655.html) to further decide which version is appropriate for you. -The Oracle JDK includes a development environment for building applications with the Java programming language. Please be aware that some elements of the Oracle JDK are proprietary, meaning that there may be licensing implications with respect to applications you develop with it. +This guide demonstrates installing Oracle's Java Standard Edition Development Kit (Oracle JDK) on Ubuntu 18.04. Installing OpenJDK is also explained for a free and open source alternative. -1. Add the Java PPA: - ``` - sudo add-apt-repository ppa:linuxuprising/java - ``` - - {{< note >}} This repository is _not_ maintained by Oracle. It does not contain actual Java files, but does allow us to download installers for Oracle Java software. Before using the installers, you'll be prompted to accept a license agreement, which can be found in its entirety [here](http://www.oracle.com/technetwork/java/javase/terms/license/index.html). {{< /note >}} - -2. Update the local package cache: - - ``` - sudo apt update -y - ``` - -3. Install the metapackage: - - ``` - sudo apt install oracle-java10-installer -y - ``` +## Before You Begin - This package will run an installer for the Oracle JDK 10, which is the current stable release as of this publication. Two prompts will appear during installation requiring you to accept license agreements. Hit `Enter` to accept both. +- You will need root access to your Linode, or a [limited user account](/docs/security/securing-your-server/#add-a-limited-user-account) with `sudo` privileges. -4. Verify that Java and the Java compiler have been properly installed: +- Update your system: - ``` - java -version - javac -version - ``` + sudo apt update && sudo apt upgrade - As of this publication, these commands should return the following: - ``` - java version "10.0.2" 2018-07-17 - Java(TM) SE Runtime Environment 18.3 (build 10.0.2+13) - Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10.0.2+13, mixed mode) +## Install Oracle JDK - javac 10.0.2 - ``` +The Linux Uprising [Oracle Java PPA](https://launchpad.net/~linuxuprising/+archive/ubuntu/java) uses packages that include install scripts to download and install the Oracle JDK from Oracle's website. Alternatively, you can manually [download the binaries](https://www.oracle.com/technetwork/java/javase/downloads/jdk11-downloads-5066655.html). -5. Since the PPA only provides an installer, and not updates for the JDK itself, you may want to delete it when you're finished in order to keep your repositories organized: +1. Add the PPA to Ubuntu. Apt will automatically refresh the package cache. - ``` - sudo add-apt-repository -r ppa:linuxuprising/java - ``` + sudo add-apt-repository ppa:linuxuprising/java -## Set Java Home Environment +1. Install the JDK. You'll be asked to agree to the licensing terms. -Many applications include code or configurations that references the `JAVA_HOME` environment variable. This variable points them to the Java binary file, allowing them to run Java code. + sudo apt install oracle-java11-installer -1. To set the variable for your system: +1. If you want Oracle JDK as the system default, install `oracle-java11-set-default`: - ``` - echo "JAVA_HOME=$(which java)" | sudo tee -a /etc/environment - ``` + sudo apt install oracle-java11-set-default -2. Reload your system's environment variables: +1. Check that the install was successful with the following command: - ``` - source /etc/environment - ``` + java --version -3. Verify the variable was set correctly: + {{< output >}} +root@ubuntu:~# java --version +java version "11.0.2" 2018-10-16 LTS +Java(TM) SE Runtime Environment 18.9 (build 11.0.2+7-LTS) +Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.2+7-LTS, mixed mode) +{{< /output >}} - ``` - echo $JAVA_HOME - ``` - This should return the path to the Java binary. +1. Reload `/etc/profile` or log out of your system and back in to set the Java environment variables: -## OpenJDK + source /etc/profile -The above installation methods allow you to use the Oracle JDK, which is bound by licensing terms and includes proprietary components. OpenJDK provides an open-source alternative that is just as easy to install. + {{< output >}} +root@ubuntu:~# echo $JAVA_HOME +/usr/lib/jvm/java-11-oracle +{{< /output >}} -To install OpenJDK 11: -``` -sudo apt install openjdk-11-jdk -y -``` +## Install Oracle OpenJDK -The installation will provide you with the OpenJDK, which includes a runtime environment and compiler. This allows you to develop your own Java applications and run them on your Linode. +OpenJDK includes a runtime environment and compiler which allows you to develop your own Java applications and run them on your Linode. If you only need to run Java applications which you've already downloaded, you can save some disk space by only installing the Java runtime environment. Both are available from Ubuntu's repository. -However, if you only need to run applications that you've already downloaded, you can save a bit of disk space by installing the OpenJRE (Java runtime environment): +- Install OpenJDK 11: -``` -sudo apt install openjdk-11-jre -``` + sudo apt install openjdk-11-jdk -Note that this is unnecessary if you've installed OpenJDK, since it includes the JRE. +- Install OpenJRE 11: -{{< caution >}} OpenJDK and Oracle Java are _not_ identical. There may be licensing, performance, and stability differences, and this should be considered carefully when developing production applications. {{< /caution >}} + sudo apt install openjdk-11-jre \ No newline at end of file diff --git a/docs/development/version-control/how-to-configure-git/index.md b/docs/development/version-control/how-to-configure-git/index.md index 752e201bb5e..3c03a53f73b 100644 --- a/docs/development/version-control/how-to-configure-git/index.md +++ b/docs/development/version-control/how-to-configure-git/index.md @@ -5,13 +5,13 @@ author: description: 'Learn the basics of Git in this guide. Discover one of the most popular distributed version control and source code management systems that make contributing to projects and working with a team easy.' keywords: ["git", "dvcs", "vcs", "scm", "gitweb", "gitolite", "ubuntu", "debian", "arch", "gentoo"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -modified: 2017-07-03 +modified: 2019-01-15 modified_by: name: Linode published: 2009-09-04 title: Getting Started with Git external_resources: - - '[Refspec Information](http://git-scm.com/book/ch9-5.html)' + - '[Refspec Information](https://git-scm.com/book/en/v2/Git-Internals-The-Refspec)' - '[Learn Git with Bitbucket Cloud](https://www.atlassian.com/git/tutorials/learn-git-with-bitbucket-cloud)' - '[Pro Git Book](https://git-scm.com/book/en/v2)' - '[Github Guides](https://guides.github.com/)' @@ -22,7 +22,7 @@ audiences: ["foundational"] ## What is Git? -Git was designed and developed by [Linus Torvalds](https://en.wikipedia.org/wiki/Linus_Torvalds) for Linux kernel development. Git provides support for non-linear, distributed development, allowing multiple contributors to work on a project simultaneously. Git is the most popular distributed version control and source code management system. This guide will walk you through the basics of getting started with Git, from installing the software to using basic commands on both local and remote repositories (repo). +Git is a distributed version control system. Git was designed and developed by [Linus Torvalds](https://en.wikipedia.org/wiki/Linus_Torvalds) for Linux kernel development. Git provides support for non-linear, distributed development, allowing multiple contributors to work on a project simultaneously. Git is the most popular distributed version control and source code management system. This guide will walk you through the basics of getting started with Git, from installing the software to using basic commands on both local and remote repositories (repo). ## Configure Git @@ -49,24 +49,38 @@ Set your default text editor, replacing `editor-name` with your desired editor: The output of `git config --list` should show echo the information you inputted: - MacBook-Pro:~ user$ git config --list + $ git config --list user.name=exampleuser user.email=user@email.com core.editor=editor-name -## Work with an Existing Local Repository (Repo) +## Use Git with a Local Repository -If you have an existing project and you want to start using Git to keep track of its changes, run `git init` from the existing project's directory: +A *repository*, or repo, is a collection of files and folders and the history of their changes. Changes are tracked through *commits*, which are like snapshots of a file at various points in the file's history. These commits are not automatic, you need to manually stage a commit after each of series of file changes. Commits allow you to refer or revert back to a place in the file's timeline if there are bugs or errors in your code. + +If you have an new or existing project and you want to start using Git to keep track of its changes, run `git init` from the existing project's directory: git init -`git init` creates a new `.git` subdirectory in the current directory. This is where Git stores your configurations. The `git add` command tells Git to track changes of files: +`git init` creates a new `.git` subdirectory in the current directory. This is where Git stores your configurations. The `git add` command tells Git to add a file to the repository and track that file's changes: git add filename -After you have added the file, stage a commit and leave a commit message. Commit message serve as a reminder of the changes that were made to a file: +After you have added the file, stage a commit and leave a commit message. Commit messages serve as a reminder of the changes that were made to a file: + + git commit -m "Initialized a Git repository for this project. Tracking changes to a file." + +{{< note >}} +It's good practice to provide clear and descriptive commit messages for every commit you stage, as this helps collaborators to understand what a commit encompasses. +{{< /note >}} - git commit -m "Initialized a Git repository for this project. tracking changes to a file" +There may be files or folders in your project directory that you do not wish to include in your Git repository. You can include these files in a `.gitignore` file, and Git will ignore them. A sample `.gitignore` file might look like the following: + +{{< file ".gitignore" >}} +.DS_Store +*.zip +__doNotInclude__/ +{{< /file >}} ### Basic Git Commands @@ -78,13 +92,13 @@ This table lists basic commands, a description, and an example of the command in | `git rm` | Remove a file from a repository. | `git rm filename` | | `git mv` | Move or rename a tracked file, directory, or symlink. | `git mv file_from file_to` | | `git branch` | List all the local and remote branches. | `git branch branchname` | -| `git commit` | Commit all staged objects. | `git commit -m "updates"` | +| `git commit` | Commit all staged objects. Optionally, you can append a message with the `-m` flag. | `git commit -m "updates"` | | `git pull` | Download all changes from the remote repo and merge them in a specified repo file. | `git pull repo refspec` | | `git push` | Publish the changes to the remote repo. | `git push repo` | ### Branches -Branches are used for editing files without disturbing the working portions of a project. The main branch is normally named `master`, it's customary to name the branch after an issue being fixed or a feature being implemented. Because Git keep tracks of file changes, you can jump from branch to branch without overwriting or interfering with other branches in the repo. +*Branches* are used for editing files without disturbing the working portions of a project. The main branch is normally named `master` and is usually reserved for clean, working code. When making changes to your code, it's customary to create a new branch and name it after the issue being fixed or the feature being implemented. Because Git keep tracks of file changes, you can jump from branch to branch without overwriting or interfering with other branches in the repo. The basic options used with the `git branch` command are: @@ -96,13 +110,42 @@ The basic options used with the `git branch` command are: | -d | Delete a branch | | -r -d | Delete a remote branch | -## Working with Remote Repositories +### Example Usage + +Consider an application with a single `master` branch. The author of the application wants to develop a new search feature. They would add a new feature branch: + + git branch new-search-feature + +Then, they would switch to that branch using the `checkout` command: + + git checkout new-search-feature + +Now they can safely develop and commit their changes to this feature branch without altering the working code of the `master` branch. At any time they could switch back to the `master` branch: -Remote repositories are hosted on a network or another location on the Internet. This section provides some basic information on navigating remote Git repositories. + git checkout master + +A shortcut for creating a branch and switching to that branch is to use the `-b` flag with the `checkout` command: + + git checkout -b new-search-feature + +Once the new search feature is finalized, the author of the application can merge the `new-search-feature` branch into the `master` branch: + + git checkout master + git merge new-search-feature + +Now the `master` branch has the new search feature. + +## Use Git with a Remote Repository + +[GitHub](https://github.com), [GitLab](https://gitlab.com), and [Bitbucket](https://bitbucket.org/) all provide ways to store Git repositories remotely and facilitate collaboration. Many of these services also include a number of other features that are vital to content development, including pull requests, continuous integration / continuous delivery pipelines (CI/CD), wikis, and webhooks. If you'd rather use a self-hosted solution, GitLab and [Gogs](https://gogs.io/) offer free locally hosted versions of their software that can easily be managed on a Linode. Check out our guides on [installing GitLab](/docs/development/version-control/install-gitlab-on-ubuntu-18-04/) and [installing Gogs](/docs/development/version-control/install-gogs-on-debian/) for more information on hosting your own remote repository software. GitHub and Bitbucket also offer paid enterprise versions of their software for local hosting. When discussing remote repositories, usually one of the aforementioned services is being referenced. + +This section provides some basic information on navigating remote Git repositories. To copy every file from a remote repository to your local system, use `git clone` followed by the remote repo's URL: - git clone remoteurl + git clone https://github.com/linode/docs.git + +You can typically find a remote repository's URL by clicking on the *Clone* or *Download* buttons of a remote repository's user interface. To check the status of the files within the current branch of your repository, use `status`: @@ -118,6 +161,8 @@ The `remote` command will display the short names of your remote repositories. I git remote -v +### Git Remote Repository Commands + Below are some basic commands for working with remote repositories: | Command | Description | @@ -129,3 +174,40 @@ Below are some basic commands for working with remote repositories: | `git remote show [remote-name]` | Display information about the remote you specified. | | `git remote rename [old-name] [new-name]` | Rename a remote. | | `git remote rm [name]` | Remove the remote you specified. | + +### Example Usage + +Continuing the example posed in the previous section, let's say that someone wants to add a functionality to an application in a remote repository. The first step would be to *fork* that repository. This feature, like cloning, is usually available in the user interface of the remote repository software. Forking creates a remote repository, which is a copy of the forked repository, on the user's own account. From their the user would clone the fork locally use the `clone` command discussed above. + + git clone https://github.com/username/Spoon-Knife.git + +Once the repository is cloned, the user would create a branch for the new feature using either the `branch` or `checkout -b` command. + + git checkout -b new-special-feature + +After the branch is made, the user would make the necessary updates or changes to the codebase, and commit them. + + git commit -m "A new special feature" + +With the commit staged, the user would then push their changes to their fork of the remote repository: + + git push origin new-special-feature + +The result of the push command is that now the new feature branch exists on the user's fork. To contribute those changes to the initial repository, known as the upstream repository, the user must now submit a *pull request* (PR). A PR is a feature that most remote repository software has that allows the user to safely contribute a commit, or series of commits, to an upstream repository. This is accomplished by navigating through the user interface to the pull requests page, selecting the master branch of the upstream repository as the target, and selecting the origin repository's feature branch as the source. A pull request will be created and the code will be up for review before being merged into the master branch of the upstream repository. + +While creating PRs is a healthy standard within the development community, it is not the only way to contribute to a remote repository. You could simply push your commits directly to the upstream repository: + + git push my-repo new-special-feature + +The downside of this approach is that there is no option to hold the pushed commits for review. However, if you are the only contributor to a project then obviously there will be no one to review your changes, so creating a fork and contributing PRs is not necessary. + +If you are collaborating with another developer, it is necessary to be able to retrieve their work. To do so, issue the `pull` command: + + git checkout new-special-feature + git pull + +Git will grab the new code from the chosen remote repository branch and merge it into your local branch. + +{{< note >}} +Sometimes two developers will edit the same section of a file at the same time and attempt to merge their changes into the codebase. When this happens, Git will throw an error called a *merge conflict*. Because Git will be unable to determine which set of changes is the correct set of changes, it will prompt you to fix the merge conflict before it moves forward with the merge. +{{< /note >}} \ No newline at end of file diff --git a/docs/development/version-control/how-to-install-git-mac/index.md b/docs/development/version-control/how-to-install-git-mac/index.md index ca83580281b..05628870ca0 100644 --- a/docs/development/version-control/how-to-install-git-mac/index.md +++ b/docs/development/version-control/how-to-install-git-mac/index.md @@ -5,9 +5,9 @@ author: description: 'Shortguide for installing Git' license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' keywords: ["mac", "git", "version"] -modified: 2017-01-05 +modified: 2019-01-24 modified_by: - name: Sam Foo + name: Linode title: "How to install Git on Mac" published: 2018-01-08 headless: true @@ -56,26 +56,30 @@ If MacPorts is already installed, proceed directly to Step 9. 3. After the download completes, click on the file to start the installer. 4. Click the **Continue** button to begin the install: - ![Begin MacPort install.](1586-MacPorts_1.png) + ![Begin MacPort install.](macports-01-install.png) + +5. Click **Continue** and then **Agree** to accept the license agreement: -5. Click **Agree** to accept the license agreement: + ![Agree to the license agreement.](macports-02-agree.png) - ![Agree to the license agreement.](1589-MacPorts_4.png) +6. Click **Continue** to continue the installation: -6. To leave the default installation location click **Install**, or to change the default installation location, click **Change Install Location**: + ![Click continue to continue the installation](macports-03-setup.png) - ![Change default location.](1590-MacPorts_5.png) +7. To leave the default installation location click **Install**, or to change the default installation location, click **Change Install Location**: -7. Enter your password when prompted and click **Install Software**: + ![Change default location.](macports-04-disk-space.png) - ![Enter your password.](1591-MacPorts_6.png) +8. Enter your password when prompted and click **Install Software**: -8. After the installation completes, click **Close** to exit the installer: + ![Enter your password.](macports-06-enter-password.png) - ![Click close to exit.](1592-MacPorts_7.png) +9. After the installation completes, click **Close** to exit the installer: -9. Open a terminal window. -10. Install Git: + ![Click close to exit.](macports-05-complete-install.png) + +10. Open a terminal window. +11. Install Git: sudo port install git +svn +doc +bash_completion +gitweb @@ -83,22 +87,30 @@ If MacPorts is already installed, proceed directly to Step 9. 1. Go to the [Git](http://git-scm.com/downloads) website and download the current installer for your operating system. 2. Double click on the downloaded file to begin the install. -3. Click on the **Continue** button to begin the install: +3. Double click the `.pkg` file. A dialog window may appear stating that the file cannot be opened: + + ![A dialog may appear stating that the file cannot be opened](macgit-01-cant-open.png) + +4. Click on the Apple icon in the top left of the screen, and go to **System Preferences > Security & Privacy**. Click **Open Anyway**: + + ![Click on "Open Anyway" in the Security & Privacy preference window.](macgit-02-open-anyway.png) + +5. Click on **Open**: - ![Click on continue.](1581-MacGit_1.png) + ![Click on "Open"](macgit-03-open.png) -4. Select **Install for all users of this computer** and click **Continue** to begin the installation: +6. Click **Continue** button to begin the install: - ![Click on install for all users of this computer.](1582-MacGit_2.png) + ![Click Continue](macgit-04-install.png) -5. If you need to change the installation location click **Change Install Location** button. Click **Install** to continue with the default location: +7. To leave the default installation location click **Install**, or to change the default installation location, click **Change Install Location**: - ![Click on change install location.](1583-MacGit_3.png) + ![Click "Continue" to use the default installation location.](macgit-05-disk-space.png) -6. Enter your username and password and click **Install Software** to continue: +8. Enter your password when prompted, and then select **Install Software**: - ![Enter username and password and click install software.](1584-MacGit_4.png) + ![Enter your password](macgit-07-enter-password.png) -7. Click the **Close** button to exit the installer: +9. After the installation completes, click **Close** to exit the installer: - ![Click close to exit.](1585-MacGit_5.png) + ![Click on "Close" to complete the install](macgit-06-complete-install.png) diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1581-MacGit_1.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1581-MacGit_1.png deleted file mode 100644 index 46d9986588d..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1581-MacGit_1.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1582-MacGit_2.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1582-MacGit_2.png deleted file mode 100644 index b6757bd565c..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1582-MacGit_2.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1583-MacGit_3.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1583-MacGit_3.png deleted file mode 100644 index 07db0c4650f..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1583-MacGit_3.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1584-MacGit_4.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1584-MacGit_4.png deleted file mode 100644 index c672be6dc8c..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1584-MacGit_4.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1585-MacGit_5.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1585-MacGit_5.png deleted file mode 100644 index c85d2a206de..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1585-MacGit_5.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1586-MacPorts_1.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1586-MacPorts_1.png deleted file mode 100644 index 4ab3cdbd967..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1586-MacPorts_1.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1589-MacPorts_4.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1589-MacPorts_4.png deleted file mode 100644 index 4e5c5685528..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1589-MacPorts_4.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1590-MacPorts_5.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1590-MacPorts_5.png deleted file mode 100644 index ebf8d74759a..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1590-MacPorts_5.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1591-MacPorts_6.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1591-MacPorts_6.png deleted file mode 100644 index 967265cb923..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1591-MacPorts_6.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1592-MacPorts_7.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1592-MacPorts_7.png deleted file mode 100644 index 6a0625fd53d..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1592-MacPorts_7.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1595-Git-Win-01.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1595-Git-Win-01.png deleted file mode 100644 index 559e04173d9..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1595-Git-Win-01.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1596-Git-Win-02.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1596-Git-Win-02.png deleted file mode 100644 index db3bfaa2482..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1596-Git-Win-02.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1597-Git-Win-03.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1597-Git-Win-03.png deleted file mode 100644 index 6f120f4ee69..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1597-Git-Win-03.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1598-Git-Win-04.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1598-Git-Win-04.png deleted file mode 100644 index ff80aa7c685..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1598-Git-Win-04.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1599-Git-Win-05.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1599-Git-Win-05.png deleted file mode 100644 index ee0ae22de80..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1599-Git-Win-05.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1600-Git-Win-06.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1600-Git-Win-06.png deleted file mode 100644 index c5e1a4cec59..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1600-Git-Win-06.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1601-Git-Win-07.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1601-Git-Win-07.png deleted file mode 100644 index 55b20b04642..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1601-Git-Win-07.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1602-Git-Win-08.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1602-Git-Win-08.png deleted file mode 100644 index d760603c6cf..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1602-Git-Win-08.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1603-Git-Win-09.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1603-Git-Win-09.png deleted file mode 100644 index 9e31c9b6481..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1603-Git-Win-09.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1604-Git-Win-10.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1604-Git-Win-10.png deleted file mode 100644 index a15e5b8c54d..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1604-Git-Win-10.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1605-Git-Win-11.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1605-Git-Win-11.png deleted file mode 100644 index 00b6cf5207a..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1605-Git-Win-11.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1606-Git-Win-12.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1606-Git-Win-12.png deleted file mode 100644 index 873f6bdc894..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1606-Git-Win-12.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1607-Git-Win-13.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1607-Git-Win-13.png deleted file mode 100644 index 914270fb407..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1607-Git-Win-13.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1608-Git-Win-14.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1608-Git-Win-14.png deleted file mode 100644 index 279b84c8c3a..00000000000 Binary files a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/1608-Git-Win-14.png and /dev/null differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/index.md b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/index.md index 4722e0cb054..32a13842d13 100644 --- a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/index.md +++ b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/index.md @@ -6,7 +6,7 @@ description: 'Installing git on GNU/Linux, MacOSX, and Windows' keywords: ["git", "dvcs", "vcs", "scm", "gitweb"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' aliases: ['linux-tools/version-control/git/','applications/development/git-source-control-management/','development/version-control/git-source-control-management/','development/version-control/how-to-install-git-on-mac-and-windows/'] -modified: 2017-07-03 +modified: 2019-01-24 modified_by: name: Linode published: 2009-09-04 diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macgit-01-cant-open.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macgit-01-cant-open.png new file mode 100644 index 00000000000..ea5925b69a8 Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macgit-01-cant-open.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macgit-02-open-anyway.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macgit-02-open-anyway.png new file mode 100644 index 00000000000..b47adc9bfc8 Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macgit-02-open-anyway.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macgit-03-open.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macgit-03-open.png new file mode 100644 index 00000000000..633e2821a44 Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macgit-03-open.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macgit-04-install.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macgit-04-install.png new file mode 100644 index 00000000000..ee2028966b2 Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macgit-04-install.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macgit-05-disk-space.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macgit-05-disk-space.png new file mode 100644 index 00000000000..850c1421cf3 Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macgit-05-disk-space.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macgit-06-complete-install.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macgit-06-complete-install.png new file mode 100644 index 00000000000..735c2fafe59 Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macgit-06-complete-install.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macgit-07-enter-password.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macgit-07-enter-password.png new file mode 100644 index 00000000000..3588dde0e2b Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macgit-07-enter-password.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macports-01-install.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macports-01-install.png new file mode 100644 index 00000000000..a83db833edc Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macports-01-install.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macports-02-agree.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macports-02-agree.png new file mode 100644 index 00000000000..e2c9803a1a8 Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macports-02-agree.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macports-03-setup.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macports-03-setup.png new file mode 100644 index 00000000000..0af085f1889 Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macports-03-setup.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macports-04-disk-space.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macports-04-disk-space.png new file mode 100644 index 00000000000..f43f134fb22 Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macports-04-disk-space.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macports-05-complete-install.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macports-05-complete-install.png new file mode 100644 index 00000000000..d8f8f5df2a9 Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macports-05-complete-install.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macports-06-enter-password.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macports-06-enter-password.png new file mode 100644 index 00000000000..3588dde0e2b Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/macports-06-enter-password.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-01-download-git.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-01-download-git.png new file mode 100644 index 00000000000..45ca9493ec3 Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-01-download-git.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-02-save-git.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-02-save-git.png new file mode 100644 index 00000000000..bbb2e62fac1 Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-02-save-git.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-03-run-git.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-03-run-git.png new file mode 100644 index 00000000000..c0cc916dcd2 Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-03-run-git.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-04-agree-license.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-04-agree-license.png new file mode 100644 index 00000000000..6ac2d2f866b Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-04-agree-license.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-05-select-folder.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-05-select-folder.png new file mode 100644 index 00000000000..543b5415f76 Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-05-select-folder.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-06-select-components.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-06-select-components.png new file mode 100644 index 00000000000..43cde64cb90 Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-06-select-components.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-07-shortcut-folder.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-07-shortcut-folder.png new file mode 100644 index 00000000000..fd03fd0731c Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-07-shortcut-folder.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-08-default-text-editor.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-08-default-text-editor.png new file mode 100644 index 00000000000..16c1c861be6 Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-08-default-text-editor.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-09-adjust-your-path.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-09-adjust-your-path.png new file mode 100644 index 00000000000..6af68c2eafa Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-09-adjust-your-path.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-10-choose-https-backend.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-10-choose-https-backend.png new file mode 100644 index 00000000000..bff5d5f990b Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-10-choose-https-backend.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-11-configure-line-endings.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-11-configure-line-endings.png new file mode 100644 index 00000000000..d95b35e2844 Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-11-configure-line-endings.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-12-terminal-emulator.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-12-terminal-emulator.png new file mode 100644 index 00000000000..b97df4b2de0 Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-12-terminal-emulator.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-13-extra-options.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-13-extra-options.png new file mode 100644 index 00000000000..4c2a54530d2 Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-13-extra-options.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-14-complete-setup.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-14-complete-setup.png new file mode 100644 index 00000000000..b2173bf54fa Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-14-complete-setup.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-15-run-gui.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-15-run-gui.png new file mode 100644 index 00000000000..e6f25dbadcb Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-15-run-gui.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-16-create-new-project.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-16-create-new-project.png new file mode 100644 index 00000000000..0d57b66232b Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-16-create-new-project.png differ diff --git a/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-17-new-project.png b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-17-new-project.png new file mode 100644 index 00000000000..cacb3d3772c Binary files /dev/null and b/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/win-17-new-project.png differ diff --git a/docs/development/version-control/how-to-install-git-windows/index.md b/docs/development/version-control/how-to-install-git-windows/index.md index 0bb83cac27c..79a3fa9dd2e 100644 --- a/docs/development/version-control/how-to-install-git-windows/index.md +++ b/docs/development/version-control/how-to-install-git-windows/index.md @@ -5,9 +5,9 @@ author: description: 'Shortguide for installing Git on Windows' license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' keywords: ["windows", "git", "version"] -modified: 2017-01-05 +modified: 2019-01-24 modified_by: - name: Sam Foo + name: Linode title: "How to install Git on Windows" published: 2018-01-08 headless: true @@ -18,60 +18,72 @@ To install Git on Windows you will need to download the installer from the [Git] 1. Download the most current version for your operating system by double clicking on the package name: - ![Click to download version for Windows.](1595-Git-Win-01.png) + ![Click to download version for Windows.](win-01-download-git.png) 2. Select **Run** to begin the installation: - ![Click Run to start the installer.](1596-Git-Win-02.png) + ![Click Run to start the installer.](win-02-save-git.png) 3. Click **Yes** to continue: - ![Click Yes to continue.](1597-Git-Win-03.png) + ![Click Yes to continue.](win-03-run-git.png) 4. Click **Next** to continue: - ![Click Next to continue.](1598-Git-Win-04.png) + ![Click Next to continue.](win-04-agree-license.png) + +5. If you need to change the installation folder, click **Browse** and select a new location. To accept the default location click on **Next**: + + ![Click Next for default or Browse to change.](win-05-select-folder.png) + +6. To accept the default components to be installed click **Next**. Otherwise, select the additional components to be installed before clicking the **Next** button: -5. Click **Next** to continue: + ![Accept the default or select additional components.](win-06-select-components.png) - ![Click Next to continue.](1599-Git-Win-05.png) +7. Accept the default Start Menu folder by clicking **Next**, or use **Browse** to select a new folder location: -6. If you need to change the installation folder, click **Browse** and select a new location. To accept the default location click on **Next**: + ![Select the start menu folder.](win-07-shortcut-folder.png) - ![Click Next for default or Browse to change.](1600-Git-Win-06.png) +8. Select the default text editor for Git, then click **Next**: -7. To accept the default components to be installed click **Next**. Otherwise, select the additional components to be installed before clicking the **Next** button: + ![Select the default text editor for Git.](win-08-default-text-editor.png) - ![Accept the default or select additional components.](1601-Git-Win-07.png) +9. Adjust your PATH environment, then click **Next**: -8. Accept the default Start Menu folder by clicking **Next**, or use **Browse** to select a new folder location: + ![Adjust Git's PATH](win-09-adjust-your-path.png) - ![Select the start menu folder.](1602-Git-Win-08.png) +10. Choose which SSL/TLS library you'll use for HTTPS. Then, click **Next**: -9. Keep the default *line ending conversion* by clicking **Next**. To change the default, choose one of the two other choices before clicking **Next**: + ![Choose your HTTPS backend.](win-10-choose-https-backend.png) - ![Select line ending conversions.](1603-Git-Win-09.png) +11. Keep the default *line ending conversion* by clicking **Next**. To change the default, choose one of the two other choices before clicking **Next**: + + ![Select line ending conversions.](win-11-configure-line-endings.png) {{< note >}} It is strongly recommended that you keep the default settings for line ending conversions. Changing from the default may result in formatting problems when viewed on your Linode. {{< /note >}} -10. The program will now begin installing: +12. Choose the terminal emulator you'll use, and then click **Next**: + + ![Select the terminal emulator you'll use.](win-12-terminal-emulator.png) + +13. Configure the extra options, and then click **Next**: - ![Git is installing.](1604-Git-Win-10.png) + ![Configure the extra options.](win-13-extra-options.png) -11. Click **Finish** to exit the installer: +14. Check **Launch Git Bash** and complete the setup by selecting **Finish**: - ![Click Finish to exit.](1605-Git-Win-11.png) + ![Complete the setup and launch the Git bash shell](win-14-complete-setup.png) -12. Once you open Git you can select either **Create New**, **Clone Existing** or **Open Existing Repository**. In this example, we create new repository: +15. You can launch the Git GUI from the bash shell. Type `git gui` at the command line and hit enter: - ![Create New Repository.](1606-Git-Win-12.png) + ![Launch the Git GUI by typing 'git gui' at the command line.](win-15-run-gui.png) -13. Enter a directory name or click on **Browse** to navigate to a directory: +16. Once you open Git you can select either **Create New**, **Clone Existing** or **Open Existing Repository**. In this example, we create new repository. Enter a directory name or click on **Browse** to navigate to a directory: - ![Directory for a new repository.](1607-Git-Win-13.png) + ![Create New Repository.](win-16-create-new-project.png) -14. A blank repository is created: +17. A blank repository is created: - ![Blank repository.](1608-Git-Win-14.png) + ![Blank repository.](win-17-new-project.png) diff --git a/docs/development/version-control/install-gitlab-on-ubuntu-14-04-trusty-tahr/index.md b/docs/development/version-control/install-gitlab-on-ubuntu-14-04-trusty-tahr/index.md index 2e2cc9b808b..3faeca2f9db 100644 --- a/docs/development/version-control/install-gitlab-on-ubuntu-14-04-trusty-tahr/index.md +++ b/docs/development/version-control/install-gitlab-on-ubuntu-14-04-trusty-tahr/index.md @@ -14,6 +14,8 @@ modified_by: name: Linode published: 2014-09-04 title: 'Install GitLab on Ubuntu 14.04 (Trusty Tahr)' +deprecated: true +deprecated_link: development/version-control/install-gitlab-on-ubuntu-18-04/ external_resources: - '[GitLab Community Edition](https://www.gitlab.com/gitlab-ce/)' - '[GitLab Documentation](https://www.gitlab.com/documentation/)' diff --git a/docs/development/version-control/install-gitlab-on-ubuntu-18-04/gitlab-password-reset.png b/docs/development/version-control/install-gitlab-on-ubuntu-18-04/gitlab-password-reset.png new file mode 100644 index 00000000000..15293f74a6f Binary files /dev/null and b/docs/development/version-control/install-gitlab-on-ubuntu-18-04/gitlab-password-reset.png differ diff --git a/docs/development/version-control/install-gitlab-on-ubuntu-18-04/gitlab-welcome.png b/docs/development/version-control/install-gitlab-on-ubuntu-18-04/gitlab-welcome.png new file mode 100644 index 00000000000..5fb95f3ed0b Binary files /dev/null and b/docs/development/version-control/install-gitlab-on-ubuntu-18-04/gitlab-welcome.png differ diff --git a/docs/development/version-control/install-gitlab-on-ubuntu-18-04/index.md b/docs/development/version-control/install-gitlab-on-ubuntu-18-04/index.md new file mode 100644 index 00000000000..465edc79c90 --- /dev/null +++ b/docs/development/version-control/install-gitlab-on-ubuntu-18-04/index.md @@ -0,0 +1,116 @@ +--- +author: + name: Linode Community + email: docs@linode.com +description: 'Install GitLab on Ubuntu 18.04 with SSL' +keywords: ['git', 'gitlab', 'version control', 'ubuntu'] +license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' +published: 2019-01-04 +modified: 2019-01-04 +modified_by: + name: Linode +title: 'Install GitLab on Ubuntu 18.04' +contributor: + name: Linode +external_resources: +- '[GitLab''s Official Documentation](https://docs.gitlab.com/ee/README.html)' +- '[GitLab''s NGINX Configurations](https://github.com/linode/docs/pull/2231)' +audiences: ["intermediate"] +--- + +GitLab is a complete solution for all aspects of your software development life-cycle. At its core, GitLab serves as your centralized Git repository. It also features built-in tools that represent every task in your development workflow, from planning to testing to releasing. You can host your own GitLab instance on a Linode, instead of using third-party hosting. Self-hosting your software development with GitLab offers total control of your codebase while providing an easy to use interface for team members. GitLab is the most popular self-hosted Git repository, so you'll benefit from a robust set of integrated tools and an active community. + +This guide will walk you through the steps to install GitLab on an 8GB Linode running Ubuntu 18.04. This installation can support up to 100 users. + +## System Requirements + +Before installing GitLab you should consider how many users will collaborate on your self-hosted instance, the size of the repositories you will store, and the recommended minimum system requirements. This criteria will will effect the needed storage, CPU, and memory. This guide will use an **8GB Linode plan** to fulfill GitLab's minimum system requirements. The suggested hardware is as follows: + +- **Storage** The required storage depends on the size of the repositories you will store in GitLab. You should plan to have *at least* as much free space as all the repositories combined require. +- **CPU:** 2 cores is the recommended number and supports up to 500 users. While you can use 1 CPU core to support 100 users, the application may run slower because all workers and background jobs will run on the same core. +- **Memory:** 8 GB to support up to 100 users. + +## Before You Begin + +1. Familiarize yourself with our [Getting Started](/docs/getting-started/) guide and complete the steps for setting your Linode's hostname and timezone. + +1. This guide will use `sudo` wherever possible. Complete the sections of our [Securing Your Server](/docs/security/securing-your-server/) to create a standard user account, harden SSH access and remove unnecessary network services. + +1. Add a domain zone, NS record, and A/AAA record for the domain you will use to access your GitLab installation. See the [DNS Manager](/docs/platform/manager/dns-manager-new-manager/#add-a-domain-zone) guide for details. If you will access your GitLab instance via your Linode’s IP address, you can skip this step. + +1. [Create an SSL Certificate](/docs/quick-answers/websites/secure-http-traffic-certbot/), if you will be using SSL encryption for your domain (this is recommended). Be sure to note the location that Certbot uses to store all generated keys and issued certificates. + +1. Update your system: + + sudo apt-get update && sudo apt-get upgrade + +## Install GitLab + +1. Install all required dependencies: + + sudo apt-get install -y curl openssh-server ca-certificates + +1. Install Postfix to send email notifications: + + sudo apt-get install -y postfix + + When prompted, select *Internet Site* and press **Enter**. Use your server's external DNS for *mail name* and press **Enter**. + +1. Add the GitLab package repository: + + curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.deb.sh | sudo bash + +1. Install the GitLab package. Replace `gitlab.example.com` with the domain you will use to access your GitLab installation. The installation will automatically configure and start GitLab. + + sudo EXTERNAL_URL="http://gitlab.example.com" apt-get install gitlab-ee + +1. In your browser of choice, navigate to the URL you provided in the previous step. You will be redirected to GitLab's password reset screen. You should provide a password for the GitLab administrator account. + + ![GitLab password reset](gitlab-password-reset.png) + +1. You will be redirected to the login screen. Enter `root` as the username and the password you just created to log in. + + ![GitLab welcome screen](gitlab-welcome.png) + +## Configure SSL Encryption + +{{< note >}} +If you did not generate an SSL certificate using Certbot prior to the installation of GitLab, you may need to first stop GitLab and then generate the SSL certificate to bypass any errors related to Certbot's certificate challenge. To stop GitLab run the following command: + + sudo gitlab-ctl stop + +Once you are done generating the certificate, restart GitLab with the following command: + + sudo gitlab-ctl start +{{}} + +1. Edit the `/etc/gitlab/gitlab.rb` to use HTTPS. This is done by modifying the value of `external_url` to use `https` instead of `http`: + + {{< file "/etc/gitlab/gitlab.rb" ruby >}} +## GitLab URL +##! URL on which GitLab will be reachable. +##! For more details on configuring external_url see: +##! https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-the-external-url-for-gitlab +external_url 'https://gitlab.example.com' + {{}} + +1. Edit the `/etc/gitlab/gitlab.rb` file to point to the location of your SSL certificate and key. The path should be the location used by Certbot to store the certificates when they were initially created. + + {{< file "/etc/gitlab/gitlab.rb" ruby >}} +nginx['ssl_certificate'] = "/etc/letsencrypt/live/gitlab.example.com/fullchain.pem" +nginx['ssl_certificate_key'] = "/etc/letsencrypt/live/gitlab.example.com/privkey.pem" + {{}} + +1. Redirect all HTTP traffic to HTTPS: + + {{< file "/etc/gitlab/gitlab.rb" ruby >}} +nginx['redirect_http_to_https'] = true + {{}} + +1. Issue the following command to enable your new configurations: + + sudo gitlab-ctl reconfigure + +1. Navigate to your GitLab instance domain and verify that you are directed to `https`. + + You are now ready to begin using GitLab as your remote version control system. Refer to [GitLab's official documentation](https://docs.gitlab.com/ee/README.html) for details on how to get started administering your GitLab instance. \ No newline at end of file diff --git a/docs/development/version-control/install-gitlab-with-docker/gitlab-docker-create-project.png b/docs/development/version-control/install-gitlab-with-docker/gitlab-docker-create-project.png new file mode 100644 index 00000000000..fdb35fb9920 Binary files /dev/null and b/docs/development/version-control/install-gitlab-with-docker/gitlab-docker-create-project.png differ diff --git a/docs/development/version-control/install-gitlab-with-docker/gitlab-docker-empty-project.png b/docs/development/version-control/install-gitlab-with-docker/gitlab-docker-empty-project.png new file mode 100644 index 00000000000..004fc8f99f5 Binary files /dev/null and b/docs/development/version-control/install-gitlab-with-docker/gitlab-docker-empty-project.png differ diff --git a/docs/development/version-control/install-gitlab-with-docker/gitlab-docker-welcome-screen.png b/docs/development/version-control/install-gitlab-with-docker/gitlab-docker-welcome-screen.png new file mode 100644 index 00000000000..e74a0247250 Binary files /dev/null and b/docs/development/version-control/install-gitlab-with-docker/gitlab-docker-welcome-screen.png differ diff --git a/docs/development/version-control/install-gitlab-with-docker/index.md b/docs/development/version-control/install-gitlab-with-docker/index.md new file mode 100644 index 00000000000..3c065ad72f4 --- /dev/null +++ b/docs/development/version-control/install-gitlab-with-docker/index.md @@ -0,0 +1,252 @@ +--- +author: + name: Linode Community + email: docs@linode.com +description: 'How to install GitLab with Docker.' +keywords: ['gitlab', 'git', 'docker'] +license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' +published: 2019-01-11 +modified: 2019-01-11 +modified_by: + name: Linode +title: "Install GitLab with Docker" +contributor: + name: Linode +external_resources: +- '[GitLab EE Docker Image](https://hub.docker.com/r/gitlab/gitlab-ee)' +- '[GitLab Docker Documentation](https://docs.gitlab.com/omnibus/docker/)' +- '[GitLab SSL Configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#lets-encrypt-integration)' +--- + +[GitLab](https://gitlab.com/) is a free Git repository management application, like GitHub or Bitbucket, that you can run on your own Linode. This guide will show you how to install GitLab using the official GitLab Docker image. + +The GitLab application has a number of services it depends on, including PostgreSQL, Nginx, and Redis. A major benefit of using Docker to install GitLab is that these dependencies are isolated to a single easy-to-update and self-contained image. + +## Before You Begin + +### Choose An Appropriately Sized Linode + +GitLab is a resource-intensive application. To get the most out of GitLab, we recommend a Linode with at least 8GB of memory and at least 2 CPU cores. For more information on system requirements, visit the [GitLab Hardware Requirements page](https://docs.gitlab.com/ce/install/requirements.html#hardware-requirements). + +{{< note >}} +This guide was written for and tested with Ubuntu 18.04. You may be able to adapt this guide to other operating systems supported by Docker. When following this guide under another OS, use the Docker [installation instructions](https://docs.docker.com/install/) for that OS. +{{< /note >}} + +### Secure your Server + +Review and implement the measures in the [How to Secure your Server](/docs/security/securing-your-server/) guide, including creating a [limited user account](/docs/security/securing-your-server/#add-a-limited-user-account). + +### Change your Linode's Default SSH Port + +One of GitLab's features is the ability for you to push and fetch code changes to and from your repository over SSH. When installing GitLab, the software will need to bind to port 22, which is the standard port for SSH. Your system's SSH service already runs on this port by default, so you will receive an error from GitLab if you don't address this conflict. + +To fix this, you'll want to change the port that your system's SSH service listens on. This can be accomplished by editing your Linode's `/etc/ssh/sshd_config` file and changing the `Port` assignment. The example snippet below changes the port from 22 to port 26: + +{{< file "/etc/ssh/sshd_config" >}} +... +Port 26 +... +{{< /file >}} + +When editing the file, you may also need to uncomment the `Port` line by removing the `#` character from the start of the line, if one is present. After updating this file and saving the change, restart the SSH service: + + sudo systemctl restart sshd + +Close your current SSH session and create a new one, making sure to specify the new port. You can do this by supplying the `-p` flag: + + ssh your_limited_user@192.0.2.2 -p 26 + +### (Optional) Update your DNS Records + +Assign a domain or subdomain to your GitLab server. This step is optional, as you can always access GitLab via your server's IP address. However, using a domain is necessary if you would like to take advantage of GitLab's built in SSL support, which uses [Let's Encrypt](https://letsencrypt.org/) to issue certificates. This guide's examples will use `gitlab.example.com`. + +It takes some time for DNS changes to propagate through the internet, so it's suggested that you do this before you set up GitLab. There are several options for updating your DNS records: + +- If you already use Linode's name servers, or if you would like to use them for your domain, review the [DNS Manager](/docs/platform/manager/dns-manager/) guide. You will need to set up an *A record* which is assigned your Linode's IP address. + +- If you use a different DNS provider, review that provider's documentation for setting up a new A record. + + {{< content "update-dns-at-common-name-server-authorities" >}} + +You can test to see if your DNS changes have propagated with the [`dig` command](/docs/networking/dns/use-dig-to-perform-manual-dns-queries/): + + dig +short gitlab.example.com + +{{< output >}} +192.0.2.2 +{{< /output >}} + +Once your changes have propagated, you can move forward with the installation. + +### Install Docker +You must have Docker installed on your Linode to continue. + +{{< content "install-docker-ce" >}} + +## Install the GitLab EE Image + +After installing Docker, download the latest GitLab Enterprise Edition [Docker image](https://hub.docker.com/r/gitlab/gitlab-ee/) from DockerHub. This image contains everything GitLab needs in order to run: PostgreSQL, Nginx, Redis, etc. To download the image, run the following `pull` command: + + sudo docker pull gitlab/gitlab-ee:latest + +{{< disclosure-note "Community Edition or Enterprise Edition?" >}} +The GitLab Enterprise Edition software does not actually require you to have a license to use it. If you do not supply a license after installation, it will automatically show you the GitLab Community Edition feature set instead. + +If you'd like, you can instead opt to download GitLab Community Edition. This will offer the same features as an unlicensed Enterprise Edition installation. The key difference between these software packages is that the features of the EE installation can be upgraded at any time by entering a license. + +The primary reason someone might download the Community Edition is if they prefer to only download open source software. For more information on GitLab's licensing, review the [GitLab article](https://about.gitlab.com/install/ce-or-ee/) on this subject. To download the GitLab CE Docker image, run this command: + + sudo docker pull gitlab/gitlab-ce:latest + +{{< /disclosure-note >}} + +It may take a few minutes to download the image. When the download is complete, you can view a list of all installed Docker images with the `images` command: + + sudo docker images + +## Configure and Run GitLab + +In order to configure and run the GitLab container, you need to provide a few options at runtime. + +1. Consider the following command, a version of which you will use to start the GitLab container: + + sudo docker run --detach \ + --hostname gitlab.example.com \ + --publish 443:443 --publish 80:80 --publish 22:22 \ + --name gitlab-linode \ + --restart always \ + --volume /srv/gitlab/config:/etc/gitlab \ + --volume /srv/gitlab/logs:/var/log/gitlab \ + --volume /srv/gitlab/data:/var/opt/gitlab \ + --env GITLAB_OMNIBUS_CONFIG="external_url 'https://gitlab.example.com/';" \ + gitlab/gitlab-ee:latest + + {{< disclosure-note "Descriptions for each option" >}} +`--detach` runs the Docker container as a background process, as opposed to running it in the foreground. + +`--hostname` defines the container's internal hostname. + +`--publish` tells the container to publish ports, or ranges of ports, to the host. Because GitLab accepts connections on the HTTP (80), HTTPS (443), and SSH (22) ports, this option is declared three times. If you wanted to access GitLab from a non-standard port on your host, you would provide the host port first, and the container port second after the semi-colon. For instance if you wanted to access GitLab SSH on port 3333, you would write `--publish 3333:22`. + +`--name` allows you to apply a label to your container, for use when referencing the container within a Docker network. + +`--restart` specifies a restart policy for the container. Here it is set to `always`, meaning that the container, if exited, will automatically be restarted. + +`--volume` defines the host mounted volumes the container uses to store persistent data. These three volumes store application data, log files, and configuration files. The value to the left of the the semi-colon is the local location, and the value to the right is the container location. + +`--env` supplies the variable `GITLAB_OMNIBUS_CONFIG`, which can hold a series of values, separated by a colon, that correspond to the [GitLab Omnibus configuration settings](https://docs.gitlab.com/omnibus/settings/configuration.html). In this case, an external URL is supplied. Some additional settings might include SMTP configuration values so that GitLab can send activity emails. + +As of GitLab 10.7, if you provide an external URL with a HTTPS protocol, GitLab will automatically set up SSL certificates using Let's Encrypt, and all traffic will be forwarded to HTTPS. For more information about this functionality, read the [GitLab SSL Documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#primary-gitlab-instance) + +As an alternative to specifying the `GITLAB_OMNIBUS_CONFIG` variable via the `--env` option, you can edit the GitLab configuration file directly. For more instructions on how to do that, visit the [Configure GitLab documentation](https://docs.gitlab.com/omnibus/docker/#configure-gitlab). +{{< /disclosure-note >}} + +1. In the above command, replace the values for the `--hostname` option and for the `external_url` configuration setting with the domain or subdomain for your GitLab site. If you did not set up DNS for your site, enter `http://your_linode_ip` (not `https`) for the `external_url` setting. Then, run the command. + + {{< note >}} +If you are using the GitLab Community Edition image, replace `gitlab/gitlab-ee:latest` with `gitlab/gitlab-ce:latest` +{{< /note >}} + + The container may take a few moments to start. After it starts, you'll be given a container ID like the following: + + {{< output >}} +1093d89f9a0af8e4c79e0352e57721b09050d07c86c37d601145a856f3ed1502 +{{< /output >}} + +1. It will take an additional few minutes to be able to access GitLab in your browser after the container starts. You can find out more information about the startup process by monitoring the logs: + + sudo docker logs -f gitlab-linode + + To exit from the log monitoring process, enter `CTRL-C`. This will not stop the container from running. + +1. Load the GitLab site in your web browser. If you try to load it too shortly after starting the container, you may see an HTTP 502 error. If this happens, try waiting for a few more minutes and then refresh your page. + +1. The first time you access the site it will prompt you to enter an administrative password. Enter a complex password and record it somewhere safe. + +1. Log in to your GitLab site by entering `root` as the user along with the password you created in the previous step. + +## Create your First Project + +Each repository in GitLab belongs to a *project*. A project includes: a repository for your files, an issues tracker, a section for merge requests, a wiki, continuous integration and continuous delivery (CI/CD) pipelines, and other features to support your development. + +1. To create your first repository, click **Create a project**. + + ![From the welcome screen, click "Create a project"](gitlab-docker-welcome-screen.png) + +1. You will be taken to the **New Project** page. Enter the project name. You can optionally alter the project's slug, enter a description, or change the visibility of the project. Once you're done, click **Create project**. + + ![Fill out the required information to make a new project](gitlab-docker-create-project.png) + +1. Once your project has been created, you'll be provided with an empty project repository: + + ![An empty project on GitLab](gitlab-docker-empty-project.png) + +1. If you didn't have GitLab create a `README.md` file during project setup, instructions on how to start using your repository from the command line will be shown. + + Enter those commands on your computer to add a new `README.md` to your repository and push it back up to your GitLab repository. Change the domain in the `git clone` command to your site's domain: + + git clone https://gitlab.example.com/testuser/example-project.git + cd example-project + touch README.md # Or create the file in your editor and enter a project description + git add README.md + git commit -m "add README" + git push -u origin master + +## Manage the GitLab Container + +To view all of your running containers, you can issue the `ps` command: + + sudo docker ps + +To stop the GitLab container, issue the `stop` command by supplying the container ID you procured with the `ps` command, or supply the container name: + + sudo docker stop gitlab-linode + +To start a stopped container, issue the `start` command by supplying the container ID or container name: + + sudo docker start gitlab-linode + +Once the container has stopped, you can remove the container using the `rm` command, again supplying the container ID or container name: + + sudo docker container rm gitlab-linode + +{{< note >}} +Removing the container will not delete your projects and repositories. +{{< /note >}} + +## Upgrading GitLab + +To upgrade GitLab to the newest version, you must stop and remove the container, pull the newest image, and then recreate the container: + + sudo docker stop gitlab-linode + sudo docker rm gitlab-linode + sudo docker pull gitlab/gitlab-ee:latest + + sudo docker run --detach \ + --hostname gitlab.example.com \ + --publish 443:443 --publish 80:80 --publish 22:22 \ + --name gitlab-linode \ + --restart always \ + --volume /srv/gitlab/config:/etc/gitlab \ + --volume /srv/gitlab/logs:/var/log/gitlab \ + --volume /srv/gitlab/data:/var/opt/gitlab \ + --env GITLAB_OMNIBUS_CONFIG="external_url 'https://gitlab.example.com/';" \ + gitlab/gitlab-ee:latest + +Remember to provide your own hostname, name, and external URL. If you are using GitLab Community Edition, specify the `gitlab/gitlab-ce:latest` image instead. + +## Next Steps + +GitLab offers many features that are worth taking the time to understand and utilize. Here are a few next steps to take after you've completed this guide: + +- [Upload an SSH key](https://docs.gitlab.com/ee/gitlab-basics/create-your-ssh-keys.html) to your GitLab account so that you can transfer files over SSH. + +- Explore [CI/CD pipelines](https://docs.gitlab.com/ee/ci/) to streamline your development practices. + +- Using your `root` GitLab account, explore the [Admin settings](https://docs.gitlab.com/ee/administration/) to customize the functionality of GitLab. + +- Review Linode's Git documentation: + + - [Getting Start with Git](/docs/development/version-control/how-to-configure-git/) + + - [How to Install Git on Linux, Mac or Windows](/docs/development/version-control/how-to-install-git-on-linux-mac-and-windows/) \ No newline at end of file diff --git a/docs/email/postfix/email-with-postfix-dovecot-and-mysql/index.md b/docs/email/postfix/email-with-postfix-dovecot-and-mysql/index.md index 89624a8bbf0..e2a9e4d89cb 100644 --- a/docs/email/postfix/email-with-postfix-dovecot-and-mysql/index.md +++ b/docs/email/postfix/email-with-postfix-dovecot-and-mysql/index.md @@ -5,7 +5,7 @@ author: description: 'Setting up a mail server with Postfix, Dovecot, and MySQL.' keywords: ["email", "mail", "server", "postfix", "dovecot", "mysql", "debian", "ubuntu", "dovecot 2"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -modified: 2018-07-11 +modified: 2019-01-11 modified_by: name: Linode published: 2013-05-13 @@ -81,8 +81,8 @@ Make a note of the certificate and key locations on the Linode. You will need th This guide uses the following package versions: -* Postfix 3.1.0 -* Dovecot 2.2.22 +* Postfix 3.3.0 +* Dovecot 2.2.33.2 * MySQL 14.14 ## MySQL @@ -288,15 +288,46 @@ smtpd_tls_security_level = may smtpd_sasl_security_options = noanonymous, noplaintext smtpd_sasl_tls_security_options = noanonymous +# Authentication +smtpd_sasl_type = dovecot +smtpd_sasl_path = private/auth +smtpd_sasl_auth_enable = yes + +# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for +# information on enabling SSL in the smtp client. + +# Restrictions +smtpd_helo_restrictions = + permit_mynetworks, + permit_sasl_authenticated, + reject_invalid_helo_hostname, + reject_non_fqdn_helo_hostname +smtpd_recipient_restrictions = + permit_mynetworks, + permit_sasl_authenticated, + reject_non_fqdn_recipient, + reject_unknown_recipient_domain, + reject_unlisted_recipient, + reject_unauth_destination +smtpd_sender_restrictions = + permit_mynetworks, + permit_sasl_authenticated, + reject_non_fqdn_sender, + reject_unknown_sender_domain +smtpd_relay_restrictions = + permit_mynetworks, + permit_sasl_authenticated, + defer_unauth_destination + # See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for # information on enabling SSL in the smtp client. -smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination + myhostname = example.com alias_maps = hash:/etc/aliases alias_database = hash:/etc/aliases mydomain = example.com myorigin = $mydomain -mydestination = $myhostname localhost.$mydomain +mydestination = localhost relayhost = mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 mailbox_size_limit = 0 @@ -313,6 +344,31 @@ virtual_mailbox_maps = mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf virtual_alias_maps = mysql:/etc/postfix/mysql-virtual-alias-maps.cf, mysql:/etc/postfix/mysql-virtual-email2email.cf +# Even more Restrictions and MTA params +disable_vrfy_command = yes +strict_rfc821_envelopes = yes +#smtpd_etrn_restrictions = reject +#smtpd_reject_unlisted_sender = yes +#smtpd_reject_unlisted_recipient = yes +smtpd_delay_reject = yes +smtpd_helo_required = yes +smtp_always_send_ehlo = yes +#smtpd_hard_error_limit = 1 +smtpd_timeout = 30s +smtp_helo_timeout = 15s +smtp_rcpt_timeout = 15s +smtpd_recipient_limit = 40 +minimal_backoff_time = 180s +maximal_backoff_time = 3h + +# Reply Rejection Codes +invalid_hostname_reject_code = 550 +non_fqdn_reject_code = 550 +unknown_address_reject_code = 550 +unknown_client_reject_code = 550 +unknown_hostname_reject_code = 550 +unverified_recipient_reject_code = 550 +unverified_sender_reject_code = 550 {{< /file >}} 1. The `main.cf` file declares the location of `virtual_mailbox_domains`, `virtual_mailbox_maps`, and `virtual_alias_maps` files. These files contain the connection information for the MySQL lookup tables created in the [MySQL](#mysql) section of this guide. Postfix will use this data to identify all domains, corresponding mailboxes, and valid users. @@ -404,7 +460,7 @@ smtp inet n - n - - smtpd #smtpd pass - - - - - smtpd #dnsblog unix - - - - 0 dnsblog #tlsproxy unix - - - - 0 tlsproxy -submission inet n - - - - smtpd +submission inet n - y - - smtpd -o syslog_name=postfix/submission -o smtpd_tls_security_level=encrypt -o smtpd_sasl_auth_enable=yes @@ -505,7 +561,7 @@ auth_mechanisms = plain login For reference, [view a complete `10-auth.conf` file](/docs/assets/1238-dovecot_10-auth.conf.txt). {{< /note >}} -1. Edit the `/etc/dovecot/conf.d/auth-sql.conf.ext` file with authentication and storage information. Ensure your file contains the following lines and that they are uncommented: +1. Edit the `/etc/dovecot/conf.d/auth-sql.conf.ext` file with authentication and storage information. Ensure your file contains the following lines. Make sure the `passdb` section is uncommented, that the `userdb` section that uses the `static` driver is uncommented and update with the right argument, and comment out the `userdb` section that uses the `sql` driver: {{< file "auth-sql.conf.ext" >}} ... @@ -514,6 +570,11 @@ passdb { args = /etc/dovecot/dovecot-sql.conf.ext } ... +#userdb { +# driver = sql +# args = /etc/dovecot/dovecot-sql.conf.ext +#} +... userdb { driver = static args = uid=vmail gid=vmail home=/var/mail/vhosts/%d/%n @@ -704,6 +765,10 @@ You can set up an email client to connect to your mail server. Many clients dete See [Install SquirrelMail on Ubuntu 16.04](/docs/email/clients/install-squirrelmail-on-ubuntu-16-04-or-debian-8/) for details on installing an email client. +{{< note >}} +The Thunderbird email client will sometimes have trouble automatically detecting account settings when using Dovecot. After it fails to detect the appropriate account settings, you can set up your email account manually. Add in the appropriate information for each setting, using the above values, leaving no setting on **Auto** or **Autodetect**. Once you have entered all the information about your mail server and account, press **Done** rather **Re-Test** and Thunderbird should accept the settings and retrieve your mail. +{{< /note >}} + ## Adding New Domains, Email Addresses, and Aliases To add new domains, email addresses, and aliases to the mailserver you will need to update the corresponding MySQL tables created in the [MySQL](#mysql) section of this guide. diff --git a/docs/email/postfix/postfix-smtp-debian7/index.md b/docs/email/postfix/postfix-smtp-debian7/index.md index 9966eb70665..2b0152f2af9 100644 --- a/docs/email/postfix/postfix-smtp-debian7/index.md +++ b/docs/email/postfix/postfix-smtp-debian7/index.md @@ -3,17 +3,20 @@ author: name: Linode Community email: docs@linode.com description: 'Learn how to use Postfix to send mail through an external SMTP server.' -keywords: ["Postfix", " Debian 7", " SMTP", " Email", " Mail"] +keywords: ["Postfix", " Debian", " SMTP", " Email", " Mail"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' contributor: name: Santiago Ti -modified: 2014-05-30 +modified: 2019-01-24 modified_by: name: Linode published: 2014-05-30 title: Configure Postfix to Send Mail Using an External SMTP Server --- +{{< note >}} +This guide was originally written for Debian 7. It has since been tested to work with Debian 9. +{{}} There are many reasons why you would want to configure Postfix to send email using an external SMTP provider such as Mandrill, SendGrid, Amazon SES, or any other SMTP server. One reason is to avoid getting your mail flagged as spam if your current server's IP has been added to a spam list. @@ -21,6 +24,7 @@ There are many reasons why you would want to configure Postfix to send email usi In this tutorial, you will learn how to install and configure a Postfix server to send email through Mandrill, or SendGrid. + ## Updated Guide for Gmail and Google Apps We've got an updated version of this guide that works with Gmail's new security features! @@ -31,7 +35,6 @@ If you're using Gmail or Google Apps, see our [Configure Postfix to Send Mail Us Before starting this tutorial, you should have: -- Debian 7 installed on your Linode - Your fully qualified domain name (FQDN) - All updates installed : @@ -167,6 +170,10 @@ The fastest way to test your configuration is to send an email to any unrelated echo "body of your email" | mail -s "This is a Subject" -a "From: you@example.com" recipient@elsewhere.com +You may have to install `mailutils` to use the `mail` command: + + sudo apt-get install mailutils + Alternatively, you can use Postfix's own sendmail implementation, by entering lines similar to those shown below: sendmail recipient@elsewhere.com diff --git a/docs/networking/dns/common-dns-configurations/index.md b/docs/networking/dns/common-dns-configurations/index.md index 8550a1701d1..c8d4bd03bda 100644 --- a/docs/networking/dns/common-dns-configurations/index.md +++ b/docs/networking/dns/common-dns-configurations/index.md @@ -4,10 +4,10 @@ author: email: docs@linode.com description: 'Configurations for common DNS records.' og_description: 'This guide explains how to use the Linode DNS manager, to configure DNS records' -keywords: ["dns"] +keywords: ["dns", "dnssec"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' aliases: ['dns-manager/','dns-guides/configuring-dns-with-the-linode-manager/'] -modified: 2018-05-22 +modified: 2019-01-28 modified_by: name: Linode published: 2015-01-20 @@ -50,4 +50,4 @@ To route email to a third-party email service, create MX records that associate A *wildcard* DNS record matches requests for non-existent domain names. For example, if you create an A record for `*.example.org`, and a user visits `nonexistantname.example.org`, that user will be redirected to `example.org`. An example wildcard DNS record is shown below. -[![Create a new A record, following the instructions in the "Adding" section. Add a single asterisk (\*) in the "Hostname" field. Set your IP address in the "IP Address" field. Then click the "Save Changes" button.](1127-dns16.png)](1127-dns16.png) \ No newline at end of file +[![Create a new A record, following the instructions in the "Adding" section. Add a single asterisk (\*) in the "Hostname" field. Set your IP address in the "IP Address" field. Then click the "Save Changes" button.](1127-dns16.png)](1127-dns16.png) diff --git a/docs/networking/dns/using-your-systems-hosts-file/index.md b/docs/networking/dns/using-your-systems-hosts-file/index.md index 9f7cf293bef..4d74ca39e6e 100644 --- a/docs/networking/dns/using-your-systems-hosts-file/index.md +++ b/docs/networking/dns/using-your-systems-hosts-file/index.md @@ -3,9 +3,9 @@ author: name: Linode email: docs@linode.com description: 'Mapping aliases, domains or hostnames to IP addresses using the system host file.' -keywords: ["hosts", "hosts file", "hostname", "alias"] +keywords: ["hosts", "hosts file", "hostname", "alias", "dnsec"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -modified: 2018-05-22 +modified: 2019-01-28 modified_by: name: Linode published: 2017-09-14 diff --git a/docs/networking/ssh/ssh-connections-using-putty-on-windows/using-putty.png b/docs/networking/ssh/ssh-connections-using-putty-on-windows/using-putty.png deleted file mode 100644 index cce1506d735..00000000000 Binary files a/docs/networking/ssh/ssh-connections-using-putty-on-windows/using-putty.png and /dev/null differ diff --git a/docs/platform/disk-images/copying-a-disk-image-over-ssh/1.png b/docs/networking/ssh/using-ssh-on-windows/cygwin-intsall-openssh.png similarity index 100% rename from docs/platform/disk-images/copying-a-disk-image-over-ssh/1.png rename to docs/networking/ssh/using-ssh-on-windows/cygwin-intsall-openssh.png diff --git a/docs/platform/disk-images/copying-a-disk-image-over-ssh/2.png b/docs/networking/ssh/using-ssh-on-windows/cygwin-openssh-options.png similarity index 100% rename from docs/platform/disk-images/copying-a-disk-image-over-ssh/2.png rename to docs/networking/ssh/using-ssh-on-windows/cygwin-openssh-options.png diff --git a/docs/networking/ssh/ssh-connections-using-putty-on-windows/index.md b/docs/networking/ssh/using-ssh-on-windows/index.md similarity index 80% rename from docs/networking/ssh/ssh-connections-using-putty-on-windows/index.md rename to docs/networking/ssh/using-ssh-on-windows/index.md index aabedde5372..0024e213f01 100644 --- a/docs/networking/ssh/ssh-connections-using-putty-on-windows/index.md +++ b/docs/networking/ssh/using-ssh-on-windows/index.md @@ -2,25 +2,40 @@ author: name: Linode email: docs@linode.com -description: 'Accessing remote servers with PuTTY, a free and open source SSH client for Windows and UNIX systems.' -keywords: ["putty", "putty ssh", "windows ssh client"] +description: 'Accessing remote servers with SSH on Windows systems.' +keywords: ["ssh", "windows", "putty", "cygwin", "openssh"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['networking/using-putty/'] -modified: 2018-08-20 +aliases: ['networking/using-putty/','networking/ssh/ssh-connections-using-putty-on-windows/'] +modified: 2019-01-14 modified_by: name: Linode published: 2009-09-20 -title: SSH Connections Using PuTTY on Windows -external_resources: - - '[PuTTY Documentation](http://www.chiark.greenend.org.uk/~sgtatham/putty/docs.html)' - - '[Xming Manual](http://www.straightrunning.com/XmingNotes/manual.php)' +title: Using SSH on Windows --- -![Using PuTTY](using-putty.png "Using PuTTY") +![Using SSH on Windows](using-ssh-on-windows.png "Using SSH on Windows") -PuTTY is a free and open source SSH client for Windows and UNIX systems. It provides easy connectivity to any server running an SSH daemon, so you can work as if you were logged into a console session on the remote system. +## OpenSSH +As of late 2018, [OpenSSH](https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_overview) is included with some versions of Windows. + +## Cygwin + +Cygwin is a utility for running popular Linux and BSD tools on Windows. It's often used as an SSH client and/or server solution on Windows systems. + +1. Download and install [Cygwin](https://www.cygwin.com/). + +1. Search for the OpenSSH package and install it. + + ![Cygwin install OpenSSH](cygwin-intsall-openssh.png "Install OpenSSH with Cygwin") -## Install PuTTY and Connect to a Remote Host +1. You should then have SSH capability. + + ![Cygwin OpenSSH Options](cygwin-openssh-options.png "Cygwin OpenSSH Options") + + +## PuTTY + +PuTTY is a free and open source SSH client for Windows and UNIX systems. It provides easy connectivity to any server running an SSH daemon, so you can work as if you were logged into a console session on the remote system. 1. Download and run the PuTTY installer from [here](http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html). @@ -54,7 +69,7 @@ For the fingerprint of an RSA key instead of elliptical curve, use: `ssh-keygen **If the fingerprints do not match, do not connect to the server!** You won't receive further warnings unless the key presented to PuTTY changes for some reason. Typically, this should only happen if you reinstall the remote server's operating system. If you receive this warning again from a system you already have the host key cached on, you should not trust the connection and investigate matters further. -## Port Forwarding (SSH Tunnels) with PuTTY +### SSH Tunneling/Port Forwarding SSH tunnels allow you to access network services running on a remote server though a secure channel. This is useful in cases where the service you wish to access doesn't run over SSL, or you do not wish to allow public access to it. As an example, you can use tunneling to securely access a MySQL server running on a remote server. @@ -70,7 +85,7 @@ To do so: Once you've connected to the remote server with this tunnel configuration, you'll be able to direct your local MySQL client to `localhost:3306`. Your connection to the remote MySQL server will be encrypted through SSH, allowing you to access your databases without running MySQL on a public IP. -## Run Remote Graphical Applications over SSH +### Remote Graphical Applications over SSH PuTTY can securely run graphical applications hosted on a remote Linux server. You can run virtually any X11 application in this manner, and the connection will be encrypted through SSH, providing a safe means of interacting with remote graphical systems. @@ -80,7 +95,7 @@ PuTTY can securely run graphical applications hosted on a remote Linux server. Y You will need the `xauth` package installed on your Linode for X11 forwarding to work correctly. It is installed by default on Debian and RedHat based systems, but may not be for other Linux distributions. {{< /note >}} -2. Tell PuTTY to forward X11 connections to your desktop: +1. Tell PuTTY to forward X11 connections to your desktop: 1. In PuTTY's configuration window, make sure the remote server's hostname or IP, and the correct port, are entered on the **Session** category. 1. In the **Connection** category, go to **SSH**, then **X11**. @@ -90,6 +105,6 @@ You will need the `xauth` package installed on your Linode for X11 forwarding to ![Configure X11 forwarding in PuTTY.](putty-x11-forwarding.png "PuTTY Configuration for X11 connections.") -3. Once you're logged into the remote server, you may start any graphical application hosted there. The application will be projected onto your local desktop. Here's the `xcalc` application running on a Windows desktop from a remote server: +1. Once you're logged into the remote server, you may start any graphical application hosted there. The application will be projected onto your local desktop. Here's the `xcalc` application running on a Windows desktop from a remote server: - ![xcalc running in PuTTY.](162-putty-03-xcalc-running.png "xcalc running in PuTTY.") + ![xcalc running in PuTTY.](xcalc-putty.png "xcalc running in PuTTY.") diff --git a/docs/networking/ssh/ssh-connections-using-putty-on-windows/putty-port-forwarding.png b/docs/networking/ssh/using-ssh-on-windows/putty-port-forwarding.png similarity index 100% rename from docs/networking/ssh/ssh-connections-using-putty-on-windows/putty-port-forwarding.png rename to docs/networking/ssh/using-ssh-on-windows/putty-port-forwarding.png diff --git a/docs/networking/ssh/ssh-connections-using-putty-on-windows/putty-session-window.png b/docs/networking/ssh/using-ssh-on-windows/putty-session-window.png similarity index 100% rename from docs/networking/ssh/ssh-connections-using-putty-on-windows/putty-session-window.png rename to docs/networking/ssh/using-ssh-on-windows/putty-session-window.png diff --git a/docs/networking/ssh/ssh-connections-using-putty-on-windows/putty-verify-host-ssh-key-fingerprint.png b/docs/networking/ssh/using-ssh-on-windows/putty-verify-host-ssh-key-fingerprint.png similarity index 100% rename from docs/networking/ssh/ssh-connections-using-putty-on-windows/putty-verify-host-ssh-key-fingerprint.png rename to docs/networking/ssh/using-ssh-on-windows/putty-verify-host-ssh-key-fingerprint.png diff --git a/docs/networking/ssh/ssh-connections-using-putty-on-windows/putty-x11-forwarding.png b/docs/networking/ssh/using-ssh-on-windows/putty-x11-forwarding.png similarity index 100% rename from docs/networking/ssh/ssh-connections-using-putty-on-windows/putty-x11-forwarding.png rename to docs/networking/ssh/using-ssh-on-windows/putty-x11-forwarding.png diff --git a/docs/networking/ssh/using-ssh-on-windows/using-ssh-on-windows.png b/docs/networking/ssh/using-ssh-on-windows/using-ssh-on-windows.png new file mode 100644 index 00000000000..b22eecfb38b Binary files /dev/null and b/docs/networking/ssh/using-ssh-on-windows/using-ssh-on-windows.png differ diff --git a/docs/networking/ssh/ssh-connections-using-putty-on-windows/162-putty-03-xcalc-running.png b/docs/networking/ssh/using-ssh-on-windows/xcalc-putty.png similarity index 100% rename from docs/networking/ssh/ssh-connections-using-putty-on-windows/162-putty-03-xcalc-running.png rename to docs/networking/ssh/using-ssh-on-windows/xcalc-putty.png diff --git a/docs/networking/vpn/set-up-a-hardened-openvpn-server/index.md b/docs/networking/vpn/set-up-a-hardened-openvpn-server/index.md index 19b643ff7d8..21c62c6c4ab 100644 --- a/docs/networking/vpn/set-up-a-hardened-openvpn-server/index.md +++ b/docs/networking/vpn/set-up-a-hardened-openvpn-server/index.md @@ -6,7 +6,7 @@ description: 'Learn how to securely tunnel your traffic with OpenVPN and OpenSSL keywords: ["openvpn", "vpn", "vpn tunnel", "openssl"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' aliases: ['networking/vpn/set-up-a-hardened-openvpn-server-on-debian-8/'] -modified: 2017-09-26 +modified: 2019-01-29 modified_by: name: Linode published: 2015-12-09 @@ -72,6 +72,14 @@ This series assumes your VPN will operate over IPv4 only. If you instead wish to 4. Add IPv4 rules: `iptables-persistent` stores its rulesets in the files `/etc/iptables/rules.v4` and `/etc/iptables/rules.v6`. Open the `rules.v4` file and replace everything in it with the information below: + {{< note >}} +By default, Linode distribution images are built with network interfaces renamed to `eth0`. If you are using a custom distribution, verify the name of your network interface, first: + + ip link show + +Replace any instances of `eth0` with the name of your network interface. + {{}} + {{< file "/etc/iptables/rules.v4" >}} *filter diff --git a/docs/networking/vpn/set-up-wireguard-vpn-on-ubuntu/index.md b/docs/networking/vpn/set-up-wireguard-vpn-on-ubuntu/index.md index c1cfe38b0e4..741fe4cc755 100644 --- a/docs/networking/vpn/set-up-wireguard-vpn-on-ubuntu/index.md +++ b/docs/networking/vpn/set-up-wireguard-vpn-on-ubuntu/index.md @@ -4,85 +4,65 @@ author: email: docs@linode.com description: 'Wireguard encrypts your traffic quickly and safely, this guide will show you how to set up WireGuard VPN server and clients.' og_description: 'This guide will show you how to install WireGuard, a fast and secure VPN, on Linode.' -keywords: ['wireguard','vpn'] +keywords: ['wireguard','vpn', 'ubuntu'] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' published: 2017-11-14 -modified: 2018-10-16 +modified: 2019-01-22 modified_by: name: Linode title: "Set Up WireGuard VPN on Ubuntu" contributor: name: Sunit Nandi - link: --- ![Set Up WireGuard VPN on Ubuntu](wireguard-vpn-ubuntu-title.jpg "Set Up WireGuard VPN on Ubuntu") -[WireGuard](https://www.wireguard.com) is a simple, fast, and modern VPN that utilizes state-of-the-art cryptography. It aims to be faster and leaner than other VPN protocols such as OpenVPN and IPSec, and has a much smaller source code footprint. WireGuard is still under development, but even in its unoptimized state it is faster than the popular OpenVPN protocol and delivers lower ping times in comparison. +[WireGuard](https://www.wireguard.com) is a simple, fast, and secure VPN that utilizes state-of-the-art cryptography. With a small source code footprint, it aims to be faster and leaner than other VPN protocols such as OpenVPN and IPSec. WireGuard is still under development, but even in its unoptimized state it is faster than the popular OpenVPN protocol. -The WireGuard configuration is as simple to configure as SSH. A connection is established by an exchange of public keys between server and client, and only a client with its public key present in its server configuration file is considered authorized. WireGuard sets up standard network interfaces (such as `wg0` and `wg1`), which behave much like the commonly found `eth0` interface. This makes it possible to configure and manage WireGuard interfaces using standard tools such as `ifconfig` and `ip`. +The WireGuard configuration is as simple as setting up SSH. A connection is established by an exchange of public keys between server and client. Only a client that has its public key in its corresponding server configuration file is allowed to connect. WireGuard sets up standard network interfaces (such as `wg0` and `wg1`), which behave much like the commonly found `eth0` interface. This makes it possible to configure and manage WireGuard interfaces using standard tools such as `ifconfig` and `ip`. -Currently, WireGuard is only available on Linux. This guide will configure a simple peer connection between a server, which will be a Linode running Ubuntu 16.04, and a client. The client can be either your local computer or another Linode. +Currently, WireGuard is only available on Linux. This guide will configure a simple peer connection between a Linode running Ubuntu 18.04, and a client. The client can be either your local computer or another Linode. {{< caution >}} Do not use WireGuard for critical applications. The project is still undergoing security testing and is likely to receive frequent critical updates in the future. {{< /caution >}} -## Update Kernel -Wireguard requires using Ubuntu's kernel rather than the Linode kernel. Follow these steps after creating your Linode: +## Before You Begin -1. In the Linode Manager, open the configuration profile for your Linode and find the **Boot Settings** section. Select **GRUB 2** as your kernel. +- You will need root access to your Linode, or a user account with `sudo` privilege. +- Set your system's [hostname](/docs/getting-started/#set-the-hostname). - ![Configuration Profile](wireguard-config.png) - -2. Boot your Linode, then connect to it with SSH and complete the standard setup procedure in our [Getting Started](/docs/getting-started) guide. - -3. Update your system: - - sudo apt update && sudo apt upgrade - -4. Update the kernel: - - sudo apt install linux-image-virtual grub2 - -5. Reboot your system from the Linode Manager. - -6. Install kernel headers: - - sudo apt install linux-headers-$(uname -r) ## Install WireGuard -1. Install `software-properties-common`: - - sudo apt install software-properties-common - -2. Add the Wireguard repository to your sources list: +1. Add the Wireguard repository to your sources list. Apt will then automatically update the package cache. sudo add-apt-repository ppa:wireguard/wireguard -3. Install Wireguard and its dependencies: +1. Install Wireguard. The `wireguard` package will install all necessary dependencies. + + sudo apt install wireguard - sudo apt update - sudo apt install wireguard-dkms wireguard-tools + DKMS will then build the Wireguard kernel module. If successful, you'll see the following output: - If the steps in the previous section were completed successfully, after the installation you will see the following console output: + {{< output >}} +wireguard: +Running module version sanity check. + - Original module + - No original module exists within this kernel + - Installation + - Installing to /lib/modules/4.15.0-43-generic/updates/dkms/ - wireguard: - Running module version sanity check. - - Original module - - No original module exists within this kernel - - Installation - - Installing to /lib/modules/4.4.0-98-generic/updates/dkms/ +depmod................... - depmod.... +DKMS: install completed. +Setting up wireguard (0.0.20181218-wg1~bionic) ... +Processing triggers for libc-bin (2.27-3ubuntu1) ... +{{< /output >}} - DKMS: install completed. - Setting up wireguard-tools (0.0.20171111-wg1~xenial) ... - Processing triggers for libc-bin (2.23-0ubuntu9) ... - {{< note >}} -If the installation completes but this output does not appear, your kernel is most likely not configured correctly (to double check, the output of `modprobe lsmod && lsmod | grep wireguard` should not be blank). Refer to the previous section to troubleshoot. + {{< note >}} +If the installation completes but the output does not appear, your kernel is most likely not configured correctly. To double check, issue the `lsmod | grep wireguard` command. Its output should not be blank. Refer to the previous section to troubleshoot. {{< /note >}} ## Configure WireGuard Server @@ -94,123 +74,163 @@ If the installation completes but this output does not appear, your kernel is mo This will save both the private and public keys to your home directory; they can be viewed with `cat privatekey` and `cat publickey` respectively. -2. Open `/etc/wireguard/wg0.conf` in a text editor and add the following content, replacing `` with the generated private key: +1. Create the file `/etc/wireguard/wg0.conf` and add the contents indicated below. You'll need to enter your server's private key in the `PrivateKey` field, and its IP addresses in the `Address` field. {{< file "/etc/wireguard/wg0.conf" conf >}} [Interface] PrivateKey = -Address = 192.168.2.1/24, fd86:ea04:1115::1/64 +Address = 203.0.113.5/24, fd86:ea04:1115::1/64 ListenPort = 51820 PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE; ip6tables -A FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -A POSTROUTING -o eth0 -j MASQUERADE PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE; ip6tables -D FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE SaveConfig = true {{< /file >}} - **Address** defines the private IPv4 and IPv6 addresses for the WireGuard server. Each peer in the VPN network should have a unique value for this field. This guide will use the 192.168.2.0/24 address block for IPv4 and the `fd86:ea04:1115::0/64` block for IPv6. + - **Address** defines the private IPv4 and IPv6 addresses for the WireGuard server. Each peer in the VPN network should have a unique value for this field. - **ListenPort** specifies the port that WireGuard will use for incoming connections. + - **ListenPort** specifies which port WireGuard will use for incoming connections. - **PostUp** and **PostDown** set steps to be run after the interface is turned on or off, respectively. In this case, `iptables` is used to set Linux IP Masquerade rules to allow all the clients to share the server's Internet IPv4 and IPv6 address, and clear the rules once the tunnel is down. + - **PostUp** and **PostDown** defines steps to be run after the interface is turned on or off, respectively. In this case, `iptables` is used to set Linux IP masquerade rules to allow all the clients to share the server's IPv4 and IPv6 address. The rules will then be cleared once the tunnel is down. - **SaveConfig** tells the config file to automatically update whenever a new peer is added while the service is running. + - **SaveConfig** tells the configuration file to automatically update whenever a new peer is added while the service is running. ## Set Up Firewall Rules 1. Allow SSH connections and WireGuard's VPN port: - ufw allow 51820/udp - ufw allow 22/tcp - ufw enable + sudo ufw allow 22/tcp + sudo ufw allow 51820/udp + sudo ufw enable -2. Verify the settings: +1. Verify the settings: - ufw status verbose + sudo ufw status verbose -## Start Wireguard Service +## Start the Wireguard Service 1. Start Wireguard: wg-quick up wg0 {{< note >}} -`wg-quick` is a convenient wrapper around many of the common functions in `wg`. You can turn off the wg0 interface with `wg-quick down wg0` +`wg-quick` is a convenient wrapper for many of the common functions in `wg`. You can turn off the wg0 interface with `wg-quick down wg0` {{< /note >}} -2. Enable the Wireguard service to automatically restart on boot: +1. Enable the Wireguard service to automatically restart on boot: - systemctl enable wg-quick@wg0 + sudo systemctl enable wg-quick@wg0 + +1. Check if the VPN tunnel is running with the following two commands: + + sudo wg show -3. Check if the VPN tunnel is running: + You should see a similar output: + + {{< output >}} +user@ubuntu:~$ sudo wg show +interface: wg0 + public key: vD2blmqeKsV0OU0GCsGk7NmVth/+FLhLD1xdMX5Yu0I= + private key: (hidden) + listening port: 51820 +{{< /output >}} - wg show ifconfig wg0 + Your output should resemble the following: + + {{< output >}} +user@ubuntu:~$ ifconfig wg0 +wg0: flags=209 mtu 1420 + inet 203.0.113.5 netmask 255.255.255.0 destination 203.0.113.5 + inet6 fd86:ea04:1115::1 prefixlen 64 scopeid 0x0 + unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 1000 (UNSPEC) + RX packets 0 bytes 0 (0.0 B) + RX errors 0 dropped 0 overruns 0 frame 0 + TX packets 0 bytes 0 (0.0 B) + TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 +{{< /output >}} + + ## Wireguard Client -The process for setting up a client is essentially the same as the server. If your client uses Ubuntu, follow the instructions above. For other Linux distributions please refer to the [WireGuard docs](https://www.wireguard.com/install/) for installation instruction. +The process for setting up a client is similar to setting up the server. When using Ubuntu as your client's operating system, the only difference between the client and the server is the contents of the configuration file. If your client uses Ubuntu, follow the steps provided in the above sections and in this section. For installation instructions on other operating systems, see the [WireGuard docs](https://www.wireguard.com/install/). 1. Generate a key pair for the client if you have not already: umask 077 wg genkey | tee privatekey | wg pubkey > publickey -2. The only difference between the client and server is the contents of the configuration file--the client's `wg0.conf` must contain its IP addresses. +1. The main difference between the client and the server's configuration file, `wg0.conf`, is it must contain *its own* IP addresses and does not contain the `ListenPort`, `PostUP`, `PostDown`, and `SaveConfig` values. {{< file "/etc/wireguard/wg0.conf" conf >}} [Interface] PrivateKey = -Address = 192.168.2.2/24, fd86:ea04:1115::5/64 +Address = 203.0.123.12/24, fd86:ea04:1115::5/64 {{< /file >}} ## Connect the Client and Server There are two ways to add peer information to WireGuard; this guide will demonstrate both methods. -1. The first method is to directly edit the client's `wg0.conf` file with the server's information: +1. The first method is to directly edit the client's `wg0.conf` file with the server's public key, public IP address, and port: {{< file "/etc/wireguard/wg0.conf" conf >}} [Peer] PublicKey = Endpoint = :51820 -AllowedIPs = 192.168.2.1/24, fd86:ea04:1115::1/64 +AllowedIPs = 203.0.123.12/24, fd86:ea04:1115::5/64 {{< /file >}} - Use the server's public key, public IP address, and port. - -3. Enable the `wg` service: +1. Enable the `wg` service: wg-quick up wg0 systemctl enable wg-quick@wg0 -The second way of adding peer information is through the command line. This information will be added to the config file automatically because of the SaveConfig option specified earlier. +1. The second way of adding peer information is using the command line. This information will be added to the config file automatically because of the `SaveConfig` option specified in the Wireguard server's configuration file. + + Run the following command from the server. Replace the example IP addresses with those of the client: + + sudo wg set wg0 peer endpoint :51820 allowed-ips 203.0.123.12/24,fd86:ea04:1115::5/64 + +1. Verify the connection. This command can be run from the client or the server: -1. Run the following command from the server, where the example IP addresses are those of the client: + sudo wg - wg set wg0 peer endpoint :51820 allowed-ips 192.168.2.2/24,fd86:ea04:1115::5/64 + Regardless of which method you choose to add peer information to WireGuard, there should be a **Peer** section in the output of the `sudo wg` command if the setup was successful. -2. Verify the connection: + {{< output >}} +user@localhost:~$ sudo wg +interface: wg0 + public key: vD2blmqeKsV0OU0GCsGk7NmVth/+FLhLD1xdMX5Yu0I= + private key: (hidden) + listening port: 51820 - wg +peer: iMT0RTu77sDVrX4RbXUgUBjaOqVeLYuQhwDSU+UI3G4= + endpoint: 203.0.123.12:51820 + allowed ips: 203.0.123.12/24, fd86:ea04:1115::/64 +{{< /output >}} -Regardless of which method you choose, there will be a **Peer** section in the output of this command if the setup was successful. This Peer section will be automatically added to `wg0.conf` when the service is restarted. If you would like to add this information immediately to the config file, you can run: + This Peer section will be automatically added to `wg0.conf` when the service is restarted. If you would like to add this information immediately to the config file, you can run: - wg-quick save wg0 + wg-quick save wg0 -Additional clients can be added using the same procedure. + Additional clients can be added using the same procedure. ## Test the Connection -Return to the client and ping the server: +1. Return to the client and ping the server: - ping 192.168.2.1 - wg + ping 192.168.2.1 + sudo wg -The last two lines of the output of wg should be similar to: + The last two lines of the output from running the `wg` command should be similar to: + {{< output >}} latest handshake: 1 minute, 17 seconds ago transfer: 98.86 KiB received, 43.08 KiB sent + {{}} -This indicates that you now have a private connection between the server and client. You can also ping the client from the server to verify that the connection works both ways. + This indicates that you now have a private connection between the server and client. You can also ping the client from the server to verify that the connection works both ways. ## Next steps diff --git a/docs/networking/vpn/set-up-wireguard-vpn-on-ubuntu/wireguard-config.png b/docs/networking/vpn/set-up-wireguard-vpn-on-ubuntu/wireguard-config.png deleted file mode 100644 index 72ac472bf31..00000000000 Binary files a/docs/networking/vpn/set-up-wireguard-vpn-on-ubuntu/wireguard-config.png and /dev/null differ diff --git a/docs/platform/api/how-to-create-an-oauth-app-with-the-linode-python-api-library/index.md b/docs/platform/api/how-to-create-an-oauth-app-with-the-linode-python-api-library/index.md new file mode 100644 index 00000000000..064ee6e9a11 --- /dev/null +++ b/docs/platform/api/how-to-create-an-oauth-app-with-the-linode-python-api-library/index.md @@ -0,0 +1,543 @@ +--- +author: + name: Linode Community + email: docs@linode.com +description: 'Create an OAuth 2 app utilizing the Linode API through the Linode APIv4 Python library.' +keywords: ['linode','api','python','library','oauth'] +license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' +published: 2019-01-21 +modified: 2019-01-21 +modified_by: + name: Linode +title: "How To Create an OAuth App with the Linode Python API Library" +contributor: + name: Linode +external_resources: +- '[Linode APIv4 Python library documentation](https://linode-api4.readthedocs.io/en/latest/index.html)' +- '[Linode APIv4 Python library example app repository](https://github.com/linode/linode_api4-python/tree/master/examples/install-on-linode)' +--- + +Linode supports the OAuth 2 authorization protocol. OAuth 2 allows a user to safely grant a third-party app permission to act on their behalf. This means that a user could authorize an app to access data and / or make changes to their Linode account and services that are exposed by the [Linode APIv4](https://developers.linode.com/api/v4). For example, an app could create or destroy Linodes, manage a NodeBalancer, or alter a domain. + +This guide will show you how to create a simple OAuth application using [Flask](http://flask.pocoo.org/) and the [Linode Python API library](https://linode-api4.readthedocs.io/en/latest/index.html). This app allows a user to log in with their Linode account and create a Linode with a StackScript. The complete code for this example is available in the [Linode APIv4 Python library example](https://github.com/linode/linode_api4-python/tree/master/examples/install-on-linode) repository. + +## Before You Begin + +1. Normally, in order to create an OAuth app with Linode your server must have HTTPS enabled. The only exceptions to this rule are `localhost` addresses, which can use HTTP. As this guide is just a primer and is not intended to supply production ready code, we will be working with a local workstation, using `localhost`. If you choose to create an app for production, you will need to generate SSL certificates for HTTPS access. + +1. Ensure that Python 3 is installed on your workstation. + +## Obtaining a Client ID and a Client Secret + +In order for Linode to verify the identity of your app, called a *client*, you will need to generate a set of credentials, specifically a client ID and a client secret. + +1. Log in to the [Linode Cloud Manager](https://cloud.linode.com) and navigate to your Account Profile. + + ![OAuth Account Profile](oauth-account.png) + +1. From there, click on the **My Apps** tab and select **Create My App**. You will be prompted to supply a label for your app and a callback URL. We will discuss the role of the callback URL in depth [later in this guide](#manage-the-oauth-2-callback-url). For now you can supply the following URL: + + http://localhost:5000/auth_callback + + Leave *Public* unchecked and click **Submit**. + + ![OAuth Account Profile](oauth-create-app.png) + +1. A window will appear with your client secret. Copy this down somewhere secure, as once you exit this window you will not be able to retrieve the client secret again, and will be forced to generate a new one. + + ![OAuth Account Profile](oauth-client-secret.png) + +1. Once you exit the client secret window your app will appear as part of a list of apps. Note your client ID, as this is the last piece of information you will need to verify your app's identity. + + ![OAuth Account Profile](oauth-client-id.png) + +In summary, you should have these three bits of information, with values similar to the ones provided here: + +- **Client ID**: ce571a8cdad1ba4a0a7d +- **Client Secret**: fab8e2222e83b9b2f50a76012122ec20a5acb005ed088f3fccda2c9c2c4e1cbd +- **Callback URL**: http://localhost:5000/auth_callback + +## OAuth 2 Authentication Exchange + +The OAuth 2 workflow is a series of exchanges between your third-party app and Linode. Below is an explanation of these exchanges. + +1. The end user visits your client application's website and attempts to login. +1. Your client application redirects the end user to the authentication server ([https://login.linode.com](https://login.linode.com/login)) with your client application’s client ID and requested OAuth scopes, which appear in the URL of the login page. +1. The end user inputs their username and password to the authorization server and authorizes the login. +1. The authorization server redirects the end user back to your client application with a temporary authorization code (sometimes called an exchange code) in the URL. +1. The client application issues a POST request to the authentication server containing the authorization code and the client application’s client secret. +1. The authentication server responds to the client application with a newly issued OAuth access token. + +In the following sections you will write the code to perform each one of these steps, using the Linode Python API library. + +## Setup Your Development Environment + +1. Create a project folder and move into that folder. + + mkdir ~/linode-oauth-project && cd ~/linode-oauth-project + +1. For this project, you will need to use pip to download and install the required Python libraries. Install pip if you do not already have it: + + apt install python-pip + +1. Install the required Python libraries: + + pip install flask flask-session linode_api4 + + +## Configure Your App + +In a text editor, create a file named `config.py`. Add the following variables and values, being sure to change the values to your own. + +The StackScript used in this example is for demo purposes. To explore other available StackScripts, visit the [Linode StackScript Library](https://www.linode.com/stackscripts). Note that the `stackscript_id` does not have quotation marks around it. The `secret key` is used for serializing session data, and should be a value only you know. + +{{< file "config.py" >}} +client_id = 'ce571a8cdad1ba4a0a7d' +client_secret = 'fab8e2222e83b9b2f50a76012122ec20a5acb005ed088f3fccda2c9c2c4e1cbd' +stackscript_id = 320826 +application_name = 'my-application-name' +secret_key = 'my-secret-key' +{{< /file >}} + +## Author an OAuth2 App + +In this section, you will write the code for the app. + +### Include Imports + +Ensure you are in the `linode-oauth-project` directory and create and open a file called `app.py` in the text editor of your choice. Include the following libraries: + +{{< file "app.py" python >}} +import re +from flask import Flask, redirect, request, render_template, session, send_from_directory +from flask_session import Session +from linode_api4 import (LinodeClient, LinodeLoginClient, StackScript, Image, Region, Type, OAuthScopes) + +import config +{{< /file >}} + +### Set Up Flask and Session Key + +Copy in the following code to set up Flask and the session secret key: + +{{< file "app.py" python >}} +... + +app=Flask(__name__) +app.config['SECRET_KEY'] = config.secret_key +{{}} + +### Create a Function to Return the Linode Login Client + +In `app.py` add the following function to return the [LinodeLoginClient class](https://linode-api4.readthedocs.io/en/latest/linode_api4/login_client.html?highlight=linode_login_client). The LinodeLoginClient class is the library's OAuth interface. Note that we are passing the `client_id` and `client_secret` parameters from our `config.py` file to the class: + +{{< file "~/linode-oauth-project/app.py" python >}} +... + +def get_login_client(): + return LinodeLoginClient(config.client_id, config.client_secret) +{{}} + +### Create an Index Route +In Flask you can create HTTP endpoints with *routes*. The index route, defined in the code below at the document root `/`, will be the route the user will see when they navigate to `http://localhost:5000/`. This route will be responsible for displaying the available Linode plan types, the available regions, and the StackScript-compatible images that a user will choose from when creating their new Linode. + +To query a list of available plan types and regions you can use the [LinodeClient class](https://linode-api4.readthedocs.io/en/latest/linode_api4/linode_client.html?highlight=LinodeClient#linodeclient-class), which is an interface for Linode's APIv4. Viewing the Linode plan types and regions does not require any sort of authorization, so you can provide a dummy value of `no-token` to instantiate the class: + +{{< file "~/linode-oauth-project/app.py" python >}} +... + +@app.route('/') +def index(): + client = LinodeClient('no-token') + types = client.linode.types(Type.label.contains("Linode")) + regions = client.regions() + stackscript = StackScript(client, config.stackscript_id) + return render_template('configure.html', + types=types, + regions=regions, + application_name=config.application_name, + stackscript=stackscript + ) +{{< /file >}} + +It is important to note that the two API queries in the above code are slightly different from one another. The `client.regions` method is a top-level method, just as it appears in the [Linode API](https://developers.linode.com/api/v4#tag/Regions). The `client.linode.types` method, on the other hand, is part of the Linode group, which is a collection of methods that deal with Linodes. Again, this is because Linode endpoints are grouped that way in the [API](https://developers.linode.com/api/v4#tag/Linode-Types). Some methods in the Linode Python library are top level, such as `domain_create`, while others, like `networking.ip_assign`, are part of a group. For more information on the top-level methods and groupings, consult the [library documentation](https://linode-api4.readthedocs.io/en/latest/linode_api4/linode_client.html#grouping). + +In addition to querying the API, the above route also renders the `configure.html` template by passing it the types, regions, application name, and StackScript object. The StackScript object contains a list of StackScript compatible images. We will cover templating in a later section. + +### Create a Login Route + +Next, create a login route in `app.py`. This route will perform two functions. First, it will serialize the user's plan type, region, and image selections into the session. + +Second, this route will redirect the user to Linode's login page where they will be prompted to authorize your client app and the *scopes* you have requested for it. Scopes are sets of permissions that define the access level of your client app. For instance, to create a Linode, your end user must authorize the `OAuthScopes.Linodes.create` scope. + +{{< file "~/linode-oauth-project/app.py" python >}} +... + +@app.route('/', methods=["POST"]) +def start_auth(): + login_client = get_login_client() + session['dc'] = request.form['region'] + session['distro'] = request.form['distribution'] + session['type'] = request.form['type'] + return redirect(login_client.generate_login_url(scopes=OAuthScopes.Linodes.create)) +{{< /file >}} + +When the user returns to your app from the Linode login page, they will be directed to the callback URL. + +{{< note >}} +Below is a list of available scopes: + +- OAuthScopes.Linodes +- OAuthScopes.Domains +- OAuthScopes.StackScripts +- OAuthScopes.Users +- OAuthScopes.NodeBalancers +- OAuthScopes.Tokens +- OAuthScopes.IPs +- OAuthScopes.Tickets +- OAuthScopes.Clients +- OAuthScopes.Account +- OAuthScopes.Events +- OAuthScopes.Volumes + +Each scope is broken into five permissions: `view`, `create`, `modify`, `delete`, and `all`. The `all` permission encompasses the other four permissions. +{{}} + +### Manage the OAuth 2 Callback URL + +The OAuth 2 callback URL has two main responsibilities. Its first responsibility is to help prove the identity of the client application. When a user attempts to log in to Linode through OAuth, instead of redirecting the user back to the page they came from, Linode's OAuth implementation matches the client ID to the callback URL you have registered with your app on Linode's system. This ensures that a nefarious third party can't just steal the client ID, which is public, and attempt to authorize their own app with it. + +The callback URL's second responsibility is to kick off the process of exchanging an authorization code for an access token. This second process is done over POST, and so it doesn't require the user to physically leave the page they are returned to after they log in to Linode. Now you will write the code that satisfies this second responsibility. + +In `app.py`, add the following lines: + +{{< file "~/linode-oauth-project/app.py" python >}} +... + +@app.route('/auth_callback') +def auth_callback(): + code = request.args.get('code') + login_client = get_login_client() + token, scopes, _, _ = login_client.finish_oauth(code) + + # ensure we have sufficient scopes + if not OAuthScopes.Linodes.create in scopes: + return render_template('error.html', error='Insufficient scopes granted to deploy {}'\ + .format(config.application_name)) + + (linode, password) = make_instance(token, session['type'], session['dc'], session['distro']) + + get_login_client().expire_token(token) + return render_template('success.html', + password=password, + linode=linode, + application_name=config.application_name + ) +{{< /file >}} + +Let's take a look at what each of the parts of this section does. + +First, a route is defined for the callback with `@app.route()`, then a function called `auth_callback` is defined that will run whenever this route is accessed: + +{{< file "~/linode-oauth-project/app.py" python >}} +... +@app.route('/auth_callback') +def auth_callback(): +... +{{}} + +When the user is returned to the callback URL, an authorization code is appended to the URL. The variable `code` is set to retrieve this value from the URL's request arguments: + +{{< file "~/linode-oauth-project/app.py" python >}} +... + code = request.args.get('code') +... +{{}} + +Then you retrieve an instance of the LinodeLoginClient class: + +{{< file "~/linode-oauth-project/app.py" python >}} +... + login_client = get_login_client() +... +{{}} + +Once you have the LinodeLoginClient class, you can pass the authorization code to the `finish_oauth` method, which is a helper method that will manage the authorization code to OAuth token exchange. This method returns an OAuth token, and the scopes the user has agreed upon. + +{{< file "~/linode-oauth-project/app.py" python >}} +... + token, scopes, _, _ = login_client.finish_oauth(code) +... +{{}} + +The next section compares the scopes your app requested from the user to the scopes returned by Linode's OAuth login page. If the returned scopes do not include the correct scopes, in this case the `OAuthScopes.Linode.create` scope, then an error template is rendered and an error message is displayed: + + +{{< file "~/linode-oauth-project/app.py" python >}} +... + # ensure we have sufficient scopes + if not OAuthScopes.Linodes.create in scopes: + return render_template('error.html', error='Insufficient scopes granted to deploy {}'\ + .format(config.application_name)) +... +{{}} + +Once your app has determined that it has the correct permissions, it creates the Linode using the Linode plan type, the region, and the image that the app serialized into session storage. You will create the `make_instance` function in [the next step](#create-a-function-to-deploy-a-linode). The `make_instance` function returns the `linode` object, which contains the Linode's label, group, and IP address, and the function also returns a randomly generated password: + +{{< file "~/linode-oauth-project/app.py" python >}} +... + (linode, password) = make_instance(token, session['type'], session['dc'], session['distro']) +... +{{}} + +Once the Linode has been created, the app expires the OAuth access token. Expiring tokens after use is a strong security measure but if your app is performing many actions on behalf of the user, you might find that time-based expiration scheme is more suitable to your needs. The app then renders the success template by passing it the `linode` object, the password, and application name: + +{{< file "~/linode-oauth-project/app.py" python >}} +... + get_login_client().expire_token(token) + return render_template('success.html', + password=password, + linode=linode, + application_name=config.application_name + ) +{{}} + +### Create a Function to Deploy a Linode + +Now, create the `make_instance` function that you referenced above: + +{{< file "~/linode-oauth-project/app.py" python >}} +... + +def make_instance(token, type_id, region_id, distribution_id): + client = LinodeClient('{}'.format(token)) + stackscript = StackScript(client, config.stackscript_id) + (linode, password) = client.linode.instance_create(type_id, region_id, + group=config.application_name, + image=distribution_id, stackscript=stackscript.id) + + if not linode: + raise RuntimeError("it didn't work") + return linode, password +{{< /file >}} + +The `make_instance` function takes an OAuth access token, the type ID, the region ID, and the image (Linux distribution) ID as parameters. It creates an instance of the LinodeClient class, and unlike the instance of LinodeClient used earlier in the guide, this one requires an OAuth token because you will be using it to create a Linode. The function then creates a Linode using the `linode.instance_create` method, returning the `linode` object and the password. + +Finally, if there was an error with the creation of the Linode, the `if not linode` statement will raise a runtime error. + +### Set the __name__ Variable + +At the end of your `app.py`, paste in the following code to make sure you can run your app: + +{{< file "~/linode-oauth-project/app.py" python >}} +if __name__ == '__main__': + app.debug=True + app.run() +{{}} + +## Create App Templates + +Now that you have written the backend code for your app, you'll need to create a frontend user interface. Begin by creating a `templates` directory in your project directory and moving into it: + + mkdir ~/linode-oauth-project/templates && cd ~/linode-oauth-project/templates + +Using your preferred text editor, create and open `base.html`. This will be the base template from which your other templates will inherit their stylesheets and JavaScript files: + +{{< file "~/linode-oauth-project/templates/base.html" html >}} + + + Install On Linode + + + + +
+ {% block content %} + {% endblock %} +
+ + + + +{{}} + +The important thing to note in the above template is the Jinja2 templating tags. They are: + + {% block content %} + {% endblock %} + +As you will see, any template that extends the `base.html` template and includes code between the opening and closing `content` block, will render the code laid out by `base.html`. + +Create a file called `configure.html`, which will be the UI a user will see when they reach the document root endpoint (`/`). Copy in the following code: + +{{< file "templates/configure.html" html >}} +{% extends 'base.html' %} +{% block content %} +
+
+

Deploy {{application_name}} to a Linode

+

+ This will create a brand new Linode running {{application_name}} on your + account and give you the credentials. +

+
+
+
+ + +
+
+ + +
+
+ + +
+
+
+ +
+
+ +{% endblock %} +{{}} + +Here the template begins with two statements: `{% extends 'base.html' %}` and a `{% block content %}` statement. These two tags tell Jinja2 to extend the code within `base.html`, and to place everything within `{% block content %} ... {% endblock %}` in `configure.html` between the corresponding `{% block content %} ... {% endblock %}` tags in `base.html`. + +`configure.html` includes Jinja2 logic, with the inclusion of `for` statements like `{% for o in regions %}`. These statements are like `for` statements in other languages, and are used to iterate over an array or list. In this example, it is iterating over the regions that [we passed to the template from the index route](#create-an-index-route). `configure.html` also contains variables, which are denoted by double curly brackets: `{{ s.id }}`. + +Create another file called `error.html`. This will be the template that appears whenever there is an error in the Linode deployment. Copy in the following code: + +{{< file "templates/error.html" html >}} +{% extends 'base.html' %} +{% block content %} +
+

Error

+

{{error}}

+
+
+ Try Again +
+{% endblock %} +{{}} + +This template works the same way that `configure.html` does, by extending `base.html` and providing its own `content` block. + +Lastly, create another file called `success.html`. This file follows the pattern set by `configure.html` and `error.html`, and will present the user with a confirmation message whenever a Linode is successfully created. This message includes the Linode's label, group, IP address, and password: + +{{< file "templates/success.html" html >}} +{% extends 'base.html' %} +{% block content %} +
+

Success!

+

{{application_name}} has been deployed to {{linode.label}} in the {{linode.group}} group.

+
+
+
+

You can access your Linode with the following command:

+ ssh root@{{linode.ipv4[0]}} +
+
+

Your root password is:

+ {{password}} +
+
+{% endblock %} +{{}} + +## Run Your App + +You are now ready to run your app. Change back to your project's main directory: + + cd ~/linode-oauth-project + +Run the `app.py` script: + + python3 app.py + +Open your browser to the following URL: + + http://localhost:5000/ + +You should be greeted with your new app. Select a plan, a region, and an image to deploy a Linode using the Linode API Python library. + +## Next Steps + +The app you've created shows off some of the aspects of the Linode API Python library. You can use `LinodeLoginClient` to authorize your OAuth app with the appropriate scopes, and can create Linodes through the use of `LinodeClient`. + +In extending this app, you might want to add multiple functionalities, like creating NodeBalancers from a list of available Linodes, or managing domains. To achieve this goal you'll probably want to separate the login logic from the Linode creation logic. One way to do this would be store the OAuth token in the session, implementing a time-based expiration mechanism to expire your tokens instead. \ No newline at end of file diff --git a/docs/platform/api/how-to-create-an-oauth-app-with-the-linode-python-api-library/oauth-account.png b/docs/platform/api/how-to-create-an-oauth-app-with-the-linode-python-api-library/oauth-account.png new file mode 100644 index 00000000000..b8f1391d76f Binary files /dev/null and b/docs/platform/api/how-to-create-an-oauth-app-with-the-linode-python-api-library/oauth-account.png differ diff --git a/docs/platform/api/how-to-create-an-oauth-app-with-the-linode-python-api-library/oauth-client-id.png b/docs/platform/api/how-to-create-an-oauth-app-with-the-linode-python-api-library/oauth-client-id.png new file mode 100644 index 00000000000..180946b6e3f Binary files /dev/null and b/docs/platform/api/how-to-create-an-oauth-app-with-the-linode-python-api-library/oauth-client-id.png differ diff --git a/docs/platform/api/how-to-create-an-oauth-app-with-the-linode-python-api-library/oauth-client-secret.png b/docs/platform/api/how-to-create-an-oauth-app-with-the-linode-python-api-library/oauth-client-secret.png new file mode 100644 index 00000000000..4b221c68885 Binary files /dev/null and b/docs/platform/api/how-to-create-an-oauth-app-with-the-linode-python-api-library/oauth-client-secret.png differ diff --git a/docs/platform/api/how-to-create-an-oauth-app-with-the-linode-python-api-library/oauth-create-app.png b/docs/platform/api/how-to-create-an-oauth-app-with-the-linode-python-api-library/oauth-create-app.png new file mode 100644 index 00000000000..3caa3daa780 Binary files /dev/null and b/docs/platform/api/how-to-create-an-oauth-app-with-the-linode-python-api-library/oauth-create-app.png differ diff --git a/docs/platform/automating-server-builds/index.md b/docs/platform/automating-server-builds/index.md index 7dcb24f8571..1b8f7054767 100644 --- a/docs/platform/automating-server-builds/index.md +++ b/docs/platform/automating-server-builds/index.md @@ -2,103 +2,110 @@ author: name: Linode email: docs@linode.com -description: Our guide to automating server builds with the Linode Manager +description: Our guide to automating server builds with the Linode Manager. keywords: ["server builds", "disks", "golden disk", "puppet", "chef"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -modified: 2015-04-15 +modified: 2019-01-11 modified_by: - name: Alex Fornuto + name: Linode published: 2013-06-28 title: Automating Server Builds --- -If you run a large website that requires multiple servers, or have a general interest in server automation, you may want to automate your server builds. You can rapidly spin up multiple servers with exactly the same configuration by creating a *golden image* that can be cloned to multiple Linodes with the intention of eliminating server discrepancies. - -Server configuration can also be automated through [Stackscripts](https://www.linode.com/stackscripts). View the [Stackscripts](/docs/platform/stackscripts/) guide for more information. - ## Why You Should Automate Server Builds -When you set up a Linode for the first time, you manually install packages and applications. For example, in the [Hosting a Website](/docs/websites/hosting-a-website/) guide, the Apache, MySQL, and PHP packages are installed. Manually installing packages is a good way to learn about virtual servers, but it's also a time-consuming process. - -It is recommended that you take steps to automate the server-provisioning process, even if you don't need multiple Linodes at this moment. By duplicating the disk or writing an install StackScript, you'll preserve the current state of your server -- including all of the packages you've installed and settings you've configured. If you want to spin up another Linode in the future, your automatic server building process will save you time. - -## Golden Disk - -The idea behind a golden disk is simple: Create the perfect image and then save it for cloning to other servers. To get started, set up a new Linode, install the desired packages, configure the settings, and then test the configuration. Once satisfied with the server configuration, shut down the Linode, duplicate the disk, and then clone it to all of your other Linodes, either manually or though [the Linode API](http://www.linode.com/api/linode/linode.clone). - -{{< note >}} -Be aware that certain files like `/etc/hosts`, `/etc/hostname`, and static networking configurations may need to be modified for individual Linodes. -{{< /note >}} - -[![Cloning your Linode disk.](1303-image_cloning_2.jpg)](1303-image_cloning_2.jpg) - -There are several places to store a golden disk: - -- **Linode Images:** [Linode Images](/docs/platform/disk-images/linode-images/) allows you to take snapshots of your disks, and then deploy them to any Linode under your account. -- **Linode Backup Service:** After enabling the Linode Backup Service, you can [make a manual backup](/docs/platform/linode-backup-service/#take-a-manual-snapshot) of your Linode (called a "snapshot"). This snapshot can function as your golden disk. Instead of cloning a disk to new Linodes, you can simply restore them from the snapshot backup. -- **Dedicated Linode:** Boot the Linode, make the desired changes, and clone the disk again. -- **Existing Linode:** You can clone from an existing Linode, but you will need to power down the Linode to ensure a consistent copy. -- **Different Computer:** You can transfer the disk to another computer. For instructions, see our guide on [Copying a Disk Over SSH](/docs/platform/disk-images/copying-a-disk-image-over-ssh/). - -These methods are discussed in further detail below, with the exception of [Linode Images](/docs/platform/disk-images/linode-images/) which has its own article. +Manually configuring systems is a good way to learn, but it's also a time consuming process which is prone to human error. There are multiple ways to automate deploying new systems and various degrees to which that automation can be applied. +For example, if your needs are relatively straightforward and concise, a shell script or Linode [StackScript](https://www.linode.com/stackscripts) could be all that is necessary. For more complex solutions, configuration orchestration and management exists to deploy and manage fleets of systems and services across multiple regions, networks, and service providers. -### Linode Backup Service +## Working with a Golden Image -If you subscribe to the [Linode Backup Service](http://www.linode.com/backups/), you can create a golden disk by making a manual backup of an existing disk. Take a *snapshot* of the disk to back it up, and then restore it to your other Linodes. There's no need to clone or resize the disk. The snapshot will be stored until you overwrite it with another backup. +Using a *golden image* as a configuration base is a frequent starting point in cloud environment automation. This helps quickly deploy multiple systems which are exactly identical. Across the industry, golden images are also referred to as *master*, *base*, or *clone* images, among other terms. Irrespective of name, the idea behind a golden disk is simple: create the desired image and preserve it for cloning/deploying to other servers, thereby simplifying the deployment process and eliminating configuration gap. -1. Use the Linode's existing disk, or [create a new disk](/docs/platform/disk-images/disk-images-and-configuration-profiles/#creating-a-disk-with-a-linux-distribution-installed). -2. Install all necessary packages and configure the system settings. -3. Verify that all installed packages are current. See [Monitoring and Maintaining Your Server](/docs/uptime/monitoring-and-maintaining-your-server/#updating-software) for instructions. -4. Test your server configuration. At a minimum, this probably includes downloading your version-controlled repository and verifying that your website or application functions properly. -5. Take a snapshot of the disk. See the [manual snapshot](/docs/platform/linode-backup-service/#take-a-manual-snapshot) instructions for more information. -6. Restore your other Linodes from the snapshot. The disk can be restored to as many Linodes as you like. See the [backup restore](/docs/platform/linode-backup-service/#restore-from-a-backup) instructions for more information. +![Cloning your Linode disk.](1303-image_cloning_2.jpg) +### Create a Golden Image +1. Create a new Linode. +1. Configure all packages, applications, and system settings as desired. +1. Remove any system users you don't want to appear on your duplicated systems. +1. Shut down the Linode and either: + - [Duplicate the disk](/docs/platform/disk-images/disk-images-and-configuration-profiles/#duplicating-a-disk). + - Alternatively, [take a snapshot](/docs/platform/disk-images/linode-backup-service/#take-a-manual-snapshot) of the disk with Linode Backups. +1. Store your golden image. This can be done in a variety of ways. A few examples are: + - As a snapshot using [Linode Images](/docs/platform/disk-images/linode-images/) or [Linode Backups](/docs/platform/linode-backup-service/#take-a-manual-snapshot). + - In a [version control](/docs/development/version-control/introduction-to-version-control/) system running on a remote or local server. + - On [local](/docs/platform/disk-images/copying-a-disk-image-over-ssh/) storage. -### Dedicated Linode +### Restore a Golden Image -A dedicated Linode can be used to store and maintain a golden disk. It can be shut down after you've created the disk, and then boot it to update the image. - -1. Use an existing Linode, or [set up a new one](/docs/getting-started/#sign-up). -2. Install all necessary packages and configure the system settings, if you haven't already done so. -3. Verify that all installed packages are current. See our [Monitoring and Maintaining Your Server](/docs/uptime/monitoring-and-maintaining-your-server/#updating-software) guide for instructions. -4. Test your server configuration. At a minimum, this probably includes downloading your version-controlled repository and verifying that your website or application functions properly. -5. Shut down your Linode. -6. [Clone the disk](/docs/platform/disk-images/disk-images-and-configuration-profiles/#cloning-disks-and-configuration-profiles) to another Linode. You can also clone the configuration profile. The disk can be cloned to as many Linodes as you like. - - -### Existing Linode - -You can create and store a golden disk using an *existing Linode*, with some drawbacks: All of the disks stored on the Linode will need to be resized to fit within your target Linode's allocated storage space, and you will have to shut down the Linode to ensure an accurate clone, which will result in downtime. - -1. Use the Linode's existing disk, or [create a new disk](/docs/platform/disk-images/disk-images-and-configuration-profiles/#creating-a-disk-with-a-linux-distribution-installed). -2. Install all necessary packages and configure the system settings, if you haven't already done so. -3. Verify that all installed packages are current. See our [Monitoring and Maintaining Your Server](/docs/uptime/monitoring-and-maintaining-your-server/#updating-software) guide for instructions. -4. Test your server configuration. At a minimum, this probably includes downloading your version-controlled repository and verifying that your website or application functions properly. -5. Shut down the Linode. -6. Resize the disk. See our [Resizing a Disk](/docs/platform/disk-images/disk-images-and-configuration-profiles/#resizing-a-disk) guide for instructions. -7. If you're planning on using the golden disk on the existing Linode, you should duplicate the disk. See our [Duplicating a Disk](/docs/platform/disk-images/disk-images-and-configuration-profiles/#duplicating-a-disk) guide for instructions. -8. [Clone the disk](/docs/platform/disk-images/disk-images-and-configuration-profiles/#cloning-disks-and-configuration-profiles) to another Linode. You can also clone the configuration profile. The disk can be cloned to as many Linodes as you like. - - -### Updating the Hostname and IP Address - -After you restore or clone a disk to another Linode, you may need to change its hostname and IP address. For instructions on changing the hostname, see [Setting the Hostname](/docs/getting-started/#setting-the-hostname). If the golden disk was configured to use a static IP address, you'll also need to replace the IP address. See [Static IP Configuration](/docs/networking/linux-static-ip-configuration/#static-network-configuration) for more information. +1. Copy the duplicate disk to your other Linodes, either using [the Linode API](https://developers.linode.com/api/v4#operation/cloneLinodeDisk) or [manually](/docs/platform/disk-images/copying-a-disk-image-to-a-different-account/#copying-the-disk). If you're using a Linode Backups snapshot, you would [restore it](/docs/platform/disk-images/linode-backup-service/#restore-from-a-backup) to the desired Linodes. +1. Create [configuration profiles](/docs/platform/disk-images/disk-images-and-configuration-profiles/#creating-a-configuration-profile) on those additional Linodes to boot using the duplicated disk. +1. Any user credentials from the golden image will also be on the duplicated disks so you should change the new system's root password. +1. Update the new Linode's [hostname](/docs/getting-started/#setting-the-hostname). +1. If your golden system was configured to use a static IP address, you'll also need to [reconfigure the IP address](/docs/networking/linux-static-ip-configuration/#static-network-configuration) on your duplicated disks. ## Third-Party Tools -Golden disks are capable of handling automated server builds for most individuals and small businesses, but if you work for a large business that manages dozens of Linodes, you may need to turn to a third-party configuration management tool such as: +Golden disks are capable of handling automated server builds for most individuals and small businesses, but if you work for a large business that manages dozens of Linodes, you may need to turn to third-party configuration management and orchestration tools, such as: - **Puppet:** An open source configuration management tool that manages systems declaratively. It can automates IT tasks like application configuration, patch management, and even infrastructure audit and compliance. See the following Puppet guides: - [Basic Puppet Setup and Configuration](/docs/websites/puppet/basic-puppet-setup-and-configuration/) - [Manage and Automate Systems Configuration with Puppet](/docs/websites/puppet/manage-and-automate-systems-configuration-with-puppet/) - -- **Chef:** An open source configuration management tool that allows you to "turn your infrastructure into code." See the [Chef website](https://www.chef.io/) for more information. The [knife Linode](https://github.com/chef/knife-linode) subcommand can also be used to manage Linodes with Chef. - -- **Ansible:** A "radically simple" open source platform for configuring and managing systems. It works by connecting to your systems via SSH — it doesn't install anything on the remote systems. See the [AnsibleWorks website](http://www.ansible.com/) for more information. Read more about the [Linode Module from Ansible](http://docs.ansible.com/ansible/latest/linode_module.html) in the official documentation. + - [Use Puppet Modules to Create a LAMP Stack](/docs/applications/configuration-management/use-puppet-modules-to-create-a-lamp-stack/) + - [Install and Manage MySQL Databases with Puppet Hiera on Ubuntu 16.04](/docs/applications/configuration-management/install-and-manage-mysql-databases-with-puppet-hiera-on-ubuntu-16-04/) + +- **Chef:** An open source configuration management tool used to turn your infrastructure into code. See the [Chef website](https://www.chef.io/) for more information. The [knife Linode](https://github.com/chef/knife-linode) subcommand can also be used to manage Linodes with Chef. See the following Chef guides to get started: + + {{< note >}} +Knife Linode is based on Linode's deprecated APIv3. + {{}} + + - [A Beginner's Guide to Chef](https://linode.com/docs/applications/configuration-management/beginners-guide-chef/) + - [Creating Your First Chef Cookbook](/docs/applications/configuration-management/creating-your-first-chef-cookbook/) + - [Install a Chef Server Workstation on Ubuntu 18.04](/docs/applications/configuration-management/install-a-chef-server-workstation-on-ubuntu-18-04/) + +- **Ansible:** An open source platform for configuring and managing systems. It works by connecting to your systems via SSH — it doesn't install anything on the remote systems. See the [AnsibleWorks website](http://www.ansible.com/) for more information. Read more about the [Linode Module from Ansible](http://docs.ansible.com/ansible/latest/linode_module.html) in the official documentation. To start using Ansible, check out the following guides: + + {{< note >}} +The Linode Module from Ansible is based on Linode's deprecated APIv3. + {{}} + + - [Learn How to Install Ansible and Run Playbooks](/docs/applications/configuration-management/learn-how-to-install-ansible-and-run-playbooks/) + - [Automatically Configure Servers with Ansible and Playbooks](/docs/applications/configuration-management/automatically-configure-servers-with-ansible-and-playbooks/) + +- **Salt:** Salt (also referred to as SaltStack) is a Python-based configuration management and orchestration system. Salt uses a master/client model in which a dedicated Salt master server manages one or more Salt minion servers. To learn more about Salt, see the following guides: + + - [A Beginner's Guide to Salt](https://www.linode.com/docs/applications/configuration-management/beginners-guide-to-salt/) + - [Getting Started with Salt - Basic Installation and Setup](/docs/applications/configuration-management/getting-started-with-salt-basic-installation-and-setup/) + - [SaltStack Command Line Reference](/docs/applications/configuration-management/salt-command-line-reference/) + - [Introduction to Jinja Templates for Salt](/docs/applications/configuration-management/introduction-to-jinja-templates-for-salt/) + - [Test Salt States Locally with KitchenSalt](/docs/applications/configuration-management/test-salt-locally-with-kitchen-salt/) + - [Secrets Management with Salt](/docs/applications/configuration-management/secrets-management-with-salt/) + - [Use and Modify Official SaltStack Formulas](/docs/applications/configuration-management/use-and-modify-official-saltstack-formulas/) + - [Use Salt States to Configure a LAMP Stack on a Minion](/docs/applications/configuration-management/use-salt-states-to-configure-a-lamp-stack-on-a-minion/) + - [Monitoring Salt Minions with Beacons](/docs/applications/configuration-management/monitoring-salt-minions-with-beacons/) + - [Create a Salt Execution Module](/docs/applications/configuration-management/create-a-salt-execution-module/) + - [Automate Static Site Deployments with Salt, Git, and Webhooks](/docs/applications/configuration-management/automate-a-static-site-deployment-with-salt/) + - [Use Salt States to Create LAMP Stack and Fail2ban Across Salt minions](/docs/applications/configuration-management/use-salt-states-to-create-lamp-stack-and-fail2ban-across-salt-minions/) + - [Configure and Use Salt Cloud and Cloud Maps to Provision Systems](/docs/applications/configuration-management/configure-and-use-salt-cloud-and-cloud-maps-to-provision-systems/) + +- **Terraform:** Terraform by HashiCorp is an orchestration tool that allows you to represent your Linode instances and other resources with declarative code inside configuration files, instead of manually creating those resources via the Linode Manager or API. This practice is referred to as Infrastructure as Code, and Terraform is a popular example of this methodology. + + {{< note >}} +The Terraform Linode provider is based on [Linode's APIv4](https://developers.linode.com/api/v4). + {{}} + + - [A Beginner's Guide to Terraform](/docs/applications/configuration-management/beginners-guide-to-terraform/) + - [Introduction to HashiCorp Configuration Language (HCL)](/docs/applications/configuration-management/introduction-to-hcl/) + - [Use Terraform to Provision Linode Environments](/docs/applications/configuration-management/how-to-build-your-infrastructure-using-terraform-and-linode/) + - [Import Existing Infrastructure to Terraform](/docs/applications/configuration-management/import-existing-infrastructure-to-terraform/) + - [Secrets Management with Terraform](/docs/applications/configuration-management/secrets-management-with-terraform/) + - [Create a NodeBalancer with Terraform](/docs/applications/configuration-management/create-a-nodebalancer-with-terraform/) + - [Deploy a WordPress Site Using Terraform and Linode StackScripts](/docs/applications/configuration-management/deploy-a-wordpress-site-using-terraform-and-linode-stackscripts/) + - [Create a Terraform Module](/docs/applications/configuration-management/create-terraform-module/) There are plenty of other third-party configuration management tools to be used should the above options not suit your needs. diff --git a/docs/platform/disk-images/copy-disk-over-ssh-short/index.md b/docs/platform/disk-images/copy-disk-over-ssh-short/index.md deleted file mode 100644 index d7cad8eabe9..00000000000 --- a/docs/platform/disk-images/copy-disk-over-ssh-short/index.md +++ /dev/null @@ -1,93 +0,0 @@ ---- -author: - name: Linode - email: docs@linode.com -description: "Shortguide for copying a disk over SSH." -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -keywords: [] -modified: 2018-08-10 -modified_by: - name: Linode -title: "Copy Disk Over SSH Shortguide" -published: 2018-08-10 -headless: true ---- - -## Download a Disk over SSH - -Downloading your disk will copy a `.img` file to your computer that encapsulates all of the data that is on your Linode's disk. - -### Prepare the Receiving Computer - -Verify that the receiving computer has SSH installed. (Most Linux/Unix-like systems have it installed by default.) If you're running Windows locally, you may wish to set up the [Cygwin](http://www.cygwin.com/) compatibility layer to provide a reasonably complete Unix-like environment. Instructions on setting up Cygwin are located [here](/docs/platform/disk-images/copying-a-disk-image-over-ssh/#windows-cygwin-instructions). - -### Start Your Linode in Rescue Mode - -Before you initiate the transfer, start the source Linode in *Rescue Mode* and start SSH by following these guides: - -1. [Start your Linode in Rescue Mode](/docs/troubleshooting/rescue-and-rebuild/#booting-into-rescue-mode). -1. [Connecting to a Linode Running in Rescue Mode via LISH](/docs/troubleshooting/rescue-and-rebuild/#connecting-to-a-linode-running-in-rescue-mode). -1. [Start the SSH server on your Linode](/docs/troubleshooting/rescue-and-rebuild/#starting-ssh). - -### Copy the Disk - -Now that the Linode is running in Rescue Mode, you can transfer the disk from the Linode to the receiving machine over SSH: - -1. Enter the following command on the receiving machine. Replace `123.45.67.89` with the Linode's IP address and `/home/archive/linode.img` with the path where you want to store the disk: - - ssh root@123.45.67.89 "dd if=/dev/sda " | dd of=/home/archive/linode.img - - {{< note >}} -The device `/dev/sda` is used for Linodes running on top of KVM. If your Linode is still using XEN, then throughout this guide you must use `/dev/xvda` instead. -{{< /note >}} - -1. The receiving machine will connect to the Linode. Type `yes` and press **Enter** to continue connecting: - - The authenticity of host '123.45.67.89 (123.45.67.89)' can't be established. - RSA key fingerprint is 39:6b:eb:05:f1:28:95:f0:da:63:17:9e:6b:6b:11:4a. - Are you sure you want to continue connecting (yes/no)? yes - -1. Enter the root password for the Linode: - - Warning: Permanently added '123.45.67.89' (RSA) to the list of known hosts. - root@123.45.67.89's password: - - The transfer starts, and you'll see output similar to the following: - - {{< output >}} - 4096000+0 records in - 4096000+0 records out - 2097152000 bytes (2.1 GB) copied, 371.632 seconds, 5.6 MB/s - 4096000+0 records in - 4096000+0 records out - 2097152000 bytes (2.1 GB) copied, 364.002 s, 5.8 MB/s - {{}} - - {{< note >}} -Copying your disk can take a while. Please be patient. If you have a slow internet connection, add the `-C` option to the SSH command; this enables gzip compression for data transfer. If you receive a `Write failed: Broken pipe` error, repeat this process. -{{< /note >}} - -### Verify the Disk - -Once the copy has completed, you can verify it by mounting the image on the receiving machine. - -1. Log in to the receiving machine as `root` by entering the following command and entering the `root` user's password: - - su - -1. Make a directory on the receiving machine by entering the following command: - - mkdir linode - -1. Mount the disk by entering the following command, replacing `linode.img` with the name of the disk: - - mount -o loop linode.img linode - -1. View the directories stored on the disk by entering the following command: - - ls linode/ - - You should see the directories on the disks, similar to the ones shown below, indicating that everything has transferred: - - bin dev home lost+found mnt proc sbin srv tmp var - boot etc lib media opt root selinux sys usr diff --git a/docs/platform/disk-images/copying-a-disk-image-over-ssh/3.png b/docs/platform/disk-images/copying-a-disk-image-over-ssh/3.png deleted file mode 100644 index a54db9481bb..00000000000 Binary files a/docs/platform/disk-images/copying-a-disk-image-over-ssh/3.png and /dev/null differ diff --git a/docs/platform/disk-images/copying-a-disk-image-over-ssh/copying_a_disk_over_ssh_smg.png b/docs/platform/disk-images/copying-a-disk-image-over-ssh/copying_a_disk_over_ssh_smg.png index 7c4f0aa3912..d04b1a2a04b 100644 Binary files a/docs/platform/disk-images/copying-a-disk-image-over-ssh/copying_a_disk_over_ssh_smg.png and b/docs/platform/disk-images/copying-a-disk-image-over-ssh/copying_a_disk_over_ssh_smg.png differ diff --git a/docs/platform/disk-images/copying-a-disk-image-over-ssh/index.md b/docs/platform/disk-images/copying-a-disk-image-over-ssh/index.md index a89a86b0f53..dc5323642c8 100644 --- a/docs/platform/disk-images/copying-a-disk-image-over-ssh/index.md +++ b/docs/platform/disk-images/copying-a-disk-image-over-ssh/index.md @@ -2,124 +2,155 @@ author: name: Linode email: docs@linode.com -description: "Our guide to copying a disk over SSH" -keywords: ["copy", "disk", "ssh"] +description: "Create a disk image using dd and download it to another machine over SSH." +keywords: ["copy", "disk", "ssh", "dd"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['migration/ssh-copy/','migrate-to-linode/disk-images/copying-a-disk-image-over-ssh/'] -modified: 2017-01-08 +aliases: ['migration/ssh-copy/','migrate-to-linode/disk-images/copying-a-disk-image-over-ssh/','platform/disk-images/copying-a-disk-image-over-ssh/'] +modified: 2019-01-14 modified_by: name: Linode published: 2012-06-04 -title: "Copying a Disk Over SSH" +title: "Copy a Disk Over SSH" --- -You can use SSH to copy a Linode's disk to a system that resides on a different network. This is an effective way to back up your Linode's disks to a personal computer or another server. In this guide, you'll learn how to use SSH to copy a Linode's disk to a local system. +![Copy a Disk Over SSH](copying_a_disk_over_ssh_smg.png "Copy a Disk Over SSH") -![Our guide to copying a disk over SSH](copying_a_disk_over_ssh_smg.png "Our guide to copying a disk over SSH") +Piping SSH commands to utilities such as `dd`, `gzip`, or `rsync` is an easy way to copy a Linode's data into a single file for later extraction. This can effectively back up your Linode's disk or migrate your installed system among Linodes. -{{< content "copy-disk-over-ssh-short" >}} +This guide demonstrates how to download a `.img` file to your computer over SSH containing a block-level copy of your Linode's disk device created with `dd`. -You have successfully transferred your Linode's disk to another host using SSH. +## Download a Disk over SSH -## Upload a Disk over SSH - -Once you have a copy of your Linode's disk you may want to upload that copy from your local machine to a Linode in the future. For example, if you previously downloaded your Linode disk and removed the Linode to halt billing on it, you can create a new Linode at a later date and upload the disk to it to resume your services. +### Boot into Rescue Mode -1. Prepare your empty Linode by creating a new disk. Access your Linode through the Linode Manager and select **Create a new disk**: +1. Prepare the receiving computer by verifying that SSH is installed. Most Linux/Unix-like systems include OpenSSH in their package base by default. If the receiving system is Microsoft Windows, there are multiple SSH solutions available such as [Cygwin and PuTTY](/docs/networking/ssh/using-ssh-on-windows). - [![Create a new disk](copydisk-create-disk.png)](copydisk-create-disk-full.png) +1. Reboot Your Linode into [rescue mode](/docs/troubleshooting/rescue-and-rebuild/#booting-into-rescue-mode) and connect to it using [Lish](/docs/platform/manager/remote-access/#console-access). -2. Enter a descriptive name in the **Label** field, and be sure the **Size** is large enough to hold the contents of the disk you are uploading. Click **Save Changes**. +1. Set a root password for the rescue system and start the SSH server: -3. [Boot the Linode in Rescue Mode and start SSH](#starting-your-linode-in-rescue-mode) + passwd + service ssh start -4. From your **receiving machine** issue the following command, replacing `/home/archive/linode.img/` with your disk image's path, and `123.45.67.89` with your Linode's IP. +### Copy and Download the Disk - dd if=/home/archive/linode.img | ssh root@123.45.67.89 "dd of=/dev/sda" +1. Copy the disk over SSH from the Linode to the receiving machine. Replace `192.0.2.9` with the Linode's IP address and `/home/archive/linode.img` with the path where you want to store the disk. - When the transfer is done, you will see an output similar to this: - - 49807360+0 records in - 49807360+0 records out - 25501368320 bytes transferred in 9468.878375 secs (2693177 bytes/sec) - 49807360+0 records in - 49807360+0 records out - 25501368320 bytes (26 GB) copied, 9462.12 s, 2.7 MB/s + ssh root@192.0.2.9 "dd if=/dev/sda " | dd of=/home/archive/linode.img {{< note >}} -Copying your disk can take a while. Please be patient. If you receive a `Write failed: Broken pipe` error, repeat this step. +The device `/dev/sda` is used for Linodes running on KVM. If your Linode is still using XEN, then use `/dev/xvda` throughout this guide instead. {{< /note >}} +1. The receiving machine will connect to the Linode. Verify the SSH key fingerprints. If valid, type `yes` and press **Enter** to continue: + + The authenticity of host '192.0.2.9 (192.0.2.9)' can't be established. + RSA key fingerprint is 39:6b:eb:05:f1:28:95:f0:da:63:17:9e:6b:6b:11:4a. + Are you sure you want to continue connecting (yes/no)? yes + +1. Enter the root password you created above for the rescue system: + + Warning: Permanently added '192.0.2.9' (RSA) to the list of known hosts. + root@192.0.2.9's password: + + When the transfer completes, you'll see a summary output similar to below: + + {{< output >}} + 4096000+0 records in + 4096000+0 records out + 2097152000 bytes (2.1 GB) copied, 371.632 seconds, 5.6 MB/s + {{}} + + Copying your disk can take a while. If you have a slow internet connection, add the `-C` option to the SSH command to enable gzip compression of the disk image. If you receive a `Write failed: Broken pipe` error, repeat this process. + ### Verify the Disk -As above, you will want to verify the disk by mounting it on the receiving Linode. Log in to your Linode through SSH. +Once the copy has completed, verify it by mounting the image on the receiving machine. + +1. Switch users to `root` on receiving machine: + + su -1. Make a new directory for the disk: +1. Make a directory to mount the disk as: mkdir linode -2. Mount the disk: +1. Mount the disk. Replace `linode.img` with the name of the of your Linode's disk. - mount /dev/sda linode + mount -o loop linode.img linode -3. View the directories in the disk: +1. List the directories on the disk to indicate if everything has transferred. Your output of `ls` is similar to below: - ls linode/ + ls linode - The output should be similar to below: + {{< output >}} + bin dev home lost+found mnt proc sbin srv tmp var + boot etc lib media opt root selinux sys usr +{{< /output >}} - bin dev home lib64 media opt root sbin srv tmp var - boot etc lib lost+found mnt proc run selinux sys usr +## Upload a Disk over SSH -### Expand the Filesystem +You may want to upload your disk image to a new server. For example, if you previously downloaded your Linode disk and deleted the Linode to halt billing on it, you can create a new Linode at a later date and upload the disk to resume your services. -If the disk you created on the receiving Linode is larger than the source disk (for example you're transferring a disk from a smaller Linode to a larger Linode), you'll have to resize the filesystem in order to make use of the new space. +1. Prepare the new Linode by first creating a new swap disk. Doing this first means that you can simply use the Linode's remaining space for the system disk without doing any subtraction. A swap disk is typically starts at 256 MB or 512 MB in size, but can be larger or smaller depending upon your needs. -You can check if this is necessary by comparing the space reported by the filesystem: +1. Access your Linode through the Linode Manager. Select **Create a new disk** and select `swap` from the **Type** drop down menu. - # df -h - Filesystem Size Used Avail Use% Mounted on - /dev/sda 24G 19G 4.0G 83% / + [![Create a new disk](copydisk-create-disk.png)](copydisk-create-disk-full.png) -To the space reported by the disk: +1. Now use the remaining disk space to create the system drive you'll copy your disk image to. Enter a descriptive name in the **Label** field, and be sure the **Size** is large enough to hold the contents of the disk you are uploading. Click **Save Changes**. - # lsblk - NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT - sda 8:0 0 30G 0 disk / +1. Reboot Your Linode into [rescue mode](#boot-into-rescue-mode) and start the SSH server as described above. -In the above example, the values in the **Size** column don't match. Although the disk is 30GB (the Linode Manager will also show the disk as 30GB), the filesystem can only see 24G. To use all available space on the new disk, execute the following from Rescue Mode: +1. Upload the disk over SSH to the Linode. Replace `192.0.2.9` with the Linode's IP address and `/home/archive/linode.img` with the disk images's path. - e2fsck -f /dev/sda - resize2fs /dev/sda + dd if=/home/archive/linode.img | ssh root@192.0.2.9 "dd of=/dev/sda" -### Create the Swap Disk + When the transfer completes, you'll see a summary output similar to below: -If your Linode still has enough space for a swap disk, simply [create](/docs/platform/disk-images/disk-images-and-configuration-profiles/#creating-a-blank-disk) one from your Linode's Dashboard by selecting `swap` from the **Type** drop down menu. A swap disk is typically between 256MB and 512MB in size, but can be made larger or smaller depending upon your needs. If all of the free space on your Linode is assigned to the disk you created previously, [resize](/docs/platform/disk-images/disk-images-and-configuration-profiles/#resizing-a-disk) the disk to make enough room for the swap disk, and then follow the steps to [create](/docs/platform/disk-images/disk-images-and-configuration-profiles/#creating-a-blank-disk) it. + {{< output >}} + 49807360+0 records in + 49807360+0 records out + 25501368320 bytes (26 GB) copied, 9462.12 s, 2.7 MB/s +{{< /output >}} + + Copying your disk can take a while. If you receive a `Write failed: Broken pipe` error, repeat this process. -### Boot from the Disk -You will now need to create a new configuration profile on the receiving Linode to boot from. +### Expand the Filesystem -1. From within your Linode Manager, select your Linode and select **Create a New Configuration Profile**. +If the disk you created on the new server is larger than the source disk (for example you're transferring a disk from a smaller Linode to a larger Linode), you'll have to resize the filesystem to make use of the new space. - [![Selecting the configuration profile](1065-migration6-small.png)](1064-migration6.png) +You can check if this is necessary by comparing the space of the filesystem to the space of the new disk: + +{{< output >}} + root@localhost:~# df -h + Filesystem Size Used Avail Use% Mounted on + /dev/sda 24G 19G 4.0G 83% / +{{< /output >}} -2. Enter the name for the configuration profile in the **Label** field, and in the **Block Device Assignment** section set the `/dev/sda` to the new disk you created earlier in this section of the guide. Set `/dev/sdb` to the swap image. Save changes. +{{< output >}} + root@localhost:~# lsblk + NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT + sda 8:0 0 30G 0 disk / +{{< /output >}} -3. Return to the Linode's dashboard manager, and select the configuration profile that you just created. Click **Reboot** to start the Linode using the disk you just transferred. +In the above example, the values in the **Size** column don't match. Although the disk is 30 GB, the filesystem can only see 24 GB. -### Windows Cygwin Instructions +To use all available space on the new disk, execute the following from rescue mode. Replace `/dev/sdx` with your system disk's device identifier (/dev/sda, /dev/sdb, etc.). -1. Download and install [Cygwin](https://www.cygwin.com/) + e2fsck -f /dev/sdx + resize2fs /dev/sdx -2. Install the OpenSSH package, by clicking on the Cygwin install **after** the install succeeds. - ![Cygwin install package](3.png) +### Boot from the Disk -3. Type `openssh` into the search bar, and press install. +You will now need to create a new configuration profile on the receiving Linode. - ![Cygwin Openssh](1.png) +1. Select your Linode and select **Create a New Configuration Profile**. + + [![Selecting the configuration profile](1065-migration6-small.png)](1064-migration6.png) -4. Test if the `ssh` function of Cygwin works: +1. Enter a name for the configuration profile in the **Label** field, and in the **Block Device Assignment** section set the `/dev/sda` to the new system disk you created earlier in this section of the guide. Set `/dev/sdb` to the swap image. - ![Cygwin ssh test](2.png) +1. The Linode is now ready to reboot using the new system disk. \ No newline at end of file diff --git a/docs/platform/disk-images/disk-images-and-configuration-profiles/index.md b/docs/platform/disk-images/disk-images-and-configuration-profiles/index.md index a256cc07e39..ca73034e254 100644 --- a/docs/platform/disk-images/disk-images-and-configuration-profiles/index.md +++ b/docs/platform/disk-images/disk-images-and-configuration-profiles/index.md @@ -3,10 +3,10 @@ author: name: Linode email: docs@linode.com description: Our guide to disks and configuration profiles -keywords: ["disks", "config profiles"] +keywords: ["disks", "config profiles", "disk space"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' aliases: ['disk-images-config-profiles/','migrate-to-linode/disk-images/disk-images-and-configuration-profiles/'] -modified: 2014-06-19 +modified: 2019-01-28 modified_by: name: Linode published: 2012-04-24 @@ -32,11 +32,11 @@ All of a Linode's disks and configuration profiles are displayed on the Linode's [![Overview of Linode Manager interface.](977-disk1-2-small.png)](978-disk1-1.png) 1. These are your configuration profiles. Select one and then click **Reboot** to start your Linode with the settings and disks you have specified in the configuration profile. -2. Click this link to create a new configuration profile. -3. These are your disks. Select one to resize it or duplicate it. -4. Click this link to create a new disk with a Linux distribution installed. -5. Click this link to create a new blank disk. -6. This box indicates how much storage space is allocated to disks. To see how much space inside your disk is currently being used, connect to your Linode via SSH and execute the `df -m` command. +1. Click this link to create a new configuration profile. +1. These are your disks. Select one to resize it or duplicate it. +1. Click this link to create a new disk with a Linux distribution installed. +1. Click this link to create a new blank disk. +1. This box indicates how much storage space is allocated to disks. To see how much space inside your disk is currently being used, connect to your Linode via SSH and execute the `df -m` command. ## Disks @@ -51,22 +51,22 @@ A new configuration profile will automatically be created when you make a disk t {{< /note >}} 1. Log in to the [Linode Manager](https://manager.linode.com). -2. Click the **Linodes** tab. -3. Select a Linode. The Linode's dashboard appears. -4. Select the **Deploy an Image** link. The webpage shown below appears. +1. Click the **Linodes** tab. +1. Select a Linode. The Linode's dashboard appears. +1. Select the **Deploy an Image** link. The webpage shown below appears. -[![Create a new disk with a Linux distribution.](980-disk2-small.png)](979-disk2.png) + [![Create a new disk with a Linux distribution.](980-disk2-small.png)](979-disk2.png) -5. Select a Linux distribution from the **Distribution** menu. +1. Select a Linux distribution from the **Distribution** menu. - {{< note >}} -You can install a customized Linux distribution on your disk by clicking **Deploying using StackScripts**. Some of the most popular StackScripts do things like install Apache, configure a firewall, and set up the WordPress. -{{< /note >}} + {{< note >}} + You can install a customized Linux distribution on your disk by clicking on the **Deploying using StackScripts** link. Some of the most popular StackScripts do things like install Apache, configure a firewall, and set up WordPress. + {{< /note >}} -6. Enter a size for the disk in the **Deployment Disk Size** field. The size must be smaller than the amount of storage space remaining in your Linode plan. -7. By default, the new disk is set to use your existing swap disk. If you'd like to create a new swap disk, select a size from the **Swap Disk** menu. -8. Enter a password for the `root` user in the **Root Password** field. -9. Click **Deploy**. +1. Enter a size for the disk in the **Deployment Disk Size** field. The size must be smaller than the amount of storage space remaining in your Linode plan. +1. By default, the new disk is set to use your existing swap disk. If you'd like to create a new swap disk, select a size from the **Swap Disk** menu. +1. Enter a password for the `root` user in the **Root Password** field. +1. Click **Deploy**. The disk will be created. Watch the *Host Job Queue* on the Dashboard to monitor the progress. When the process is complete, the disk will appear on the dashboard. @@ -75,16 +75,16 @@ The disk will be created. Watch the *Host Job Queue* on the Dashboard to monitor Create a blank disk if you need detachable storage space or want to download and install your own Linux distribution. Here's how: 1. Log in to the [Linode Manager](https://manager.linode.com). -2. Click the **Linodes** tab. -3. Select a Linode. The Linode's dashboard appears. -4. Select the **Create a new disk** link. The *Edit disk* webpage appears, as shown below. +1. Click the **Linodes** tab. +1. Select a Linode. The Linode's dashboard appears. +1. Select the **Create a new disk** link. The *Edit disk* webpage appears, as shown below. -[![Create a new blank disk.](982-disk3-small.png)](983-disk3.png) + [![Create a new blank disk.](982-disk3-small.png)](983-disk3.png) -5. Enter a name for the disk in the **Label** field. -6. By default, **Type** is set to `ext4`. We recommend that you stick with this setting. -7. Enter a size for the disk in the **Deployment Disk Size** field. The size must be smaller than the amount of storage space remaining in your Linode plan. -8. Click **Save Changes**. +1. Enter a name for the disk in the **Label** field. +1. By default, **Type** is set to `ext4`. We recommend that you stick with this setting. +1. Enter a size for the disk in the **Deployment Disk Size** field. The size must be smaller than the amount of storage space remaining in your Linode plan. +1. Click **Save Changes**. The disk will be created. Watch the *Host Job Queue* on the Dashboard to monitor the progress. When the process is complete, the disk will appear on the dashboard. You'll need to create or modify a configuration profile to mount the new disk. For instructions, see [Configuration Profiles](#configuration-profiles). @@ -93,13 +93,13 @@ The disk will be created. Watch the *Host Job Queue* on the Dashboard to monitor Resizing allows you to allocate more storage to a disk so you can store more files, or shrink a disk so you have more storage for the other disks in your account. Here's how to resize a disk: 1. Log in to the [Linode Manager](https://manager.linode.com). -2. Click the **Linodes** tab. -3. Select a Linode. The Linode's dashboard appears. -4. Click **Shut down** to turn your Linode off. Watch the *Host Job Queue* for confirmation that the Linode has shut down. -5. Select the disk you want to resize. The *Edit disk* webpage appears. -6. In the **New Size** field, enter a different size for the disk in megabytes. -7. Click **Save Changes**. The Linode's dashboard appears. Watch the *Host Job Queue* for confirmation that the disk has been resized. -8. Click **Boot** to turn on the Linode. +1. Click the **Linodes** tab. +1. Select a Linode. The Linode's dashboard appears. +1. Click **Shut down** to turn your Linode off. Watch the *Host Job Queue* for confirmation that the Linode has shut down. +1. Select the disk you want to resize. The *Edit disk* webpage appears. +1. In the **New Size** field, enter a different size for the disk in megabytes. +1. Click **Save Changes**. The Linode's dashboard appears. Watch the *Host Job Queue* for confirmation that the disk has been resized. +1. Click **Boot** to turn on the Linode. You have successfully resized the disk. @@ -108,12 +108,12 @@ You have successfully resized the disk. You can create an exact copy of a disk by duplicating it. This is an effective way to back up your server or clone an existing Linode to a new Linode. (To clone a disk, see [Cloning disks and Configuration Profiles](#cloning-disks-and-configuration-profiles).) Here's how to duplicate a disk: 1. Log in to the [Linode Manager](https://manager.linode.com). -2. Click the **Linodes** tab. -3. Select a Linode. The Linode's dashboard appears. -4. Click **Shut down** to turn your Linode off. Watch the *Host Job Queue* for confirmation that the Linode has shut down. -5. Select the disk you want to duplicate. The *Edit disk* webpage appears. -6. Click **Duplicate Disk**. The Linode's dashboard appears. Watch the *Host Job Queue* for confirmation that the disk has been duplicated. -7. Click **Boot** to turn on the Linode. +1. Click the **Linodes** tab. +1. Select a Linode. The Linode's dashboard appears. +1. Click **Shut down** to turn your Linode off. Watch the *Host Job Queue* for confirmation that the Linode has shut down. +1. Select the disk you want to duplicate. The *Edit disk* webpage appears. +1. Click **Duplicate Disk**. The Linode's dashboard appears. Watch the *Host Job Queue* for confirmation that the disk has been duplicated. +1. Click **Boot** to turn on the Linode. The disk will be duplicated. Watch the *Host Job Queue* on the Dashboard to monitor the progress. When the process is complete, the disk will appear on the dashboard. @@ -122,14 +122,14 @@ The disk will be duplicated. Watch the *Host Job Queue* on the Dashboard to moni You can delete a disk to remove it from your Linode and reallocate its storage space to another disk. This action is permanent and cannot be undone. Here's how to delete a disk: 1. Log in to the [Linode Manager](https://manager.linode.com). -2. Click the **Linodes** tab. -3. Select a Linode. The Linode's dashboard appears. -4. Click **Shut down** to turn your Linode off. Watch the *Host Job Queue* for confirmation that the Linode has shut down. -5. Select the **Remove** link next to the Linode you want to delete, as shown below. The disk will be deleted. Watch the *Host Job Queue* for confirmation that the disk has been duplicated. +1. Click the **Linodes** tab. +1. Select a Linode. The Linode's dashboard appears. +1. Click **Shut down** to turn your Linode off. Watch the *Host Job Queue* for confirmation that the Linode has shut down. +1. Select the **Remove** link next to the Linode you want to delete, as shown below. The disk will be deleted. Watch the *Host Job Queue* for confirmation that the disk has been duplicated. [![Deleting a disk](987-disk5-1-small.png)](986-disk5-1.png) -6. Click **Boot** to turn on the Linode. +1. Click **Boot** to turn on the Linode. The disk will be deleted. The storage space the disk was using is now available to other disks. @@ -142,19 +142,19 @@ Linode's configuration profiles are similar to [GNU GRUB](http://en.wikipedia.or Making a new configuration profile allows you to create a new and separate boot configuration for your system. You can specify boot settings and disks to mount. Here's how to create a new configuration profile: 1. Log in to the [Linode Manager](https://manager.linode.com). -2. Click the **Linodes** tab. -3. Select a Linode. The Linode's dashboard appears. -4. Select the **Create a new Configuration Profile** link. The *Configuration Profile* webpage appears, as shown below. +1. Click the **Linodes** tab. +1. Select a Linode. The Linode's dashboard appears. +1. Select the **Create a new Configuration Profile** link. The *Configuration Profile* webpage appears, as shown below. [![Creating a configuration profile](createconfprofile-small.png)](createconfprofile.png) -5. Enter a descriptive name for the configuration profile in the **Label** field. This name appears on the dashboard and will help you differentiate it from other configuration profiles. -6. You can enter comments or notes about this configuration profile in the **Notes** field. -7. Select a Linux kernel from the **Kernel** menu. We recommend selecting one of the current and latest kernels. -8. In the *Block Device Assignment* section, select a bootable disk, a swap disk (optional), and any other disks that you would like to mount at start up. -9. Select a bootable disk from the **root device Standard** menu. -10. Leave the settings in the *Filesystem/Boot Helpers* section alone, unless you have a specific reason to change them. -11. Click **Save Changes**. +1. Enter a descriptive name for the configuration profile in the **Label** field. This name appears on the dashboard and will help you differentiate it from other configuration profiles. +1. You can enter comments or notes about this configuration profile in the **Notes** field. +1. Select a Linux kernel from the **Kernel** menu. We recommend selecting one of the current and latest kernels. +1. In the *Block Device Assignment* section, select a bootable disk, a swap disk (optional), and any other disks that you would like to mount at start up. +1. Select a bootable disk from the **root device Standard** menu. +1. Leave the settings in the *Filesystem/Boot Helpers* section alone, unless you have a specific reason to change them. +1. Click **Save Changes**. The configuration profile will be created and will appear on the Linode's dashboard. @@ -163,11 +163,11 @@ The configuration profile will be created and will appear on the Linode's dashbo You can edit existing configuration profiles to change boot settings, set other disks to mount, and more. Here's how to edit a configuration profile: 1. Log in to the [Linode Manager](https://manager.linode.com). -2. Click the **Linodes** tab. -3. Select a Linode. The Linode's dashboard appears. -4. Select a configuration profile. The *Configuration Profile* appears. -5. Edit the settings as necessary. -6. When finished, click **Save Changes**. +1. Click the **Linodes** tab. +1. Select a Linode. The Linode's dashboard appears. +1. Select a configuration profile. The *Configuration Profile* appears. +1. Edit the settings as necessary. +1. When finished, click **Save Changes**. The changes to the configuration profile have been saved. You may need to reboot your Linode to activate the changes. @@ -176,13 +176,13 @@ The changes to the configuration profile have been saved. You may need to reboot You can create and store many different configuration profiles in the Linode Manager, but you can only boot your Linode from one configuration profile at a time. Here's how to select a configuration profile and boot your Linode from it: 1. Log in to the [Linode Manager](https://manager.linode.com). -2. Click the **Linodes** tab. -3. Select a Linode. The Linode's dashboard appears. -4. Click the button to select a configuration profile, as shown below. +1. Click the **Linodes** tab. +1. Select a Linode. The Linode's dashboard appears. +1. Click the button to select a configuration profile, as shown below. [![Selecting a configuration profile](993-disk7-1.png)](993-disk7-1.png) -5. Click **Reboot** to restart your Linode using the selected configuration profile. +1. Click **Reboot** to restart your Linode using the selected configuration profile. You have successfully selected and booted your Linode from a configuration profile. @@ -191,9 +191,9 @@ You have successfully selected and booted your Linode from a configuration profi You can remove a configuration profile from the Linode Manager at anytime. Here's how: 1. Log in to the [Linode Manager](https://manager.linode.com). -2. Click the **Linodes** tab. -3. Select a Linode. The Linode's dashboard appears. -4. Select the **Remove** link next to the configuration profile you want to remove. +1. Click the **Linodes** tab. +1. Select a Linode. The Linode's dashboard appears. +1. Select the **Remove** link next to the configuration profile you want to remove. The configuration profile is removed from the dashboard. diff --git a/docs/platform/disk-images/linode-backup-service-new-manager/index.md b/docs/platform/disk-images/linode-backup-service-new-manager/index.md index 820cc09c1f5..bb885e9b307 100644 --- a/docs/platform/disk-images/linode-backup-service-new-manager/index.md +++ b/docs/platform/disk-images/linode-backup-service-new-manager/index.md @@ -209,6 +209,11 @@ There are some limitations to what the Linode Backup Service can back up. Here a - The Backup Service must be able to mount your disks. If you've created partitions, configured full disk encryption, or made other changes that prevent us from mounting the disk as a filesystem, you will likely not be able to use the Linode Backup Service. The backup system operates at the file level, not at the block level. - Because the Backup Service is file-based, the number of files stored on disk will impact both the time it takes for backups and restores to complete, and your ability to successfully take and restore backups. Customers who need to permanently store a large number of files may want to archive bundles of smaller files into a single file, or consider other backup services. + + {{< note >}} +The percentage of customers who may run into this limitation is low. If you are not sure if this limitation applies to you, please [contact Linode Support](/docs/platform/billing-and-support/support/#contacting-linode-support). +{{< /note >}} + - Backups taken of ext4 or ext3 filesystems will be restored as ext4. Backups taken of other mountable filesystem types will have their contents restored using ext4. - Files that have been modified but have the same size and modify time will not be considered "changed" during a subsequent backup. ACLs and extended attributes are *not* tracked. - The Backup Service uses a snapshot of your disks to take consistent backups while your Linode is running. This method is very reliable, but can fail to properly back up the data files for database services like MySQL. If the snapshot occurs during a transaction, the database's files may be backed up in an unclean state. We recommend scheduling routine dumps of your database to a file on the filesystem. The resulting file will then be backed up, allowing you to restore the contents of the database if you need to restore from a backup. diff --git a/docs/platform/disk-images/linode-backup-service/index.md b/docs/platform/disk-images/linode-backup-service/index.md index 4c859f0d2ff..830ff9dc984 100644 --- a/docs/platform/disk-images/linode-backup-service/index.md +++ b/docs/platform/disk-images/linode-backup-service/index.md @@ -2,10 +2,10 @@ author: name: Alex Fornuto description: 'Use the Linode Backup Service to protect and secure your data.' -keywords: ["backup service", "linode platform", "linode backup service", "enable a backup", "manage a backup", "schedule a backup", "disable a backup", "restore from a backup", "boot from a backup"] +keywords: ["backup service", "linode platform", "linode backup service", "enable a backup", "manage a backup", "schedule a backup", "disable a backup", "restore from a backup", "boot from a backup", "disk space"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' aliases: ['backup-service/','platform/backup-service/', 'security/backups/linode-backup-service/','platform/linode-backup-service/'] -modified: 2018-12-20 +modified: 2019-01-28 modified_by: name: Linode published: 2012-03-14 @@ -122,6 +122,10 @@ The Linode Backup Service initiates the manual snapshot. Be patient. Creating th {{< content "restore-backup-image-short" >}} +### Restore to a Linode in a Different Data Center + +To restore a backup to a different data center from the source Linode's location, first restore to a Linode in the same data center, creating a new one if necessary. Once the restore is complete, use the [Clone](/docs/migrate-to-linode/disk-images/clone-your-linode/) tab to copy the disk(s) to a Linode in a different data center. + ## Boot from a Backup After the backup has been restored, the disks and configuration profiles will be available to the Linode you selected. Select the restored configuration profile and reboot your Linode to start up from the restored disks. Here's how: @@ -145,6 +149,11 @@ There are some limitations to what the Linode Backup Service can back up. Here a - The Backup Service must be able to mount your disks. If you've created partitions, configured full disk encryption, or made other changes that prevent us from mounting the disk as a filesystem, you will likely not be able to use the Linode Backup Service. The backup system operates at the file level, not at the block level. - Because the Backup Service is file-based, the number of files stored on disk will impact both the time it takes for backups and restores to complete, and your ability to successfully take and restore backups. Customers who need to permanently store a large number of files may want to archive bundles of smaller files into a single file, or consider other backup services. + + {{< note >}} +The percentage of customers who may run into this limitation is low. If you are not sure if this limitation applies to you, please [contact Linode Support](/docs/platform/billing-and-support/support/#contacting-linode-support). +{{< /note >}} + - Backups taken of ext4 or ext3 filesystems will be restored as ext4. Backups taken of other mountable filesystem types will have their contents restored using ext4. - Files that have been modified but have the same size and modify time will not be considered "changed" during a subsequent backup. ACLs and extended attributes are *not* tracked. - The Backup Service uses a snapshot of your disks to take consistent backups while your Linode is running. This method is very reliable, but can fail to properly back up the data files for database services like MySQL. If the snapshot occurs during a transaction, the database's files may be backed up in an unclean state. We recommend scheduling routine dumps of your database to a file on the filesystem. The resulting file will then be backed up, allowing you to restore the contents of the database if you need to restore from a backup. diff --git a/docs/platform/disk-images/resizing-a-linode/index.md b/docs/platform/disk-images/resizing-a-linode/index.md index b697db4715d..cd30ec5cb71 100644 --- a/docs/platform/disk-images/resizing-a-linode/index.md +++ b/docs/platform/disk-images/resizing-a-linode/index.md @@ -3,10 +3,10 @@ author: name: Linode email: docs@linode.com description: Our guide to upgrading and resizing your Linode -keywords: ["upgrading", "resizing"] +keywords: ["upgrading", "resizing", "disk space"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' aliases: ['resizing/','migrate-to-linode/disk-images/resizing-a-linode/'] -modified: 2017-03-02 +modified: 2019-01-28 modified_by: name: Linode published: 2017-02-14 diff --git a/docs/platform/disk-images/restore-backup-image-short/backup-service-dashboard-storage-indicator.png b/docs/platform/disk-images/restore-backup-image-short/backup-service-dashboard-storage-indicator.png deleted file mode 100644 index 3138e043c98..00000000000 Binary files a/docs/platform/disk-images/restore-backup-image-short/backup-service-dashboard-storage-indicator.png and /dev/null differ diff --git a/docs/platform/disk-images/restore-backup-image-short/backup-service-restore-to-a-linode.png b/docs/platform/disk-images/restore-backup-image-short/backup-service-restore-to-a-linode.png deleted file mode 100644 index ae306db9f7f..00000000000 Binary files a/docs/platform/disk-images/restore-backup-image-short/backup-service-restore-to-a-linode.png and /dev/null differ diff --git a/docs/platform/disk-images/restore-backup-image-short/backup-service-restore-to-link.png b/docs/platform/disk-images/restore-backup-image-short/backup-service-restore-to-link.png deleted file mode 100644 index a020183c5ca..00000000000 Binary files a/docs/platform/disk-images/restore-backup-image-short/backup-service-restore-to-link.png and /dev/null differ diff --git a/docs/platform/disk-images/restore-backup-image-short/backup-service-restore-to-new.png b/docs/platform/disk-images/restore-backup-image-short/backup-service-restore-to-new.png deleted file mode 100644 index 6b48461cab2..00000000000 Binary files a/docs/platform/disk-images/restore-backup-image-short/backup-service-restore-to-new.png and /dev/null differ diff --git a/docs/platform/disk-images/restore-backup-image-short/backup-service-storage-indicator-resize.png b/docs/platform/disk-images/restore-backup-image-short/backup-service-storage-indicator-resize.png deleted file mode 100644 index b4553e329c7..00000000000 Binary files a/docs/platform/disk-images/restore-backup-image-short/backup-service-storage-indicator-resize.png and /dev/null differ diff --git a/docs/platform/disk-images/restore-backup-image-short/backup-service-total-size-required.png b/docs/platform/disk-images/restore-backup-image-short/backup-service-total-size-required.png deleted file mode 100644 index c7b3c922678..00000000000 Binary files a/docs/platform/disk-images/restore-backup-image-short/backup-service-total-size-required.png and /dev/null differ diff --git a/docs/platform/disk-images/restore-backup-image-short/index.md b/docs/platform/disk-images/restore-backup-image-short/index.md index d92f5ab3c00..e5d27986d0f 100644 --- a/docs/platform/disk-images/restore-backup-image-short/index.md +++ b/docs/platform/disk-images/restore-backup-image-short/index.md @@ -25,15 +25,13 @@ The size of the disk(s) created by the restore process will only be slightly lar Some applications, like databases, need some amount of free unused space inside the disk in order to run. As a result, you may want to [increase your disk(s) size](/docs/platform/disk-images/resizing-a-linode/) after the restore process is completed. {{< /note >}} -To restore a backup to a different data center, first restore to a Linode in the same data center, creating a new one if necessary. Once the restore is complete, use the [Clone](/docs/migrate-to-linode/disk-images/clone-your-linode/) tab to copy the disk(s) to a Linode in a different data center. - ### Restore to a New Linode You can restore a backup to any Linode located in the same data center, even if the target does not have the Backup Service enabled. This section covers how to restore a backup to a new Linode that does not have any disks deployed to it. If you wish to restore your backup to an existing Linode, see the [next section](#restore-to-an-existing-linode). 1. From the Dashboard of the Linode whose backups you intend to restore, click on the **Backups** tab. Select the **Restore to...** link beneath the backup version that you want to restore. -1. Under the *Select* column, click the **Restore to this Linode** link next to your new Linode: +1. Under the *Select* column, click the **Restore to this Linode** link next to your new Linode. The backup disks and configuration profiles will be restored to the Linode you selected. Watch the *Host Job Queue* to monitor the progress. Restoring from a backup can take several minutes depending on the size of your Linode and the amount of data you have stored on it. @@ -42,7 +40,7 @@ You can restore a backup to any Linode located in the same data center, even if To restore a backup to an existing Linode, you will need to make sure that you have enough storage space that is not currently assigned to disk images. {{< note >}} -If you are attempting to restore a disk to the same Linode the backup was created from, the restoration process will not delete the original disk for you. Manually delete the original disk to make room for the backup. +If you are attempting to restore a disk to the same Linode the backup was created from, the restoration process will not delete the original disk for you. You can manually delete the original disk if you prefer to, but this is not strictly required. {{< /note >}} 1. Start by confirming the size of the backup that you wish to restore. From the **Backups** tab in your Linode's Dashboard, click the **Restore to...** link beneath your desired backup version. @@ -53,10 +51,13 @@ If you are attempting to restore a disk to the same Linode the backup was create 1. Next, you'll confirm the total space assigned to disk images on your Linode, via the **Storage** indicator on your Linode's Dashboard. -1. If the amount of space available is greater than the size of the backup, you can proceed with restoring. If the amount of unallocated space is less than the size of the backup, you can [resize your existing disks](/docs/platform/disk-images/disk-images-and-configuration-profiles/#resizing-a-disk) to make room for it. +1. If the amount of space available is greater than the size of the backup, you can proceed with restoring. If the amount of unallocated space is less than the size of the backup, you can [shrink your existing disks](/docs/platform/disk-images/disk-images-and-configuration-profiles/#resizing-a-disk) to make room for it. {{< note >}} -In some cases, you will not be able to shrink your disks enough to fit the restored backup. As an alternative, you can [change your Linode's plan](/docs/platform/disk-images/resizing-a-linode/) to a higher tier that offers more disk space. +In some cases, you will not be able to shrink your disks enough to fit the restored backup. Other alternatives for restoring to your existing Linode include: + +- [Changing your Linode's plan](/docs/platform/disk-images/resizing-a-linode/) to a higher tier that offers more disk space. +- [Removing other disks](/docs/platform/disk-images/disk-images-and-configuration-profiles/#removing-a-disk) from your Linode. {{< /note >}} 1. Once the disk resize has completed, check the storage indicator on your Linode's Dashboard to confirm that you've freed up enough space for your backup. @@ -64,3 +65,7 @@ In some cases, you will not be able to shrink your disks enough to fit the resto 1. From the **Backups** tab, click the **Restore to this Linode** link next to your Linode. Your backup will begin restoring to your Linode, and you can monitor its progress from the *Host Job Queue* in your Linode's Dashboard tab. Note that the time it takes to restore your backup will vary depending upon the restore size, and the number of files being restored. + + {{< note >}} +In order to access the files from your restored disks, **you will need to [reboot your Linode](/docs/platform/disk-images/disk-images-and-configuration-profiles/#selecting-and-using-a-configuration-profile) under the new configuration profile that was created by the restore process**. The restored disks are assigned to the new profile, and your backed up data will be accessible when you boot from that profile. +{{< /note >}} diff --git a/docs/platform/network-helper-new-manager/index.md b/docs/platform/network-helper-new-manager/index.md new file mode 100644 index 00000000000..de3a2a2ab7c --- /dev/null +++ b/docs/platform/network-helper-new-manager/index.md @@ -0,0 +1,173 @@ +--- +author: + name: Linode + email: docs@linode.com +description: 'What is Network Helper and how to use it.' +keywords: ["network helper", "static"] +license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' +modified: 2019-01-07 +published: 2014-12-08 +modified_by: + name: Linode +title: Network Helper +hiddenguide: true +--- + +![Network Helper](network-helper-title-graphic.jpg "Network Helper") + +## What is Network Helper? + +Network Helper is a host-side service which automatically sets a static IPv4 address and gateway for your Linode. This means you do not need to manually reconfigure your Linode's network addressing when you: + + - Deploy a Linode + - Add a public or private IPv4 address + - Restore from a backup + - Migrate your Linode to a new data center + - Clone from another Linode + +Network Helper is enabled by default, and works by detecting which distribution is booting and then modifying the appropriate network configuration files. If Network Helper is unable to determine the operating system during boot, or if you boot an unsupported operating system, Network Helper will not attempt to write any new configuration files. Be aware that Network Helper configures only IPv4 addressing; your Linode's IPv6 address is assigned by SLAAC. + +{{< caution >}} +If you instead choose to manually configure your Linode's network interface settings, be the IPv4 or IPv6, you must disable Network Helper for that Linode or your configuration will overwritten during the next boot. +{{< /caution >}} + + +## Network Helper Settings + + Network Helper can be enabled or disabled globally for your account, or on a per-Linode basis. Network Helper can still be toggled on and off for specific Linodes, regardless of whether enabled or disabled globally. + +### Global + +When Network Helper is enabled globally, all new Linodes created on your account will have Network Helper enabled by default. + +1. Click on **Account** in the left sidebar of the Linode Cloud Manager. + +1. Click on the **Global Settings** tab. Set the toggle slider under the **Network Helper** section to the desired setting. Blue is enabled, gray is disabled. + + [![The Network Helper Default Behavior option](network-helper-linode-account-settings-small.png)](network-helper-linode-account-settings.png) + +1. Click the **Save** button. + +### Single (Per-Linode) + +1. Click on **Linodes** in the left sidebar of the Linode Cloud Manager. + +1. Expand the overflow menu (the three dots at the far right) of the Linode which you want to enable Network Helper for. Then click **Settings**. + + ![Navigate to the Linode's Settings page](network-helper-linode-settings-link.png) + +1. Under **Advanced Configurations**, expand the overflow menu for your configuration profile and click **Edit**: + + [![Select the configuration profile Edit Menu](network-helper-linode-settings-page-small.png)](network-helper-linode-settings-page.png) + +1. A menu will appear with that configuration profile's settings. Under the **Filesystem/Boot Helpers** section, toggle **Auto-configure networking** to the desired setting. Blue is enabled, gray is disabled. + + [![Toggle the 'auto-configure networking' box](network-helper-linode-config-profile-settings-small.png)](network-helper-linode-config-profile-settings.png) + +1. Click **Submit**. + + +## What Files are Modified + +The specific files Network Helper modifies varies by distribution. + +### Arch + +Network Helper configures `/etc/systemd/network/05-eth0.network`. + +### CentOS & Fedora + +Network Helper configures `/etc/sysconfig/network-scripts/ifcfg-eth0`. + +### Debian & Ubuntu + +Network helper configures `/etc/network/interfaces` and `/etc/resolv.conf`. + +### Gentoo + +Network Helper configures `/etc/conf.d/net` and `/etc/resolv.conf`. + +### OpenSUSE + +Network Helper configures `/etc/sysconfig/network/ifcfg-eth0`, `/etc/sysconfig/network/routes` and `/etc/resolv.conf`. + +### Slackware + +Network Helper configures `/etc/rc.d/rc.inet1.conf` and `/etc/resolv.conf`. + + +## What is Modified in Those Files + +Below are example network configuration files for a Debian 9 Linode with Network Helper enabled: + +{{< file "/etc/network/interfaces" >}} +# Generated by Linode Network Helper +# Wed Jan 9 21:30:02 2019 UTC +# +# This file is automatically generated on each boot with your Linode's +# current network configuration. If you need to modify this file, please +# first disable the 'Auto-configure Networking' setting within your Linode's +# configuration profile: +# - https://manager.linode.com/linodes/config/lin1?id=13561415 +# +# For more information on Network Helper: +# - https://www.linode.com/docs/platform/network-helper +# +# A backup of the previous config is at /etc/network/.interfaces.linode-last +# A backup of the original config is at /etc/network/.interfaces.linode-orig +# +# /etc/network/interfaces + +auto lo +iface lo inet loopback + +auto eth0 +allow-hotplug eth0 + +iface eth0 inet6 auto + +iface eth0 inet static + address 203.0.113.5/24 + gateway 203.0.113.1 +{{< /file >}} + +{{< file "/etc/resolv.conf" >}} +# Generated by Linode Network Helper +# Wed Jan 9 21:30:02 2019 UTC +# +# This file is automatically generated on each boot with your Linode's +# current network configuration. If you need to modify this file, please +# first disable the 'Auto-configure Networking' setting within your Linode's +# configuration profile: +# - https://manager.linode.com/linodes/config/lin1?id=13561415 +# +# For more information on Network Helper: +# - https://www.linode.com/docs/platform/network-helper +# +# A backup of the previous config is at /etc/.resolv.conf.linode-last +# A backup of the original config is at /etc/.resolv.conf.linode-orig +# +domain members.linode.com +search members.linode.com +nameserver 66.228.53.5 +nameserver 96.126.122.5 +nameserver 96.126.124.5 +nameserver 96.126.127.5 +nameserver 198.58.107.5 +nameserver 198.58.111.5 +nameserver 23.239.24.5 +nameserver 173.255.199.5 +nameserver 72.14.179.5 +nameserver 72.14.188.5 +{{< /file >}} + + +In addition to network interface file file (again, specific to this Debian example), Network Helper will create: + +- A copy of the interface and resolver file as the distribution provided it: `.interfaces.linode-orig` and `/etc/.resolv.conf.linode-orig`. Note that Network Helper does not modify `/etc/resolv.conf` on all of our distributions. + +- A copy of the interface and resolver files from the previous boot: `.interfaces.linode-last` and `/etc/.resolv.conf.linode-last`. If you manually changed either of these file before the previous boot, you'll find them saved there. + +Use the following command to restore manual changes made before the previous reboot. Be sure to replace `/etc/network/interfaces` with the network interface file for your distribution from [above](#what-files-are-modified). + + mv /etc/network/.interfaces.linode-last /etc/network/interfaces diff --git a/docs/platform/network-helper-new-manager/network-helper-linode-account-settings-small.png b/docs/platform/network-helper-new-manager/network-helper-linode-account-settings-small.png new file mode 100644 index 00000000000..39bc01a5f16 Binary files /dev/null and b/docs/platform/network-helper-new-manager/network-helper-linode-account-settings-small.png differ diff --git a/docs/platform/network-helper-new-manager/network-helper-linode-account-settings.png b/docs/platform/network-helper-new-manager/network-helper-linode-account-settings.png new file mode 100644 index 00000000000..97b9aee83ce Binary files /dev/null and b/docs/platform/network-helper-new-manager/network-helper-linode-account-settings.png differ diff --git a/docs/platform/network-helper-new-manager/network-helper-linode-config-profile-settings-small.png b/docs/platform/network-helper-new-manager/network-helper-linode-config-profile-settings-small.png new file mode 100644 index 00000000000..f8faaab35b0 Binary files /dev/null and b/docs/platform/network-helper-new-manager/network-helper-linode-config-profile-settings-small.png differ diff --git a/docs/platform/network-helper-new-manager/network-helper-linode-config-profile-settings.png b/docs/platform/network-helper-new-manager/network-helper-linode-config-profile-settings.png new file mode 100644 index 00000000000..88df06786ad Binary files /dev/null and b/docs/platform/network-helper-new-manager/network-helper-linode-config-profile-settings.png differ diff --git a/docs/platform/network-helper-new-manager/network-helper-linode-settings-link.png b/docs/platform/network-helper-new-manager/network-helper-linode-settings-link.png new file mode 100644 index 00000000000..dbc7ff24615 Binary files /dev/null and b/docs/platform/network-helper-new-manager/network-helper-linode-settings-link.png differ diff --git a/docs/platform/network-helper-new-manager/network-helper-linode-settings-page-small.png b/docs/platform/network-helper-new-manager/network-helper-linode-settings-page-small.png new file mode 100644 index 00000000000..036ee1c328b Binary files /dev/null and b/docs/platform/network-helper-new-manager/network-helper-linode-settings-page-small.png differ diff --git a/docs/platform/network-helper-new-manager/network-helper-linode-settings-page.png b/docs/platform/network-helper-new-manager/network-helper-linode-settings-page.png new file mode 100644 index 00000000000..aaa7d5203dc Binary files /dev/null and b/docs/platform/network-helper-new-manager/network-helper-linode-settings-page.png differ diff --git a/docs/platform/network-helper-new-manager/network-helper-title-graphic.jpg b/docs/platform/network-helper-new-manager/network-helper-title-graphic.jpg new file mode 100644 index 00000000000..b349e8d2d73 Binary files /dev/null and b/docs/platform/network-helper-new-manager/network-helper-title-graphic.jpg differ diff --git a/docs/platform/network-helper/index.md b/docs/platform/network-helper/index.md index 1e5a442ffe6..9357e66de8f 100644 --- a/docs/platform/network-helper/index.md +++ b/docs/platform/network-helper/index.md @@ -5,11 +5,12 @@ author: description: 'Information on the Network Helper option.' keywords: ["network", " networking", " network helper", " ip", " ip address", " static ip", ""] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -modified: 2017-04-28 +modified: 2018-12-10 published: 2014-12-08 modified_by: name: Linode title: Network Helper +cloud_manager_link: platform/network-helper-new-manager/ --- ![Network Helper](network-helper-title-graphic.jpg "Network Helper") diff --git a/docs/security/backups/download-backups-locally/bs-pv-sda-sdc.gif b/docs/security/backups/download-backups-locally/bs-pv-sda-sdc.gif deleted file mode 100644 index b7a213f42af..00000000000 Binary files a/docs/security/backups/download-backups-locally/bs-pv-sda-sdc.gif and /dev/null differ diff --git a/docs/security/backups/download-backups-locally/bs-rescue-mode-sdc.png b/docs/security/backups/download-backups-locally/bs-rescue-mode-sdc.png deleted file mode 100644 index 51723a22a12..00000000000 Binary files a/docs/security/backups/download-backups-locally/bs-rescue-mode-sdc.png and /dev/null differ diff --git a/docs/security/backups/download-backups-locally/index.md b/docs/security/backups/download-backups-locally/index.md deleted file mode 100644 index df64c0a92c0..00000000000 --- a/docs/security/backups/download-backups-locally/index.md +++ /dev/null @@ -1,125 +0,0 @@ ---- -author: - name: Edward Angert - email: docs@linode.com -description: "Learn how to download a local copy of your Linode backups." -keywords: ["backup", "backups", "rsync", "restore", "local"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -modified: 2018-08-08 -modified_by: - name: Linode -published: 2018-08-08 -title: Download a Local Copy of your Linode Backup ---- - -The [Linode Backups](/docs/platform/disk-images/linode-backup-service/) service can create automatic and manual snapshots of your Linode. A completed backup can be directly restored to the origin Linode or to a new Linode in the same data center. These workflows make it easy to revert to a working configuration if you run into any unexpected issues with your software. - -Linode's backups are stored in a way that is only directly readable by the Linode Backups service. A common question for the service is how you can download the content from your Linode Backups to another storage location, like your home computer. This can be accomplished in two phases: - -1. Restore a backup to a new Linode in the same data center. -2. Download either specific files or the entire disk image from that Linode, as needed. - -## Before You Begin - -### Account Permissions and Billing - -Several of the steps in this guide involve adding services to or removing services from a Linode account. Visit our guide on [Users and Permissions](/docs/platform/manager/accounts-and-passwords/#users-and-permissions) for more information about restricted Linode users. - -Note that the cost of adding Backups service and adding a Linode to your account is billed, prorated per hour. If the backups service is only enabled for a few hours, you will only be charged for a few hours of the service. See the [Backups pricing details](/docs/platform/disk-images/linode-backup-service/#pricing) for more information. Likewise, when you create a Linode, you will be billed per hour that the Linode exists, whether it is powered on or not. - -The steps in this guide have been designed to minimize the potential costs associated with this process. Additionally, keep the following in mind: - -- Removing a Linode from your account also cancels the associated Backup service for that Linode. -- A Linode's backups are deleted when a Linode is deleted. -- If you choose to leave the Backups service enabled, or if you do not remove the additional Linode from your account, you will be automatically billed. If you only power the Linode off, [you will still be billed for it](/docs/platform/billing-and-support/billing-and-payments/#if-my-linode-is-powered-off-will-i-be-billed). - -### Enable Backups and Take a Snapshot - -These steps are the minimum required for the scope of this guide. Visit our [Backups guide](/docs/platform/disk-images/linode-backup-service/) for information about how to implement regular backups of your Linode. - -1. Go to your Linode's dashboard, click **Backups**, click **Enable backups for this Linode »** and confirm the additional cost per month. - -1. This guide focuses on saving a snapshot or specific backup. Click **Take a New Snapshot Now**. - - - The snapshot appears in the *Backup History* at the bottom of the page. - -{{< content "restore-backup-image-short" >}} - -## Download Specific Files or Directories over SSH - -If you just need specific files from your Linode, you can download those over SSH. In order to do so, you'll first need to [reboot your Linode](/docs/platform/disk-images/disk-images-and-configuration-profiles/#selecting-and-using-a-configuration-profile) under the new configuration profile that was created by the restore process. This new profile is assigned to the restored disks, and your backed up data will be accessible when you boot from them. - -Downloading files over SSH can be done at a command-line interface, or with a graphical *SFTP* file browser. - -### Using SCP - -To retrieve a specific directory or file via the command-line, you can use the secure copy (SCP) command from your computer. SCP is installed by default on most Mac and Linux systems, and you can install a tool like [Cygwin](/docs/platform/disk-images/copying-a-disk-image-over-ssh/#windows-cygwin-instructions) to use it on Windows. - -- The syntax for using SCP to copy a file from your Linode into a directory on your computer is: - - scp your_linode_username@your_linode_ip:/path/to/your/file.txt /path/to/your/local/directory/ - - The file will be saved inside `/path/to/your/local/directory/` on your computer. - -- To copy a file from your Linode to your computer and give it a specific name (in this case, `file.txt.backup`): - - scp your_linode_username@your_linode_ip:/path/to/your/file.txt /path/to/your/local/directory/file.txt.backup - -- To copy a directory from your Linode to your computer: - - scp -r your_linode_username@your_linode_ip:/path/to/your/directory /path/to/your/local/directory - - If `/path/to/your/local/directory` already exists on your computer, then the copied directory will be placed inside `/path/to/your/local/directory` (i.e. `/path/to/your/local/directory/directory`). - - If `/path/to/your/local/directory` does not already exist, then the copied directory will be created with that name. - -For example: - -* Download an NGINX configuration file to your home folder: - - scp your_linode_username@your_linode_ip:/etc/nginx/conf.d/example.com.conf ~/example.com.conf.backup - -* Download an Apache configuration file to your home folder: - - scp your_linode_username@your_linode_ip:/etc/apache2/sites-available/example.com.conf ~/example.com.conf.backup - -* Copy the entire document root from a web server: - - scp -r your_linode_username@your_linode_ip:/var/www/html/ ~/html_backup - -If you intend to repeat this process regularly, consider [using rsync](/docs/security/backups/backing-up-your-data/#understand-the-rsync-command) to create additional local copies of your data. rsync is capable of performing incremental file copies, which means you do not have to fully transfer each file every time you download your data. - -### Using FileZilla - -As an alternative to the command-line, you can download and install an *SFTP* client. These applications provide a graphical user interface for your Linode's filesystem. - -*FileZilla* is a popular free example. Windows and OS X users can download FileZilla [here](https://filezilla-project.org/download.php?show_all=1). To install FileZilla on Linux: - -- Debian/Ubuntu: - - sudo apt-get install filezilla - -- CentOS/Fedora: - - sudo yum install filezilla - -After you've installed FileZilla on your computer: - -{{< content "filezilla-shortguide" >}} - -For more information on FileZilla, [review our full guide](/docs/tools-reference/file-transfer/filezilla/) on using the application. - -### Using mysqldump to Back Up a Database - -Special care is needed when downloading data from a database. Before it can be downloaded, the data in a database needs to first be *dumped* to a file. This file can then be transferred just as any other normal file type. - -{{< content "mysqldump-database-backup-short" >}} - -For more information on MySQL database backups, including how to restore the data in a dump file to a MySQL installation, review [our guide](/docs/databases/mysql/use-mysqldump-to-back-up-mysql-or-mariadb/#restore-a-backup) on the subject. An alternative to using `mysqldump` is to create [*physical* backups](/docs/databases/mysql/create-physical-backups-of-your-mariadb-or-mysql-databases/). It's also possible to [backup PostgreSQL databases](/docs/databases/postgresql/how-to-back-up-your-postgresql-database/). - -{{< content "copy-disk-over-ssh-short" >}} - - -## Clean Up after Your Download - -After you've finished downloading your files or disks, you can optionally [delete the restored disks](/docs/platform/disk-images/disk-images-and-configuration-profiles/#removing-a-disk). If you created a new Linode to perform the restore, consider [deleting the Linode](/docs/platform/billing-and-support/billing-and-payments/#removing-services). As a reminder, billing for that Linode will continue automatically if you do not remove it. If you only power the Linode off, [you will still be billed for it](/docs/platform/billing-and-support/billing-and-payments/#if-my-linode-is-powered-off-will-i-be-billed). \ No newline at end of file diff --git a/docs/security/data-portability/_index.md b/docs/security/data-portability/_index.md new file mode 100644 index 00000000000..b126127e965 --- /dev/null +++ b/docs/security/data-portability/_index.md @@ -0,0 +1,4 @@ +--- +title: Data Portability +show_in_lists: true +--- \ No newline at end of file diff --git a/docs/security/data-portability/download-backups-locally/index.md b/docs/security/data-portability/download-backups-locally/index.md new file mode 100644 index 00000000000..24c8f06b14d --- /dev/null +++ b/docs/security/data-portability/download-backups-locally/index.md @@ -0,0 +1,53 @@ +--- +author: + name: Edward Angert + email: docs@linode.com +description: "Learn how to download a local copy of your Linode backups." +keywords: ["backup", "backups", "rsync", "restore", "local"] +license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' +modified: 2019-01-15 +modified_by: + name: Linode +published: 2018-08-08 +title: Download a Local Copy of your Linode Backup +--- + +The [Linode Backups](/docs/platform/disk-images/linode-backup-service/) service can create automatic and manual snapshots of your Linode. A completed backup can be directly restored to the origin Linode or to a new Linode in the same data center. These workflows make it easy to revert to a working configuration if you run into any unexpected issues with your software. + +Linode's backups are stored in a way that is only directly readable by the Linode Backups service. A common question for the service is how you can download the content from your Linode Backups to another storage location, like your home computer. This can be accomplished in two phases: + +1. [Restore a backup](#restore-from-a-backup) to a new or existing Linode. +2. Download either [specific files](#download-specific-files-or-directories-over-ssh) or the [entire disk image](#download-a-disk-over-ssh) from that Linode, as needed. + +## Before You Begin + +### Account Permissions and Billing + +Several of the steps in this guide involve adding services to or removing services from a Linode account. Visit our guide on [Users and Permissions](/docs/platform/manager/accounts-and-passwords/#users-and-permissions) for more information about restricted Linode users. + +Note that the cost of adding Backups service and adding a Linode to your account is billed, prorated per hour. If the backups service is only enabled for a few hours, you will only be charged for a few hours of the service. See the [Backups pricing details](/docs/platform/disk-images/linode-backup-service/#pricing) for more information. Likewise, when you create a Linode, you will be billed per hour that the Linode exists, whether it is powered on or not. + +The steps in this guide have been designed to minimize the potential costs associated with this process. Additionally, keep the following in mind: + +- Removing a Linode from your account also cancels the associated Backup service for that Linode. +- A Linode's backups are deleted when a Linode is deleted. +- If you choose to leave the Backups service enabled, or if you do not remove the additional Linode from your account, you will be automatically billed. If you only power the Linode off, [you will still be billed for it](/docs/platform/billing-and-support/billing-and-payments/#if-my-linode-is-powered-off-will-i-be-billed). + +### Enable Backups and Take a Snapshot + +These steps are the minimum required for the scope of this guide. Visit our [Backups guide](/docs/platform/disk-images/linode-backup-service/) for information about how to implement regular backups of your Linode. + +1. Go to your Linode's dashboard, click **Backups**, click **Enable backups for this Linode »** and confirm the additional cost per month. + +1. This guide focuses on saving a snapshot or specific backup. Click **Take a New Snapshot Now**. + + - The snapshot appears in the *Backup History* at the bottom of the page. + +{{< content "restore-backup-image-short" >}} + +{{< content "download-files-from-your-linode-shortguide" >}} + + +## Clean Up after Your Download + +After you've finished downloading your files or disks, you can optionally [delete the restored disks](/docs/platform/disk-images/disk-images-and-configuration-profiles/#removing-a-disk). If you created a new Linode to perform the restore, consider [deleting the Linode](/docs/platform/billing-and-support/billing-and-payments/#removing-services). As a reminder, billing for that Linode will continue automatically if you do not remove it. If you only power the Linode off, [you will still be billed for it](/docs/platform/billing-and-support/billing-and-payments/#if-my-linode-is-powered-off-will-i-be-billed). \ No newline at end of file diff --git a/docs/security/data-portability/download-files-from-your-linode-shortguide/index.md b/docs/security/data-portability/download-files-from-your-linode-shortguide/index.md new file mode 100644 index 00000000000..00c04b4fff1 --- /dev/null +++ b/docs/security/data-portability/download-files-from-your-linode-shortguide/index.md @@ -0,0 +1,76 @@ +--- +author: + name: Linode + email: docs@linode.com +description: "Learn how to download files, database dumps, or whole disks from your Linodes." +keywords: ["download", "files", "disk", "ssh"] +license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' +modified: 2018-09-25 +modified_by: + name: Linode +published: 2018-09-25 +title: Download Files from your Linode - Shortguide +headless: true +--- + +## Download Specific Files or Directories over SSH + +If you just need specific files from your Linode, you can download those over SSH. Downloading files over SSH can be done at a command-line interface, or with a graphical *SFTP* file browser. + +### Secure Copy Protocol (SCP) + +You can use SCP to retrieve a specific directory or file via the command-line. SCP is installed by default on most macOS and Linux systems, and is available with [Cygwin or PuTTY](/docs/networking/ssh/using-ssh-on-windows) for Windows. + +- The syntax for using SCP to copy a file from your Linode into a directory on another computer is: + + scp your_linode_username@your_linode_ip:/path/to/your/file.txt /path/to/your/local/directory/ + + The file will be saved inside `/path/to/your/local/directory/`. + +- To copy a file from your Linode to another computer and give it a specific name (in this case, `file.txt.backup`): + + scp your_linode_username@your_linode_ip:/path/to/your/file.txt /path/to/your/local/directory/file.txt.backup + +- To copy an entire directory: + + scp -r your_linode_username@your_linode_ip:/path/to/your/directory /path/to/your/local/directory + + If `/path/to/your/local/directory` already exists on your computer, then the copied directory will be placed inside `/path/to/your/local/directory` (i.e. `/path/to/your/local/directory/directory`). + + If `/path/to/your/local/directory` does not already exist, then the copied directory will be created with that name. + +For example: + +* Download an NGINX configuration file to your user's `/home` folder: + + scp your_linode_username@your_linode_ip:/etc/nginx/conf.d/example.com.conf ~/example.com.conf.backup + +* Download an Apache configuration file to your `/home` folder: + + scp your_linode_username@your_linode_ip:/etc/apache2/sites-available/example.com.conf ~/example.com.conf.backup + +* Copy the entire document root from a web server: + + scp -r your_linode_username@your_linode_ip:/var/www/html/ ~/html_backup + +If you intend to repeat this process regularly, consider [using rsync](/docs/security/backups/backing-up-your-data/#understand-the-rsync-command) to create additional local copies of your data. rsync is capable of performing incremental file copies, which means you do not have to fully transfer each file every time you download your data. + +### FileZilla + +FileZilla is a popular free and open source FTP, FTPS, and SFTP client which has a GUI but can also take CLI arguments. In contrast to to SCP, SFTP can list directory contents, create or delete files, and resume interrupted file transfers. + +See our [FileZilla guide](http://localhost:1313/docs/tools-reference/file-transfer/filezilla/) for more information. + +### Downloading Data from a Database + +Special care is needed when downloading data from a database. Before it can be downloaded, the data in a database needs to first be *dumped* to a file. This database dump file can then be transferred just as any other normal file type. + +- To create a dump of a MySQL (or MariaDB) database, [use the `mysqldump` command](/docs/databases/mysql/use-mysqldump-to-back-up-mysql-or-mariadb). **You can only use this tool if your database process is accessible and running.** + +- If your MySQL database won't run for some reason, follow the instructions for creating [*physical* backups](/docs/databases/mysql/create-physical-backups-of-your-mariadb-or-mysql-databases/). + +- If you use PostgreSQL, follow the [How to Back Up Your PostgreSQL Database](/docs/databases/postgresql/how-to-back-up-your-postgresql-database/) guide. + +## Download a Disk over SSH + +Downloading your disk will copy a `.img` file to your computer that encapsulates all of the data that is on your Linode’s disk. This *disk image* can later be re-uploaded to the Linode service at a later date, which can be useful if you'd like to temporarily remove your Linode and stop service. Follow our [Copy a Disk over SSH](/docs/platform/disk-images/copying-a-disk-image-over-ssh/) guide for further instructions. \ No newline at end of file diff --git a/docs/security/data-portability/download-files-from-your-linode/index.md b/docs/security/data-portability/download-files-from-your-linode/index.md new file mode 100644 index 00000000000..b4b94fc1ae0 --- /dev/null +++ b/docs/security/data-portability/download-files-from-your-linode/index.md @@ -0,0 +1,23 @@ +--- +author: + name: Linode + email: docs@linode.com +description: "Learn how to download files, database dumps, or whole disks from your Linodes." +keywords: ["download", "files", "disk", "ssh"] +license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' +modified: 2019-01-15 +modified_by: + name: Linode +published: 2018-09-25 +title: Download Files from Your Linode +--- + +There are a number of open source tools you can use to download files from your Linode. You have the option to either [download specific files and directories](#download-specific-files-or-directories-over-ssh) or [copy complete images](#download-a-disk-over-ssh) of your Linodes' disks. + +{{< content "download-files-from-your-linode-shortguide" >}} + +## Download Data from a Block Storage Volume + +1. [Attach and mount](/docs/platform/block-storage/how-to-use-block-storage-with-your-linode/#add-a-volume-from-your-account-s-volume-list) the block storage volume. + +2. Download files from it by following the same instructions in the [Download Specific Files or Directories over SSH](#download-specific-files-or-directories-over-ssh) section of this guide. \ No newline at end of file diff --git a/docs/tools-reference/custom-kernels-distros/install-freebsd-on-linode/index.md b/docs/tools-reference/custom-kernels-distros/install-freebsd-on-linode/index.md index d36cce30209..ce54a46cade 100644 --- a/docs/tools-reference/custom-kernels-distros/install-freebsd-on-linode/index.md +++ b/docs/tools-reference/custom-kernels-distros/install-freebsd-on-linode/index.md @@ -5,7 +5,7 @@ author: description: 'FreeBSD is a free and open source operating system based on the Berkeley Software Distribution from the late 1970''s. Today FreeBSD is used all over the world.' keywords: ["freebsd", "bsd"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -modified: 2019-01-02 +modified: 2019-01-07 modified_by: name: Linode published: 2016-03-14 @@ -70,13 +70,13 @@ Begin by creating the Linode and making some preliminary changes. curl $latest | dd of=/dev/sda - As of this guide's writing, the latest release is version 10.2. The following command will download the 10.2 release image to your Linode: + As of this guide's writing, the latest release is version 12.0. The following command will download the 12.0 release image to your Linode: - curl ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/10.2/FreeBSD-10.2-RELEASE-amd64-memstick.img | dd of=/dev/sda + curl ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/12.0/FreeBSD-12.0-RELEASE-amd64-memstick.img | dd of=/dev/sda 6. When the command finishes, reboot into your **Installer profile**. -7. Go to the **Remote Access** tab in the Linode Manager. Access your Linode using [Glish](/docs/platform/manager/using-the-linode-graphical-shell-glish/) to start the installation. +7. Go to the **Remote Access** tab in the Linode Manager. Access your Linode using [Glish](/docs/platform/manager/using-the-linode-graphical-shell-glish/) to start the installation. Note that Glish **must** be used to complete the installation of FreeBSD. ## Installing FreeBSD diff --git a/docs/tools-reference/file-transfer/filezilla-shortguide/index.md b/docs/tools-reference/file-transfer/filezilla-shortguide/index.md deleted file mode 100644 index 3429e9c3a69..00000000000 --- a/docs/tools-reference/file-transfer/filezilla-shortguide/index.md +++ /dev/null @@ -1,27 +0,0 @@ ---- -author: - name: Linode - email: docs@linode.com -description: 'Shortguide which shows how to use FileZilla after it has been installed.' -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -keywords: [] -aliases: ['networking/file-transfer/transfer-files-filezilla/'] -modified: 2018-08-14 -modified_by: - name: Linode -published: 2018-08-14 -title: 'Transfer Files with FileZilla' -headless: true ---- - -1. Open FileZilla from your Windows start menu, OS X Launchpad, or the launcher provided by your Linux distribution of choice. - -1. Enter your Linode's IP address or domain name in the `Host` field. Enter the account username you wish to connect as in the `Username` field. Please note that this must be a user account on your Linode; if in doubt, enter `root` to log in as the root user. Enter the account's password in the `Password` field, and enter "22" in the `Port` field. Click **Quickconnect** to initiate the file transfer session. - - {{< image src="filezilla-quick-connect.png" alt="Quickconnect" title="FileZilla Quickconnect screenshot" >}} - -1. If this is the first time you've connected to your Linode with an SSH or SFTP program, you'll receive a warning that the host key is unknown. Place a check mark in the box next to `Always trust this host, add this key to the cache`. Checking this box prevents further warnings unless the key presented to FileZilla changes; this should only happen if you reinstall the remote server's operating system. - - {{< image src="filezilla-unknown-key.png" alt="Unknown Key" title="FileZilla Unknown Key warning" >}} - -1. Click the **OK** button to proceed. You'll be presented with a split view, with your local filesystem on the left and your Linode's filesystem on the right. You may transfer files by dragging and dropping them between each side. diff --git a/docs/tools-reference/file-transfer/filezilla-shortguide/filezilla-quick-connect.png b/docs/tools-reference/file-transfer/filezilla/filezilla-quick-connect.png similarity index 100% rename from docs/tools-reference/file-transfer/filezilla-shortguide/filezilla-quick-connect.png rename to docs/tools-reference/file-transfer/filezilla/filezilla-quick-connect.png diff --git a/docs/tools-reference/file-transfer/filezilla-shortguide/filezilla-unknown-key.png b/docs/tools-reference/file-transfer/filezilla/filezilla-unknown-key.png similarity index 100% rename from docs/tools-reference/file-transfer/filezilla-shortguide/filezilla-unknown-key.png rename to docs/tools-reference/file-transfer/filezilla/filezilla-unknown-key.png diff --git a/docs/tools-reference/file-transfer/filezilla/index.md b/docs/tools-reference/file-transfer/filezilla/index.md index e344f444ba2..c2f20464d1f 100644 --- a/docs/tools-reference/file-transfer/filezilla/index.md +++ b/docs/tools-reference/file-transfer/filezilla/index.md @@ -3,72 +3,70 @@ author: name: Linode email: docs@linode.com description: 'Securely copying files to and from your Linode with FileZilla, a free and open source file transfer client for Linux, OS X, and Windows systems.' -keywords: ["filezilla", "ftp", "linux scp", "sftp", "linux sftp program", "linux ftp"] +keywords: ["filezilla", "ftp", "scp", "sftp", "transfer"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' aliases: ['networking/file-transfer/transfer-files-filezilla/'] -modified: 2014-10-13 +modified: 2019-01-15 modified_by: - name: James Stewart + name: Linode published: 2014-10-13 title: 'Transfer Files with FileZilla' external_resources: - - '[FileZilla Documentation](http://wiki.filezilla-project.org/Documentation)' - - '[FileZilla SSH Key Documentation](https://wiki.filezilla-project.org/Howto#SFTP_using_SSH2:_Key_based_authentication)' - - '[Tools & Reference](/docs/tools-reference/)' + - '[FileZilla Documentation](https://wiki.filezilla-project.org/Documentation)' --- -FileZilla is a free, open source file transfer program written for Linux, Mac OS X, and Windows systems. It implements several file transfer protocols, most notably SFTP via SSH. This tool allows you to securely transfer files to and from your Linode using an encrypted channel, avoiding the security problems and usability issues inherent in traditional FTP client/server systems. FileZilla can send both your login credentials and file transfers over the network securely encrypted (provided you're using SFTP), while standard FTP clients send this information as plaintext. - ![Transfer Files with FileZilla](transfer-files-with-filezilla.png "Transfer Files with FileZilla") -## Prerequisites +[FileZilla](https://filezilla-project.org/) is a free and open source multi-platform file transfer application. It implements several file transfer protocols, most notably SFTP via SSH. SFTP encrypts both your login credentials and files before sending over a network, while FTP clients communicates only as plaintext. -Prior to following this guide, you will need to ensure that the following steps have been taken on your Linode. -- Created a user as per the instructions in our [securing your server](/docs/security/securing-your-server/) guide. +## Before You Begin -- Ensure that you can connect to your Linode [via SSH](/docs/getting-started#connect-to-your-linode-via-ssh). +- You will need root access to your Linode, or a user account with `sudo` privilege. Note that if you transfer files as `root`, you may need to change file [ownership and permissions](/docs/tools-reference/linux-users-and-groups) afterwards. +- Set your system's [hostname](/docs/getting-started/#setting-the-hostname). +- Update your system. -- This guide is written for a non-root user. Commands that require elevated privileges are prefixed with ``sudo``. If you're not familiar with the ``sudo`` command, you can check our [Users and Groups](/docs/tools-reference/linux-users-and-groups) guide. -{{< note >}} -If you wish, you may use the `root` account on your Linode to perform file transfers, although you may need to change [file ownership and permissions](/docs/tools-reference/linux-users-and-groups) on the server after doing so. -{{< /note >}} +## Install FileZilla -## Installing FileZilla +1. Download FileZilla [here](https://filezilla-project.org/download.php) for macOS and Windows. Use your distribution's package manager to install FileZilla on Linux: -Windows and OS X users can download FileZilla [here](https://filezilla-project.org/download.php?show_all=1). + - CentOS: -Linux users can run the following commands to install FileZilla. + sudo yum install filezilla -Debian/Ubuntu: + - Debian / Ubuntu: - sudo apt-get install filezilla + sudo apt install filezilla -CentOS/Fedora +## Using FileZilla - sudo yum install filezilla +1. Launch FileZilla and enter your Linode's IP address or domain name in the `Host` field. Enter the username and password for the account on the Linode you want to connect as, and enter "22" in the `Port` field. Click **Quickconnect** to initiate the file transfer session. -## Using FileZilla + ![FileZilla Quickconnect](filezilla-quick-connect.png "FileZilla Quickconnect") + +1. If this is the first time you've connected to your Linode by SSH or SFTP, you'll receive a warning that the host key is unknown. Verify the host key fingerprints and if valid, check the box next to `Always trust this host, add this key to the cache`. This prevents further warnings unless the key presented to FileZilla changes, which should only happen if you reinstall the remote server's operating system. + + ![FileZilla unknown SSH host key](filezilla-unknown-key.png "FileZilla unknown SSH host key") -{{< content "filezilla-shortguide" >}} +1. Click the **OK** button to proceed. You'll be presented with a split view, with your local filesystem on the left and your Linode's filesystem on the right. You can transfer files by dragging and dropping them from one side to the other. -### If You Use Public Key Authentication +## FileZilla with Public Key Authentication -If you have followed our [Securing Your Server](/docs/security/securing-your-server) guide, you won't be able to connect to your Linode using a password. If you are using Linux or OS X, the keys that you generated while following that guide will be automatically used for authentication. +If you followed our [Securing Your Server](/docs/security/securing-your-server) guide, you won't be able to connect to your Linode using a password. If you are using Linux or macOS, the SSH keypair generated while following that guide will be automatically used for authentication. {{< note >}} -If you are using OS X, the passphrase for your key will need to be stored in your keychain in order to successfully connect via SSH key. FileZilla will not be able to use a key that was generated without a passphrase to connect to your Linode. +If you are using macOS, the passphrase for your key will need to be stored in your keychain to successfully connect using an SSH keypair. FileZilla will not be able to use a key that was generated without a passphrase to connect to your Linode. {{< /note >}} If you are using Windows, you'll need to follow a few additional steps to enable key based authentication. The instructions below assume that you have already completed the guide for [generating your SSH key with Putty](/docs/security/use-public-key-authentication-with-ssh#windows-operating-system). 1. Install Pageant from the [Putty site](http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html) -2. Start Pagent from your Start menu. In your system tray, you'll see the Pageant icon appear. +1. Start Pagent from your Start menu. In your system tray, you'll see the Pageant icon appear. -3. Right-click the icon and select `Add Key`. +1. Right-click the icon and select `Add Key`. -4. Navigate to the location where you stored your keys and select your private key (PPK) file. You will be prompted for your passphrase if you provided one when creating the key. +1. Navigate to the location where you stored your keys and select your private key (PPK) file. You will be prompted for your passphrase if you provided one when creating the key. -5. Launch FileZilla and connect to your server with your username and an empty password. Your key will be used as authentication as long as the Pagent software is running. +1. Launch FileZilla and connect to your server with your username and an empty password. Your key will be used as authentication as long as the Pagent software is running. diff --git a/docs/troubleshooting/rescue-and-rebuild/index.md b/docs/troubleshooting/rescue-and-rebuild/index.md index b93d8bceeeb..73b1ecc789a 100644 --- a/docs/troubleshooting/rescue-and-rebuild/index.md +++ b/docs/troubleshooting/rescue-and-rebuild/index.md @@ -6,7 +6,7 @@ description: 'Our guide to rescuing and rebuilding your Linode.' keywords: ["rescue", "rebuild"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' aliases: ['rescue-and-rebuild/','troubleshooting/finnix-rescue-mode/'] -modified: 2016-08-18 +modified: 2019-01-14 modified_by: name: Linode published: 2012-05-31 @@ -130,7 +130,7 @@ The Finnix recovery distribution does not automatically start an SSH server, but 2. Enter the password for the `root` user. 3. Start the SSH server by entering the following command: - /etc/init.d/ssh start + service ssh start Now you can access mounted disks with an SFTP client by using the `root` user and the password you just set. For instructions on connecting with an SFTP client, see the [File Transfer reference manuals](/docs/tools-reference/file-transfer/). For instructions on copying an entire disk over SSH, see [Copy a Disk Over SSH](/docs/platform/disk-images/copying-a-disk-image-over-ssh/). diff --git a/docs/web-servers/nginx/build-nginx-with-pagespeed-from-source/index.md b/docs/web-servers/nginx/build-nginx-with-pagespeed-from-source/index.md index ef1bcdd42d0..a1ef8607f85 100644 --- a/docs/web-servers/nginx/build-nginx-with-pagespeed-from-source/index.md +++ b/docs/web-servers/nginx/build-nginx-with-pagespeed-from-source/index.md @@ -24,6 +24,10 @@ There are currently two ways to get PageSpeed and NGINX working together: - Compile NGINX with support for PageSpeed, then compile PageSpeed. - Compile PageSpeed as a [dynamic module](https://www.nginx.com/blog/compiling-dynamic-modules-nginx-plus/) to use with NGINX, whether NGINX was installed from source or a binary. + {{< note >}} +Installing NGINX from source requires several manual installation steps and will require manual maintenance when performing tasks like version upgrades. To install NGINX using a package manager see the [NGINX](/docs/web-servers/nginx/) section. + {{}} + This guide will show how to compile both NGINX and PageSpeed. If you would prefer to use PageSpeed as a module for NGINX, see [this NGINX blog post](https://www.nginx.com/blog/optimize-website-google-pagespeed-dynamic-module-nginx-plus/) for instructions. @@ -65,7 +69,11 @@ configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-p ## Build NGINX and PageSpeed -The official [PageSpeed documentation](https://www.modpagespeed.com/doc/build_ngx_pagespeed_from_source) provides a bash command which pulls script to automate the installation process. +The official [PageSpeed documentation](https://www.modpagespeed.com/doc/build_ngx_pagespeed_from_source) provides a bash script to automate the installation process. + +{{< note >}} +The automated installation script will install several compilation tools needed to install PageSpeed. If you are using a production environment, ensure you uninstall any packages that are no longer needed after the installation has completed. +{{}} 1. If you plan to serve your website using TLS, install the SSL libraries needed to compile the HTTPS module for NGINX: