diff --git a/README.md b/README.md index ee8539dad2..eb25f2328b 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,6 @@ -[NuGet Gallery](http://nuget.org/) — Where packages are found +[NuGet Gallery](http://nuget.org/) — Where packages are found ======================================================================= -[![Build status](https://ci.appveyor.com/api/projects/status/6ob8lbutfecvi5n3/branch/master?svg=true)](https://ci.appveyor.com/project/NuGetteam/nugetgallery/branch/master) - This is an implementation of the NuGet Gallery and API. This serves as the back-end and community website for the NuGet client. For information about the NuGet project, visit the [Home repository](https://github.com/nuget/home). @@ -49,6 +47,10 @@ You can undo this with this command: This should help prevent unwanted file commits. +## Deploy + +You will find instructions on how to deploy the Gallery to Azure [here](https://github.com/NuGet/NuGetGallery/blob/master/docs/Deploying/README.md). + ## Contribute If you find a bug with the gallery, please visit the [Issue tracker](https://github.com/NuGet/NuGetGallery/issues) and create an issue. If you're feeling generous, please search to see if the issue is already logged before creating a diff --git a/appveyor.yml b/appveyor.yml deleted file mode 100644 index f211ee8502..0000000000 --- a/appveyor.yml +++ /dev/null @@ -1,8 +0,0 @@ -version: 3.1.{build} -configuration: Release -nuget: - disable_publish_on_pr: true -build_script: -- cmd: build.cmd -artifacts: -- path: 'artifacts\*.nupkg' diff --git a/docs/APIv3.md b/docs/APIv3.md deleted file mode 100644 index 19e010ab52..0000000000 --- a/docs/APIv3.md +++ /dev/null @@ -1,12 +0,0 @@ -# Deploying the v2 Gallery in APIv3 land -The v2 Gallery can be deployed using mostly the same process as in v2. For each deployment, a new SQL user should be created using the following command: - -```posh -nucmd db createuser -sv v2gallery -s dbo -db legacy -dc 0 -clip -``` - -**NOTE:** The server and database name should match the "Sql.Legacy" configuration value in the V3 config. The user name and password will be different. - -This will place a new connection string in the clipboard, which you can use to update the "Gallery.SqlServer" setting. - -Also, since the Work service and the V2Gallery service BOTH access the Legacy storage account, the Account Keys must be synchronized between the "Storage.Legacy" setting in the APIv3 service config and the "Gallery.AzureStorageConnectionString" value in the V2 config. diff --git a/docs/Deploying/CloudServices/README.md b/docs/Deploying/CloudServices/README.md deleted file mode 100644 index 6c9868fbd2..0000000000 --- a/docs/Deploying/CloudServices/README.md +++ /dev/null @@ -1,41 +0,0 @@ -# Deploying the NuGet Gallery to Windows Azure Cloud Services - -## Setting up resources -To run the NuGet Gallery on Windows Azure Cloud Services you need to provision the following Azure resources: - -1. An Azure Cloud Service -2. An Azure SQL Database (preferably in a dedicated Azure SQL Server) to hold the package metadata -3. An Azure Storage account to hold package files, diagnostics data, etc. - -It is assumed you've already provisioned the SQL Database and Storage Account using the [main guide](../README.md). - -## Provisioning the Frontend -Create an Azure Cloud Service. Poof, done. Ok, not quite. Grab the [src\NuGetGallery.Cloud\ServiceConfiguration.Local.cscfg](../../../src/NuGetGallery.Cloud/ServiceConfiguration.Local.cscfg) file and copy it to somewhere **OUTSIDE** the repository. Give it a name that you'll remember like "MyNuGetGallery.TestEnvironment.cscfg". Update the settings as per the comments inside the file using the connection strings you received while provisioning resources for the site (SQL DB, Azure Storage, etc.). - -Save this file somewhere safe and **SECURE** as it now has passwords for your database inside! - -## Deploying the Service -These steps should be repeated each time you have new code to deploy. - -### Migrate the Database -Migrate the database by running "Update-Database" from the Package Manager Console in a Visual Studio session with NuGetGallery.sln. You must select "NuGetGallery" as the default project. Use the "-ConnectionString" parameter to specify the connection string to the target database. - -We attempt to ensure most migrations are additive, but please verify this before migrating your database. Our development process attempts to ensure that old code can read new data for a short transition period. If you keep your code up to date with ours (we deploy approximately fortnightly) then you should be able to match this process. - -### Updating Service Credentials -We highly recommend updating service credentials (SQL passwords, Azure Storage keys, etc.) on each deployment to reduce the risk of credential leakage. Some incomplete notes on doing this are listed below: - -1. Use the CreateSqlUser task in the 'galops' executable (implemented in NuGetGallery.Operations) to create a SQL User suitable for the Gallery Frontend and update the SQL Connection String for the Frontend to use it. NOTE: DO NOT remove the old user until the old code is offline. ALSO: DO NOT use this user in the Backend as the Backend requires administrator permissions. -2. Regenerate the Azure Storage keys that are NOT CURRENTLY BEING USED by the live service and update the following settings with it. NOTE: There are multiple storage accounts (Primary, Backup and optionally Diagnostics). All storage account keys should be refreshed like this. - -### Deploy the package -1. Click on the cloud service -2. Click "Staging" at the top -3. Click "Update" at the bottom -4. Upload/Select from blob storage the CSCFG and CSPKG files produced by building the solution -5. Select "ALL" roles and check all the checkboxes -6. Enter a meaningful name for the deployment (i.e. "Oct14 @ 1529 (deadbeef21)", where deadbeef21 is the commit hash of the build being used) -7. GO! -8. Once deployment completes, verify the staging URL -9. If the staging URL checks out, VIP Swap! -10. Repeat steps 1-7 to deploy the same code to the Staging slot so you can easily make configuration changes and VIP Swap to reduce downtime. \ No newline at end of file diff --git a/docs/Deploying/README.md b/docs/Deploying/README.md index e116c36922..703c9b2081 100644 --- a/docs/Deploying/README.md +++ b/docs/Deploying/README.md @@ -1,119 +1,61 @@ # Deploying the NuGet Gallery -To run the NuGet Gallery you need to provision the following common resources: +To run the NuGet Gallery in Azure you need to provision the following resources: -1. An SQL Database to hold the package metadata. +1. A SQL Database to hold the package metadata. 2. A location in which to store package files. The Gallery supports two at the moment: Local File System and Azure Storage Account. 3. A Web Frontend to host the Gallery. -4. (Optional) A Worker Backend to perform offline processing tasks. ## Deploying to Azure -We suggest using Windows Azure for hosting the gallery, as that is the environment used by http://www.nuget.org itself. When doing so, we suggest using Azure SQL Databases (formerly SQL Azure) for the database and Azure Storage Accounts to store package files. +We suggest using Windows Azure for hosting the gallery, as that is the environment used by http://www.nuget.org itself. When doing so, we suggest using Azure SQL Databases for the database and Azure Storage Accounts to store package files. -We support two profiles of Azure deployment: [Azure Websites](Websites) and [Azure Cloud Services](CloudServices). Before using those guides, however, you need to ensure you provisiong the supporting resources (Database, Storage, etc.). +This guide will instruct you on hosting the Gallery to an Azure App Service. We will start with provisiong the supporting resources (Database, Storage, etc.). ## Provisioning for Azure ### Provisioning a Database -We recommend provisioning a dedicated Azure SQL Databases Server for the Gallery. If you are going to use our backend worker to generate statistics reports from data in a NuGet Warehouse, we recommend provisioning a separate Server for that database in production (the Warehouse may be co-located with the Gallery database if necessary though, and we do co-locate them in our non-production environments). +We recommend provisioning a dedicated Azure SQL Databases Server for the Gallery. -So, first create a database: - -![Creating the Database](images/01-CreateDB.png) - -Next, we need to create a user for the site to use. Open the **server** in the Azure Portal and click on the URL under `Manage URL` - -![Manage URL](images/02-ManageUrl.png) - -Type "master" in the database field and log in using the SA password for the server (which you should have copied down when you created the server ;)). - -If you receive an error about firewall rules, go to the SQL Server entry in the Azure Portal, click the Configure tab and add your current client IP address to the list of allowed IP addresses (don't forget to click Save!). While here, double check that Allow Windows Azure Services is set to "YES". - -Once logged in to the SQL management portal, create a new query and type the following (of course, replace 'some password here' with an actual, secure, password :)): - -```SQL -CREATE LOGIN [nuget-site] WITH PASSWORD='some password here' -``` - -We use "nuget-site" as the name, but feel free to use any user name you want. - -Now, log out and log in again, but this time, specify the name of the gallery database in the "Database" field (i.e. NuGetGallery). Then, create a new query and run these scripts (one at a time): - -```SQL -CREATE USER [nuget-site] FROM LOGIN [nuget-site] +Follow the instrctions [here](https://docs.microsoft.com/en-us/azure/sql-database/sql-database-get-started-portal) to create an Azure SQL DB. +Copy the connection string from the portal. It should look something like: ``` - -```SQL -EXEC sp_addrolemember 'db_owner', 'nuget-site'; +Server=[servername].database.windows.net;Database=NuGetGallery;User ID=[username];Password=[password];Trusted_Connection=False;Encrypt=True ``` -If you used a different site username, use it in place of 'nuget-site' in the scripts. +Now, it's time to update your new DB with the Gallery SQL schema. -Now, log off the management portal and switch to VS. Open the NuGetGallery solution and expand the "Package Manager Console" tool window: +1. Open the NuGetGallery solution in Visual Studio. -![Package Manager Console](images/03-PackageManagerConsole.png) - -Craft your connection string using notepad or some other text editor. You want it to take the following form: +2. Open the [web.config](https://github.com/NuGet/NuGetGallery/blob/master/src/NuGetGallery/Web.config#L183) and replace the Gallery.SqlServer connection string with this value. -``` -Server=[servername].database.windows.net;Database=NuGetGallery;User ID=sa@[servername];Password=[sapassword];Trusted_Connection=False;Encrypt=True -``` +3. Expand the "Package Manager Console" tool window: -Replacing '[servername]', 'sa' and '[sapassword]' with the name of your Azure SQL Server, the SA user name and the SA password respectively. Keep this handy for the next step but make sure you store it securely, as this contains a password which grants **full admin access** to your database. +![Package Manager Console](images/03-PackageManagerConsole.png) -In the Package Manager console, type the following command +4. In the Package Manager console, type the following command: ```PowerShell -Update-Database -ConnectionString "[ConnectionString]" -ConnectionProviderName System.Data.SqlClient +Update-Database -ConfigurationTypeName MigrationsConfiguration ``` -Replacing '[ConnectionString]' with the connection string you just crafted. The command should succeed and you should have a fully prepared database! - ### Provisioning Storage Accounts -We also recommend the following Storage Accounts for each environment you intend to deploy (i.e. development, test, production, etc): - -1. A Primary Storage account to hold package files and other supporting content. -2. A Backup Storage account to hold package and database backups in case the primary storage account is lost. -3. A Diagnostics Storage account to hold logs and other data. In non-production environments, we recommend using the Primary storage account to hold this data. In production, however, your traffic may be large enough that you may wish to move diagnostics data (Web Server logs, etc.) to a separate storage account - -### Gathering configuration -Now that you've got the database ready, all the resources are ready for you to deploy the site. First, though, we need to configure the Website so that it will be able to talk to the database and storage when it is deployed. To do this, go to the website in the Azure Portal and select the Configure tab. - -Craft a connection string using the 'nuget-site' user you created earlier (by taking the connection string above and replacing sa and '[sapassword]' with the username/password for that user). Then, in the portal, add the database connection string like so: - -![Adding Connection String](images/04-ConnectionString.png) - -Then, go to the storage account you created in the portal and select "Manage Keys". Use the name and primary key on that page to build a connection string like this: +Follow the instruction [here](https://docs.microsoft.com/en-us/azure/storage/common/storage-quickstart-create-account?tabs=portal) to create an Azure storage account. +Copy the connection string from the portal. It should like something like: ``` DefaultEndpointsProtocol=https;AccountName=[account name];AccountKey=[primary key]; ``` - -Now that you have your dependent resources and their configuration, move on to... +To configure Gallery to use your new storage account: +1. Open the [web.config](https://github.com/NuGet/NuGetGallery/blob/master/src/NuGetGallery/Web.config#L27) +2. Set Gallery.StorageType to 'AzureStorage' +3. Replace all settings starting with 'Gallery.AzureStorage.' with your connection string. ## Deploying the Frontend/Backend -Once you've provisioned these resources, continue to the guide for deploying to the profile of your choice: -1. [Azure Cloud Services](CloudServices/README.md) - Recommended for extremely high availability services -2. [Azure Websites](Websites/README.md) - Recommended for simple deployment scenarios - -We do not currently recommend Azure Websites for extremely high availability deployments of the Gallery. Other Azure features for traffic management, such as Azure Traffic Manager, do not support Azure Websites at this time. Also, Azure Websites does not provide the Production/Staging VIP Swap mechanism used in Cloud Services to allow for staging of production changes. - -## Making an Admin -Once you've got your gallery deployed, you probably want an admin user. That's pretty easy to do. First, register your admin user through the site. Then log in to the database using the Azure SQL Management Portal and the 'nuget-site' user (as we did above). Then run this SQL: - -```SQL -INSERT INTO Roles(Name) VALUES('Admins') - -DECLARE @adminId int -SELECT @adminId = [Key] FROM Roles WHERE Name = 'Admins' - -DECLARE @userId int -SELECT @userId = [Key] FROM Users where Username = 'username' - -INSERT INTO UserRoles(UserKey, RoleKey) VALUES(@userId, @adminId) -``` +You are almost done! Here are additional configurations in web.config: +1. Gallery.SiteRoot - set with the URL of your Gallery website. For example: _https://mygallery.azurewebsites.net_ +2. Gallery.SmtpUri (optional)- set SMTP credentials if you would like to receive e-mails from the service. -Replacing 'username' with the name of the user you just created. Now log out and back in with that user and you should see the Admin tab! **NOTE** The name of the role is important, it is "Admins" (plural!). \ No newline at end of file +Now you are ready to publish the Gallery to your own Azure app service. To do this through Visual Studio follow the instructions [here](https://docs.microsoft.com/en-us/visualstudio/deployment/quickstart-deploy-to-azure). diff --git a/docs/Deploying/Websites/README.md b/docs/Deploying/Websites/README.md deleted file mode 100644 index 2a88a8de6b..0000000000 --- a/docs/Deploying/Websites/README.md +++ /dev/null @@ -1,51 +0,0 @@ -# Deploying the NuGet Gallery to Windows Azure Websites - -## Setting up resources -To run the NuGet Gallery on Windows Azure Websites you need to provision the following Azure resources: - -1. An Azure Website running .Net 4.5 to run the gallery frontend -2. An Azure SQL Database (preferably in a dedicated Azure SQL Server) to hold the package metadata -3. An Azure Storage account to hold package files, diagnostics data, etc. -4. [Optional] An Azure Cloud Service to run the gallery backend worker -5. [Optional] An Azure Cloud Service to provide SSL Forwarding and Traffic Management to the gallery frontend - -It is assumed you've already provisioned the SQL Database and Storage Account using the [main guide](../README.md). - -## Provisioning the Frontend -Create an Azure Website and configure it for deployment however you choose (Git deployment, FTP, etc.). - -Using the connection strings you received while provisioning resources for the site (SQL DB, Azure Storage, etc.), go to the Configure tab of the website and set the AppSettings. To determine what to set, open the [src\NuGetGallery\Web.config](../../../src/NuGetGallery/Web.config) file up and use the comments in the AppSettings section to assist you. - -Save the changes and get ready to deploy! - -## Migrate the Database -Migrate the database by running "Update-Database" from the Package Manager Console in a Visual Studio session with NuGetGallery.sln. You must select "NuGetGallery" as the default project. Use the "-ConnectionString" parameter to specify the connection string to the target database. - -We attempt to ensure most migrations are additive, but please verify this before migrating your database. Our development process attempts to ensure that old code can read new data for a short transition period. If you keep your code up to date with ours (we deploy approximately fortnightly) then you should be able to match this process. - -## Deploying the Frontend with Git -(If you want to use a different deployment mechanism, you're on your own :)) - -Go back to the dashboard tab and set up deployment from source control. Assuming you cloned this repo from Git, you probably want to choose "Local Git Repository". That is the case we will cover here. - -Once you complete this wizard, you'll need to set up deployment credentials. I'll assume you've got that under control ;). Go to the gallery and checkout the branch you want to deploy (say 'master'): - -``` -git checkout master -``` - -Add Azure as remote and push that branch up! - -``` -git remote add azure [git url] -git push azure master -``` - -NOTE: If you don't use master, make sure to update the "Branch to Deploy" setting in the Configure tab. - -After pushing, the site will build, which may take a while. Once finished, just browse to the site and it should start up! The first request may take quite a while, so be prepared to wait a few minutes. - -Now that you've got the site up, try registering a user and uploading a package! - -## Deploying the backend -TODO. diff --git a/docs/Deploying/images/01-CreateDB.png b/docs/Deploying/images/01-CreateDB.png deleted file mode 100644 index eed3ad8cfa..0000000000 Binary files a/docs/Deploying/images/01-CreateDB.png and /dev/null differ diff --git a/docs/Deploying/images/02-ManageUrl.png b/docs/Deploying/images/02-ManageUrl.png deleted file mode 100644 index 20756eba5c..0000000000 Binary files a/docs/Deploying/images/02-ManageUrl.png and /dev/null differ diff --git a/docs/Deploying/images/04-ConnectionString.png b/docs/Deploying/images/04-ConnectionString.png deleted file mode 100644 index da8932bfac..0000000000 Binary files a/docs/Deploying/images/04-ConnectionString.png and /dev/null differ diff --git a/docs/FunctionalTests.md b/docs/FunctionalTests.md deleted file mode 100644 index 80dc01d79d..0000000000 --- a/docs/FunctionalTests.md +++ /dev/null @@ -1,40 +0,0 @@ -## Adding new tests: - - * Add a new unit test or web test based on the feature. - - * For unit tests, extend the test from "GalleryTestBase" class. This is a base class which does some initialization and cleanup. - - * Make sure that Urls are picked up from "UrlHelper" and any const strings are added to "Constants". - - * For validations, check if the helper you are looking for is present in the helper project. If not add a new one. - - * Even for web tests, try to make the validations as much client SDK based as possible. Example : PackagesPageTest. - - * Add the tests to appropriate test suite in the NugetGallery.FunctionalTests.vsmdi file. - - - -## Running tests locally : - - * The tests pick up the test email account, password and mail server from the environment variables "TestAccountName", "TestAccountPassword" and "TestEmailServerHost" respectively. - Make sure that these variables are set to appropriate values before running tests. Also execute the Nuget.exe SetAPI command to set the API keys for the test account which you would be using. - A script file which does these pre-req steps can be found @ http://sharepoint/sites/nuget/Internal%20Documents/FunctionalTests/TestRunSetupScript.cmd. - Folks inside microsoft corpnet shall copy it locally to "FunctionalTests\Scripts" folder and run it. - - - * Execute "RunTests.cmd" from the scripts folder to run all tests. [this would run all happy path tests which doesn’t require read-only mode or SQL to be down). - RunTests.cmd takes an input parameter to set the Gallery Url to point to. - - Example: RunTests.cmd https://preview.nuget.org/ would run the tests against preview. - By default it points to the bvt test environment. - - * Execute "RunSpecificTests.cmd" to run specific tests based on a criteria. - - Example: To run all tests which has "Download" in the test name run the below command : - RunSpecificTests.cmd Download. - - - * To run tests from VS IDE, use Test Explorer to run unit tests. For web tests, open the test in IDE code editor and use "Run coded web performance test" context menu. Same step for debugging tests as well. - - - diff --git a/docs/v3/WorkService/Deploying.md b/docs/v3/WorkService/Deploying.md deleted file mode 100644 index 49373f1248..0000000000 --- a/docs/v3/WorkService/Deploying.md +++ /dev/null @@ -1,48 +0,0 @@ -# Deploying the Work Service - -## Initial Provisioning -The work service requires the following resources: - -1. A SQL Server with a database: 'nuget-[environment]-[dc]' (for example: 'nuget-int-0') -2. A Primary Storage Account with the name: 'nuget[environment][dc]' (for example: 'nugetint0') -3. An SSL Certificate for the target service -4. An RDP Certificate for the target service - -Optional resources include: - -1. The Connection String to the Legacy (APIv2) Storage account and Database -2. A Backup storage account -3. A Warehouse database connection string - -To provision the service, create a Cloud Service in Azure with the name 'nuget-[environment]-[dc]-[service]' (for example: 'nuget-int-0-work') and upload the SSL and RDP certificates. - -Provisioning the Primary SQL Database: - -1. Create a database -2. Create two logins in the primary SQL Server with random passwords named 'primary' and 'secondary' -3. Publish the NuGet.Services.Work.Database project to the SQL Database -4. Create the two users using the following script (replace '[primary]' with '[secondary]' to create the second user): - - CREATE USER [primary] FROM LOGIN [primary] - -5. Grant those users access to the work schema (replace '[primary]' with '[secondary]' to authorize the second user): - - GRANT CONTROL, ALTER ON SCHEMA :: [work] TO [primary] - -6. Put primary's credentials in the CSCFG. - -## Initial Configuration -1. Copy an existing config from the cscfg store -2. Update the 'Host.*' settings with the values for this service -3. Set the Sql.* and Storage.* settings as necessary -4. Add the thumbprints of the SSL and RDP certificates to the config -5. Use the Get-RemoteDesktopPassword function in the NuGet Ops console (".\ops" from the root of a NuGetGallery enlistment) to generate a new password. Update the Sharepoint site with the plaintext value and the cscfg with the ciphertext value. - -## Per-Deployment Reconfiguration -1. Determine which Database user is going to be the 'target' user by looking at the current connection string. If it is 'primary', then you will be using 'secondary' in the below steps, otherwise, you'll be using 'primary' (this is key rotation, just like with Storage Accounts). In the below steps 'target' refers to the user you identified - 1. Change the password for the 'target' login on the SQL Server - 2. Update the cscfg with target's credentials -2. Rotate the Storage Account keys in the cscfg as well - 1. If the cscfg uses the Primary key, regenerate the Secondary and put it in the cscfg - 2. If the cscfg uses the Secondary key, regenerate the Primary and put it in the cscfg -3. Generate a new RDP Password using the Get-RemoteDesktopPassword function in the NuGet Ops console. diff --git a/src/NuGetGallery/Web.config b/src/NuGetGallery/Web.config index e7a37ab0db..b1790967bc 100644 --- a/src/NuGetGallery/Web.config +++ b/src/NuGetGallery/Web.config @@ -72,10 +72,6 @@ - - - - @@ -138,7 +134,7 @@ - +