This sample provides an Azure Function based implementation of running the Cloud Scanner tool
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
RunRules Deploy to Functions app on commit to master Oct 26, 2018
ScanResources Setup Travis build (#4) Oct 25, 2018
StoreResources Setup Travis build (#4) Oct 25, 2018
TagResources Setup Travis build (#4) Oct 25, 2018
TaskScheduler Setup Travis build (#4) Oct 25, 2018
config-generator fix setup file Oct 30, 2018
deployment Updated travis deployment script (#14) Oct 26, 2018
docs
tests Setup Travis build (#4) Oct 25, 2018
.funcignore Moved deploy scripts out to shell script (#13) Oct 26, 2018
.gitignore Setup Travis build (#4) Oct 25, 2018
.travis.yml Updated travis deployment script (#14) Oct 26, 2018
LICENSE init commit Oct 12, 2018
Makefile Setup Travis build (#4) Oct 25, 2018
README.md Updated readme with deployment info (#15) Oct 26, 2018
extensions.csproj init commit Oct 12, 2018
host.json init commit Oct 12, 2018
requirements-dev.txt Deploy to Functions app on commit to master Oct 26, 2018
requirements.txt Deploy to Functions app on commit to master Oct 26, 2018
settings.py init commit Oct 12, 2018

README.md

Cloud Scanner - Azure Functions Sample App

Build Status

This is an example Azure Function App that demonstrates the use of the cloud-scanner library and its providers. cloud-scanner is a Python package that pulls cloud resources from different providers (Azure, AWS, GCP) and puts the metadata into data stores.

Note: This library is NOT affiliated with the Azure team at Microsoft and was developed by the Commercial Software Engineering team as a tool for the Open Source community to use and contribute to as they see fit. Use at your own risk!

Related Projects

The following are a list of related projects that are dependencies for the Azure Functions sample:

  1. cloud-scanner

    Core library components for cloud-scanner project

  2. cloud-scanner-azure

    Azure components for cloud-scanner project. Includes Azure support for discovering Azure resources in addition to storage and workflow components for hosting on Azure

  3. cloud-scanner-generic

    Generic components for cloud-scanner project including MySql, ElasticSearch and more

Running Locally

  1. Create Python 3.6 virtualenv env with all dependencies installed
    python3.6 -m virtualenv env
    source env/bin/activate
    (env) pip install -r requirements.txt
    (env) pip install -r requirements-dev.txt
    
    If running on Windows CMD Prompt/Powershell:
    python3.6 -m virtualenv env
    .env\Scripts\activate
    (env) pip install -r requirements.txt
    (env) pip install -r requirements-dev.txt
    
  2. Create an Azure Service Principal
  3. Create .env file in root directory and populate with appropriate data:
    # Azure authentication
    AZURE_CLIENT_ID=<service-principal-app-id>
    AZURE_CLIENT_SECRET=<service-principal-secret>
    AZURE_STORAGE_ACCOUNT=<azure-storage-account-name>
    AZURE_STORAGE_KEY=<azure-storage-account-key
    AZURE_TENANT_ID=<service-principal-tenant-id>
    AzureWebJobsStorage=<azure-storage-connection-string>
    
    # Container & Queue Names
    CONFIG_CONTAINER=config-files
    PAYLOAD_QUEUE_NAME=resource-payloads
    TAG_UPDATES_QUEUE_NAME=resource-tag-updates
    TASK_QUEUE_NAME=resource-jobs
    
    # Resource types
    RESOURCE_STORAGE_TYPE=rest_storage_service
    STORAGE_CONTAINER_TYPE=azure_storage
    QUEUE_TYPE=azure_storage_queue
    
    # Needed for 'rest_storage_service' storage type
    REST_STORAGE_URL=<api-url-for-posting-resources>
    
    # App Insights (optional)
    APPINSIGHTS_APPID=<app-insights-app-id>
    APPINSIGHTS_INSTRUMENTATIONKEY=<app-insights-instrumentation-key>
    
  4. Create local.settings.json file in root directory and populate:
     {
         "IsEncrypted": false,
         "Values": {
             "AzureWebJobsStorage": "<azure-storage-connection-string",
             "FUNCTIONS_WORKER_RUNTIME": "python"
         }
     }
  5. Generate cloud config file
    (env) generate-config -p <cloud-provider1,cloud-provider2> -t <resource-type1,resource-type2>
    Example: Generate config to pull all Azure resources
    (env) generate-config -p azure -t "*"
  6. Install Azure Core Tools
  7. Register binding types
    (env) func extensions install
  8. Deploy necessary resources
  9. Run Azure Functions locally
    (env) func host start

Publish Function App to Azure

Azure Resources

The Azure Functions Sample runs on a Linux consumption plan with python support. The ARM template is complete and deploys the following resources:

  • Functions App
  • Linux Consumption App Service Plan
  • Storage Account
  • Application Insights

Automatically deploy using the deployments/deploy.ps1 or the following Azure CLI commands.

$resourceGroupName = <resource-group-name>
az group create -l westus -n $resourceGroupName

az group deployment create 
    --resource-group $resourceGroupName 
    --name cloud-scanner 
    --template-file deploy.azure.json 
    --parameters parameters.json 
    --parameters 
        prefix=$prefix 
        location_abbr=$locationAbbr 
        environment=$environment 
        application_name=$applicationName

Deploying Function App Code

This repo is set up to use Travis CI to deploy code from the master branch. In order for this to work properly, you will need to use the service principal created previously (or create a new one) and add the following environment variables to your Travis settings with the appropriate values:

AZ_SP_ID=<service-principal-app-id>
AZ_SP_SECRET=<service-principal-secret>
AZ_TENANT_ID=<service-principal-tenant-id>
GITHUB_TOKEN=<github-token>
AZ_FUNCTIONS_APP_NAME=<name-of-functions-app>

Login to your Azure account with standard username or password or with the service principal you created

az login --service-principal --username "$AZ_APP_ID" --password "$AZ_SP_SECRET" --tenant "$AZ_TENANT_ID"
az account get-access-token --query "accessToken"

To deploy from your machine, create and activate your virtual environment and run:

func azure functionapp publish $AZ_FUNCTIONS_APP_NAME --build-native-deps --force

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.