Compliance Rules failing -

[jonshern]

Micro services

• Spring boot Applications:
• Compliance Service

Rule Engine

Rules

ETL

Webapp

• OS Type: Windows/Linux/MacOS
• 1.8
• 1.7

Summary

I noticed that in this environment i was 100% compliant.
Which surprised me, especially since it is sandbox.
So i tried creating a new Elastic IP and not using it to validate a failed rule state.
PacMan_UnusedElasticIpRule_version-1_UnusedElasticIpRule_elasticip-job

Reproduce steps

Added a new EIP and did not use it.

Expected Results

A Failed test based on the unused rule

Actual Results

All of the tests are passing.
I am bit surprised that things seem to be failing and i get a dashboard saying everything is passing.
The Pacman Rule Engine Job Stats are 1000+ Failed and 105 Succeeded.

Would seem like there is something broken with the deployment, but not sure what specifically i would need to fix.

Here are the logs
23:08:22,926 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@27ddd392 - Registering current configuration as safe fallback point
23:08:22,926 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@27ddd392 - Registering current configuration as safe fallback point
23:08:22
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
23:08:23
2018-10-29 23:08:23 [main] DEBUG c.t.pacman.executor.RuleExecutor - rule Param String {"autofix": false, "ruleType": "ManageRule", "alexaKeyword": "UnusedElasticIpRule", "ruleId": "PacMan_UnusedElasticIpRule_version-1_UnusedElasticIpRule_elasticip", "ruleRestUrl": "", "environmentVariables": [], "targetType": "elasticip", "ruleUUID": "09159bf1-a452-4746-bccf-6f9b162824ab", "params": [{"encrypt":
23:08:23
2018-10-29 23:08:23 [main] DEBUG c.t.pacman.executor.RuleExecutor - target Type :elasticip
23:08:23
2018-10-29 23:08:23 [main] DEBUG c.t.pacman.executor.RuleExecutor - rule Key : check-for-unused-elastic-ip
23:08:23
2018-10-29 23:08:23 2b571590-2e58-4ee3-a690-8f3e382c7283 [main] DEBUG c.t.pacman.executor.RuleExecutor - uncaught exception handler engaged.
23:08:23
2018-10-29 23:08:23 2b571590-2e58-4ee3-a690-8f3e382c7283 [main] DEBUG c.t.pacman.executor.RuleExecutor - shutdown hook engaged.
23:08:24
2018-10-29 23:08:24 2b571590-2e58-4ee3-a690-8f3e382c7283 [main] DEBUG com.tmobile.pacman.util.ESUtils - querying ES for target type:elasticip
23:08:24
2018-10-29 23:08:24 2b571590-2e58-4ee3-a690-8f3e382c7283 [main] ERROR com.tmobile.pacman.util.CommonUtils - {"query":{"bool":{"must":[{"term":{"latest":"true"}}]}}}
23:08:24
2018-10-29 23:08:24 2b571590-2e58-4ee3-a690-8f3e382c7283 [main] ERROR com.tmobile.pacman.util.CommonUtils - error closing issueunable to execute post request because Not Found
23:08:24
2018-10-29 23:08:24 2b571590-2e58-4ee3-a690-8f3e382c7283 [main] ERROR com.tmobile.pacman.util.ESUtils - error getting total documents
23:08:24
java.lang.Exception: unable to execute post request because Not Found
23:08:24
at com.tmobile.pacman.util.CommonUtils.doHttpPost(CommonUtils.java:201)
23:08:24
at com.tmobile.pacman.util.ESUtils.getTotalDocumentCountForIndexAndType(ESUtils.java:143)
23:08:24
at com.tmobile.pacman.util.ESUtils.getResourcesFromEs(ESUtils.java:90)
23:08:24
at com.tmobile.pacman.executor.RuleExecutor.run(RuleExecutor.java:182)
23:08:24
at com.tmobile.pacman.executor.RuleExecutor.main(RuleExecutor.java:91)
23:08:24
2018-10-29 23:08:24 2b571590-2e58-4ee3-a690-8f3e382c7283 [main] DEBUG com.tmobile.pacman.util.ESUtils - total resource count-1
23:08:24
2018-10-29 23:08:24 2b571590-2e58-4ee3-a690-8f3e382c7283 [main] DEBUG com.tmobile.pacman.util.ESUtils - inventory query{"size":10000,"query":{"bool":{"must":[{"term":{"latest":"true"}}]}}}
23:08:24
2018-10-29 23:08:24 2b571590-2e58-4ee3-a690-8f3e382c7283 [main] ERROR com.tmobile.pacman.util.CommonUtils - {"size":10000,"query":{"bool":{"must":[{"term":{"latest":"true"}}]}}}
23:08:24
2018-10-29 23:08:24 2b571590-2e58-4ee3-a690-8f3e382c7283 [main] ERROR com.tmobile.pacman.util.CommonUtils - error closing issueunable to execute post request because Not Found
23:08:24
2018-10-29 23:08:24 2b571590-2e58-4ee3-a690-8f3e382c7283 [main] ERROR com.tmobile.pacman.util.ESUtils - error retrieving inventory from ES
23:08:24
java.lang.Exception: unable to execute post request because Not Found
23:08:24
at com.tmobile.pacman.util.CommonUtils.doHttpPost(CommonUtils.java:201)
23:08:24
at com.tmobile.pacman.util.ESUtils.getDataFromES(ESUtils.java:373)
23:08:24
at com.tmobile.pacman.util.ESUtils.getResourcesFromEs(ESUtils.java:92)
23:08:24
at com.tmobile.pacman.executor.RuleExecutor.run(RuleExecutor.java:182)
23:08:24
at com.tmobile.pacman.executor.RuleExecutor.main(RuleExecutor.java:91)
23:08:24
2018-10-29 23:08:24 2b571590-2e58-4ee3-a690-8f3e382c7283 [main] ERROR c.t.pacman.executor.RuleExecutor - unable to get inventory for aws-all--elasticip
23:08:24
java.lang.Exception: unable to execute post request because Not Found
23:08:24
at com.tmobile.pacman.util.CommonUtils.doHttpPost(CommonUtils.java:201)
23:08:24
at com.tmobile.pacman.util.ESUtils.getDataFromES(ESUtils.java:373)
23:08:24
at com.tmobile.pacman.util.ESUtils.getResourcesFromEs(ESUtils.java:92)
23:08:24
at com.tmobile.pacman.executor.RuleExecutor.run(RuleExecutor.java:182)

[santhoshigorle]

Hello John,

From the above logs I have noticed Rule Engine is not able to fetch the inventory "com.tmobile.pacman.util.ESUtils - error getting total documents
23:08:24
java.lang.Exception: unable to execute post request because Not Found"

Can you please check whether environment variable in Cloud watch rules ES_URI is correctly set(http://YourEShost:YourESPort)

Generally this error comes when the ES_URI value is invalid.

As Rule Engine was not able to fetch the data , Rule is not able to evaluate your assets and that's reason you are seeing 100% compliant in the Dashboard.

[mikael-lindstrom]

I have the same problem and for me it seems to be caused by a missing index in ElasticSearch. The query that fails for me is: http://{ES_Host}:80/aws-all/elasticip/_count which returns a 404.

> curl http://$ES_HOST:80/aws-all/elasticip/_count?pretty=true
{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index",
        "resource.type" : "index_or_alias",
        "resource.id" : "aws-all",
        "index_uuid" : "_na_",
        "index" : "aws-all"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index",
    "resource.type" : "index_or_alias",
    "resource.id" : "aws-all",
    "index_uuid" : "_na_",
    "index" : "aws-all"
  },
  "status" : 404
}

If I instead create a new Asset Group through the UI the index is there and rules are evaluated after changing them to target the new group.

> curl http://$ES_HOST:80/test/elasticip/_count?pretty=true
{
  "count" : 5,
  "_shards" : {
    "total" : 150,
    "successful" : 150,
    "failed" : 0
  }
}

I suspect something is missing in the installer to correctly setup the aws-all Asset Group but I'm unsure what.

Also the specific rule UnusedElasticIpRule is not working correctly for me since it marks all ElasticIPs as unused under Policy Compliance but if I check the resource directly its in 100% compliant. But several other rules are working correctly now such as EC2WithPublicIPAccess.

[anilcs81]

@kaykumar @santhoshigorle , it would possibly due to the missing index related to sticky exception. Could you please update with the index creation script?

[johnakash]

@jonshern @mikael-lindstrom ,

Could you please check the availability of an index "exceptions" in elasticsearch? If it is there please create it with the below given mapping. It may will your issue.

PUT exceptions
{
    "mappings": {
      "sticky_exceptions": {
        "properties": {
          "assetGroup": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "dataSource": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "exceptionName": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "exceptionReason": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "expiryDate": {
            "type": "date"
          },
          "targetTypes": {
            "properties": {
              "name": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "rules": {
                "properties": {
                  "ruleId": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "ruleName": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
}

You can save it as exceptions.json and posted it with this: curl -X PUT vpc-pacman-esdomain-.us-east-2.es.amazonaws.com/exceptions -d @exceptions.json

[anilcs81]

@jonshern , is this fixed the issue? if so , we could close this one.

[mikael-lindstrom]

It seems to solve the issue for me (have not tried it on a fresh installation). I think either the documentation or the installer script should be updated with this before we close this issue since I suspect everyone will run into this problem.

[anilcs81]

This is to be fixed in the coming release as we introduced an initializer to take care of these missing indexes etc. I will add this to the faq for now

https://github.com/tmobile/pacbot/wiki/FAQS#i-see-the-compliance-as-100-for-all-rules-which-probably-wont-be-true-is-there-anything-missing

[santhoshigorle]

Issue is fixed as part of 1.1 release.