[EAGLE-888] Application submitted to Storm is always shown as “HBaseAuditLogApp”

[DadanielZ]

Reordered the "withFallBack" function to get user's topology configuration

[haoch]

@wujinhu please help review the PR about app/server config changes

[wujinhu]

@DadanielZ Do you find this problem in your dev env?

[DadanielZ]

@wujinhu yes, I find this when I start eagle from Intellij on mac.

[wujinhu]

@DadanielZ
There are a lot of application.conf files in eagle. I think this is the root cause.
Besides, I think server config has higher priority than generated config(context)

[DadanielZ]

@wujinhu here is what I observed:

1. In site "sandbox", I have "alert engine" started. In storm UI, topology is running as "HBaseAuditMonitor"
   2.When run in Debug mode in Intellij, "server config" has "siteId = testSite", "appId = HBaseAuditLog", and config context has "siteId = sandbox" and "appId = ALERT_UNIT_TOPOLOGY_APP_SANDBOX".
   Because server config has higher priority, these context info are overridden.

By the way, could you explain a little bit on "There are a lot of application.conf files in eagle. I think this is the root cause"?

Thanks and regards,
Da

[wujinhu]

IMO, "server config" has "siteId = testSite", "appId = HBaseAuditLog" may be wrong. It should not contains these. I mean wrong application.conf has been loaded probably.

[DadanielZ]

@wujinhu I see, so this only happens in dev env. Since "Typesafe Config" is loading all "applicaiton.conf" in the classPath, parameters like "siteId", "appId",etc are preloaded in "server config".

Is there a good way to solve this issue? Or I have to update all "application.conf" files in the maven project manually?

[wujinhu]

@DadanielZ I think you can add application-debug.conf to eagle-server/src/test/resources/ if you run Class ServerDebug. Pls have a try. Thanks.

[DadanielZ]

@wujinhu thanks 👍 , this issue is solved.

But I come up with another one when tring to view the alerts from UI:

After configuring the "application-debug.conf" properly with "jdbc" as the storage type, I started eagle from Intellij successfully and started alert-engine and hdfsAuditLogApp smoothly, then I created a hdfs policy.

I saw alerts are generated and saved in my mysql database.

But when I tried to view the alerts from UI, ERROR occurs:

ERROR [2017-02-10 01:19:40,433] org.apache.eagle.service.generic.GenericEntityServiceResource: unknown storage type: jdbc, support: [hbase]
! org.apache.eagle.storage.exception.IllegalDataStorageTypeException: unknown storage type: jdbc, support: [hbase]

During debugging I see storageServiceProviders only contains hBaseStorage, but no jdbcStorage. Could you tell me the reason or give me a hint on which module to look into?

Thanks and regards,
Da

Here is the configuration file I used:

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# ---------------------------------------------
# Eagle REST Web Service Configuration
# ---------------------------------------------
service {
  env = "testing"
  host = "localhost"
  port = 9090
  username = "admin"
  password = "secret"
  readTimeOutSeconds = 60
  context = "/rest"
  timezone = "UTC"
}

zookeeper {
  zkQuorum = "localhost:2181"
  zkSessionTimeoutMs : 15000
  zkRetryTimes : 3
  zkRetryInterval : 20000
}

# ---------------------------------------------
# Eagle Deep Storage Configuration
# ---------------------------------------------

storage {
  type = "jdbc"
  jdbc {
    adapter = "mysql"
    username = "root"
    password = root
    database = eagle
    connectionUrl = "jdbc:mysql://localhost:3306/eagle"
    connectionProps = "encoding=UTF-8"
    driverClass = "com.mysql.jdbc.Driver"
    connectionMax = 8
  }
}

# ---------------------------------------------
# Eagle Metadata Store Configuration
# ---------------------------------------------


metadata {
  store = "org.apache.eagle.metadata.store.jdbc.JDBCMetadataStore"
  jdbc {
    username = root
    password = root
    driverClassName = com.mysql.jdbc.Driver
    database = "eagle"
    connection = "jdbc:mysql://localhost:3306/eagle?user=root"
  }
}
# ---------------------------------------------
# Eagle Application Configuration
# ---------------------------------------------
application {
  stream {
    provider = org.apache.eagle.app.messaging.KafkaStreamProvider
  }
  storm {
    nimbusHost = "localhost"
    nimbusThriftPort = 6627
  }
  updateStatus: {
    initialDelay: 10
    period: 10
  }
  healthCheck {
    initialDelay = 30
    period = 60
    publisher {
      publisherImpl = org.apache.eagle.app.service.impl.ApplicationHealthCheckEmailPublisher
      dailySendHour = 11
      mail.smtp.host = "mail.host.com"
      mail.smtp.port = 25
      mail.smtp.recipients = "someone@email.com"
      mail.smtp.subject = "Eagle Application Health Check"
      mail.smtp.template = "HealthCheckTemplate.vm"
    }
  }
  mailService {
    mailSmtpServer = "",
    mailSmtpPort = 25,
    mailSmtpAuth = "false"
    //mailSmtpConn = "plaintext",
    //mailSmtpUsername = ""
    //mailSmtpPassword = ""
    //mailSmtpDebug = false
  }
  dailyJobReport {
    reportHourTime: 1
    reportPeriodInHour: 12
    numTopUsers : 10
    jobOvertimeLimitInHour: 6
    subject: "Job Report For 12 hours"
    recipients: "nobody@abc.com"
    template: "JobReportTemplate.vm"
  }
}

# ---------------------------------------------
# Eagle Alert Engine Configuration
# ---------------------------------------------

# Coordinator Configuration
coordinator {
  policiesPerBolt = 5
  boltParallelism = 5
  policyDefaultParallelism = 5
  boltLoadUpbound = 0.8
  topologyLoadUpbound = 0.8
  numOfAlertBoltsPerTopology = 5
  zkConfig {
    zkQuorum = "localhost:2181"
    zkRoot = "/alert"
    zkSessionTimeoutMs = 10000
    connectionTimeoutMs = 10000
    zkRetryTimes = 3
    zkRetryInterval = 3000
  }
  metadataService {
    host = "localhost",
    port = 9090,
    context = "/rest"
  }
  metadataDynamicCheck {
    initDelayMillis = 1000
    delayMillis = 30000
    stateClearPeriodMin = 1440
    stateReservedCapacity = 100
  }
}

[qingwen220]

Hi @DadanielZ,

We have not added this module into Eagle server. So the service loader cannot detect it.

[DadanielZ]

@qingwen220 Got it.. When will we add this module and what is the target release date for Eagle 0.5?

[qingwen220]

@DadanielZ Could you help to fix it? I have a ticket here
https://issues.apache.org/jira/browse/EAGLE-902

[DadanielZ]

@qingwen220 sure, will do it.

[haoch]

@DadanielZ is this PR still valid?

[DadanielZ]

Issues resolved, hence closing it.