failed jobs: cf-mysql-broker, quota-enforcer

[YunSangJun]

I did checkout developer version and have deployed.

While deploying, following error occured.

$ bosh deploy
 ...
  Started preparing deployment > Preparing deployment. Done (00:00:00)
  Started preparing package compilation > Finding packages to compile. Done (00:00:00)
  Started updating job cf-mysql-broker > cf-mysql-broker/0 (8772b80e-dcfe-4160-a4e6-f0e16f98a1a2) (canary). Failed: `cf-mysql-broker/0 (8772b80e-dcfe-4160-a4e6-f0e16f98a1a2)' is not running after update. Review logs for failed jobs: cf-mysql-broker, quota-enforcer (00:10:24)
Error 400007: `cf-mysql-broker/0 (8772b80e-dcfe-4160-a4e6-f0e16f98a1a2)' is not running after update. Review logs for failed jobs: cf-mysql-broker, quota-enforcer

I connected to cf-mysql-broker via ssh and reviewed logs cf-mysql-broker, quota-enforcer

$ vi /var/vcap/sys/log/cf-mysql-broker/db_migrate.combined.log
NoMethodError: undefined method `[]' for nil:NilClass
/var/vcap/data/packages/cf-mysql-broker/47cdbff74dadd949abd599bbd259fe7a53bbd68f.1-c79f348b93e1d2afe8343be5b8ee889dbfd02936/config/initializers/omniauth.rb:5:in `<top (required)>'
/var/vcap/packages/cf-mysql-broker/vendor/bundle/ruby/2.2.0/gems/activesupport-4.0.13/lib/active_support/dependencies.rb:223:in `load'
/var/vcap/packages/cf-mysql-broker/vendor/bundle/ruby/2.2.0/gems/activesupport-4.0.13/lib/active_support/dependencies.rb:223:in `block in load'
/var/vcap/packages/cf-mysql-broker/vendor/bundle/ruby/2.2.0/gems/activesupport-4.0.13/lib/active_support/dependencies.rb:214:in `load_dependency'
/var/vcap/packages/cf-mysql-broker/vendor/bundle/ruby/2.2.0/gems/activesupport-4.0.13/lib/active_support/dependencies.rb:223:in `load'
/var/vcap/packages/cf-mysql-broker/vendor/bundle/ruby/2.2.0/gems/railties-4.0.13/lib/rails/engine.rb:609:in `block (2 levels) in <class:Engine>'
/var/vcap/packages/cf-mysql-broker/vendor/bundle/ruby/2.2.0/gems/railties-4.0.13/lib/rails/engine.rb:608:in `each'
/var/vcap/packages/cf-mysql-broker/vendor/bundle/ruby/2.2.0/gems/railties-4.0.13/lib/rails/engine.rb:608:in `block in <class:Engine>'
/var/vcap/packages/cf-mysql-broker/vendor/bundle/ruby/2.2.0/gems/railties-4.0.13/lib/rails/initializable.rb:30:in `instance_exec'
/var/vcap/packages/cf-mysql-broker/vendor/bundle/ruby/2.2.0/gems/railties-4.0.13/lib/rails/initializable.rb:30:in `run'
/var/vcap/packages/cf-mysql-broker/vendor/bundle/ruby/2.2.0/gems/railties-4.0.13/lib/rails/initializable.rb:55:in `block in run_initializers'
/var/vcap/packages/cf-mysql-broker/vendor/bundle/ruby/2.2.0/gems/railties-4.0.13/lib/rails/initializable.rb:54:in `run_initializers'
/var/vcap/packages/cf-mysql-broker/vendor/bundle/ruby/2.2.0/gems/railties-4.0.13/lib/rails/application.rb:215:in `initialize!'
/var/vcap/packages/cf-mysql-broker/vendor/bundle/ruby/2.2.0/gems/railties-4.0.13/lib/rails/railtie/configurable.rb:30:in `method_missing'
/var/vcap/data/packages/cf-mysql-broker/47cdbff74dadd949abd599bbd259fe7a53bbd68f.1-c79f348b93e1d2afe8343be5b8ee889dbfd02936/config/environment.rb:5:in `<top (required)>'
/var/vcap/packages/cf-mysql-broker/vendor/bundle/ruby/2.2.0/gems/activesupport-4.0.13/lib/active_support/dependencies.rb:229:in `require'
/var/vcap/packages/cf-mysql-broker/vendor/bundle/ruby/2.2.0/gems/activesupport-4.0.13/lib/active_support/dependencies.rb:229:in `block in require'
/var/vcap/packages/cf-mysql-broker/vendor/bundle/ruby/2.2.0/gems/activesupport-4.0.13/lib/active_support/dependencies.rb:214:in `load_dependency'
/var/vcap/packages/cf-mysql-broker/vendor/bundle/ruby/2.2.0/gems/activesupport-4.0.13/lib/active_support/dependencies.rb:229:in `require'
/var/vcap/packages/cf-mysql-broker/vendor/bundle/ruby/2.2.0/gems/railties-4.0.13/lib/rails/application.rb:189:in `require_environment!'
/var/vcap/packages/cf-mysql-broker/vendor/bundle/ruby/2.2.0/gems/railties-4.0.13/lib/rails/application.rb:260:in `block in run_tasks_blocks'
Tasks: TOP => db:migrate => environment
(See full trace by running task with --trace)
Migrations failed at Tue Mar 22 08:21:13 UTC 2016
mysql_broker already exists
rake aborted!

$ vi /var/vcap/sys/log/quota-enforcer/quota_enforcer.stdout.log
{"timestamp":"1458635302.694883823","source":"Quota Enforcer","message":"Quota Enforcer.Enforcing Failed: Finding violators: Error executing 'quota violator'.All: Error 1146: Table 'mysql_broker.service_instances' doesn't exist","log_level":1,"data":{}}

What is problem? what does the logs mean?

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: https://www.pivotaltracker.com/story/show/116108131.

[mfine30]

Hey YunSangJan,

Glad to hear you were able to get it set up and mostly working Openstack. From the logs alone, I can tell you that the Quota Enforcer is failing because the broker is failing. Past that, it is difficult to do much debugging without your manifest. If you could please provide us a copy of your manifest with redacted secrets we can continue to help you.

Best,
Morgan
CF Core Services

[YunSangJun]

Following are my manifest file.

<cf-mysql.yml>
name: mysql-service 
director_uuid: 496d2aa3-****-4685-****-7287a484e327 #bosh director uuid 

releases: 
- name: cf-mysql      
  version: 26 #cf-mysql-<N>.yml

update:
  canaries: 1
  canary_watch_time: 30000-600000
  max_in_flight: 1
  update_watch_time: 30000-600000

compilation:
  cloud_properties:
     instance_type: m1.small
  network: mysql1     
  reuse_compilation_vms: true
  workers: 4

jobs:
- instances: 1
  name: mysql_z1 
  networks:
  - name: mysql1
    static_ips: 10.10.0.90              #mysql_z1 ip
  persistent_disk: 8196
  properties:
    admin_password: admin
    cluster_ips:
    - 10.10.0.90                        #mysql_z1 ip
    network_name: mysql1
    seeded_databases: null
    syslog_aggregator: null
    collation_server: utf8_unicode_ci
    character_set_server: utf8
  release: cf-mysql                     #cf-mysql-release/releases/cf-mysql-<N>.yml file name
  resource_pool: services-small
  template: mysql

- instances: 1
  name: proxy
  networks:
  - name: mysql1
    static_ips: 10.10.0.93        #proxy ip
  properties:
    cluster_ips:
    - 10.10.0.90                  #mysql_z1 ip
    external_host: dev2.***.io
    nats:
      machines:
      - 10.10.1.127               #cf nats ip
      password: admin
      port: 4222
      user: nats
    network_name: mysql1
    proxy:
      api_password: admin
      api_username: api
      api_force_https: false
    syslog_aggregator: null
  release: cf-mysql
  resource_pool: services-small
  template: proxy

- instances: 1
  name: cf-mysql-broker
  networks:
  - name: mysql1
    static_ips: 10.10.0.94     #mysql broker ip
  properties:
    jdbc_ip: 10.10.0.93        #proxy ip
    jdbc_pwd: admin
    jdbc_port: 3306
    log_dir: cf-mysql-broker  
    log_file: cf-mysql-broker
    log_level: INFO
    network_name: mysql1
  release: cf-mysql
  resource_pool: services-small
  template: cf-mysql-broker

- instances: 1
  lifecycle: errand
  name: broker-registrar
  networks:
  - name: mysql1
  properties:
    broker:
      host: 10.10.0.94                      #mysql broker ip
      name: mysql-service-broker
      password: cloudfoundry
      username: admin
      protocol: http
      port: 8080
    cf:
      admin_password: admin
      admin_username: admin
      api_url: https://api.dev2.***.io    #replace your domain
      skip_ssl_validation: true
  release: cf-mysql
  resource_pool: services-small
  template: broker-registrar

- instances: 1
  lifecycle: errand
  name: broker-deregistrar
  networks:
  - name: mysql1
  properties:
    broker:
      name: mysql-service-broker
    cf:
      admin_password: admin
      admin_username: admin
      api_url: https://api.dev2.***.io    #replace your domain
      skip_ssl_validation: true
  release: cf-mysql
  resource_pool: services-small
  template: broker-deregistrar

meta:
  apps_domain: dev2.***.io                #replace your domain
  environment: null
  external_domain: dev2.***.io            #replace your domain
  nats:
    machines:
    - 10.10.1.127    #cf nats ip
    password: admin
    port: 4222
    user: nats
  syslog_aggregator: null

networks:
- name: mysql1
  subnets:
  - cloud_properties:
      net_id: 9914f1b5-****-4b06-****-8d7d19a5b16e  #mysql net subnet id
      security_groups:
      - cf               #cf security group
    dns:
    - 8.8.8.8
    gateway: 10.10.0.1   #mysql net subnet g/w ip
    range: 10.10.0.0/24  #mysql net subnet CIDR
    reserved:
    - 10.10.0.2 - 10.10.0.9     #mysql net subnet ip
    static:
    - 10.10.0.10 - 10.10.0.100  #mysql net subnet static ip
  type: manual

properties: 
  nats:
    machines:
    - 10.10.1.127        #cf nats ip
    password: admin
    port: 4222
    user: nats
  cf:
    admin_password: admin
    admin_username: admin
    api_url: https://api.dev2.***.io   #replace your domain
    skip_ssl_validation: true
  cf_mysql:
    host: 10.10.0.90
    mysql:
      admin_username: admin
      admin_password: admin
      cluster_ips:
      - 10.10.0.90
      database_startup_timeout: 600
      roadmin_enabled: true
      roadmin_password: admin
      bootstrap_endpoint:
        username: admin 
        password: admin
      persistent_disk: 8196
    proxy:
      api_username: admin
      api_password: admin
      proxy_ips: 10.10.0.93 
    external_host: dev2.***.io
    broker:
      auth_username: admin
      auth_password: admin
      cookie_secret: 94046872-2602-4ca6-8d07-8b0da9762477
      dashboard_secret: p-mysql-secret
      plans:
      - name: 100mb
        id: ab08f1bc-e6fc-4b56-a767-ee0fea6e3f20
        description: Shared MySQL Server
        max_storage_mb: 10
        max_user_connections: 20
      - name: 1gb
        id: 11d0aa36-dcec-4021-85f5-ea4d9a5c8342
        description: Shared MySQL Server
        max_storage_mb: 20
        max_user_connections: 40
      services: {}

resource_pools:
- cloud_properties:
    instance_type: m1.small
  name: services-small
  network: mysql1
  stemcell:
    name: bosh-openstack-kvm-ubuntu-trusty-go_agent
    version: "3197"     #stemcell version

[aditya87]

Hi YungSanJun, you have not attached any service(s) to your broker yet, which is why your broker is failing to come up. Are you using our generate-deployment-manifest script to create your manifest? I'm assuming you created an iaas-settings.yml file for OpenStack and then called our script, right?

Your broker does not have anything in the services hash. Please look at (https://github.com/cloudfoundry/cf-mysql-release/blob/develop/manifest-generation/cf-mysql-template.yml#L101)

If you generated your manifest using our script, it will automatically fill in this hash.

[YunSangJun]

Thank you. it works after add services hash.

And there is no iaas-settings.yml file for Openstack currently.
#95

[sxd]

Hi!

Sorry for reopen this bug, but how can I attach a service to the broker? or the new versions doesn't have that? I haven't found anything on the documentation on this repository about add a service to the broker previous the the bosh deploy

[kamoljan]

+1

[menicosia]

@sxd @kamoljan it turns out that commenting on an issue doesn't automatically re-open it, so this issue has remained closed since last March, 2016.

@sxd if there's something I can do, I think I need more context. If there's more we can do, will you consider opening a new GH issue? More importantly, now is a great time to try cf-mysql-deployment if you're so inclined.

@kamoljan please let me know if there's something specific we can do for you as well, either through a new issue or via Pivotal Support, if you are a Pivotal customer.

Thanks, and sorry for being slow to respond!

--
Marco Nicosia
Product Manager
Pivotal Software, Inc.