loss of conectivity after neutron deployment #176

Closed
bunnis opened this Issue Apr 20, 2015 · 21 comments

Comments

Projects
None yet
4 participants

bunnis commented Apr 20, 2015

Hi
Im trying to setup an openstack envrionement using puppet. I tried both allinone and multinode deployments. The debug files point to loss of conectivity on neutron deployment, specifically upon the creation of brex bridge. If i delete this bridge, i restore connectivity. I only lose connectivity on the external network (which has the internet, and therefore, the install process of horizon and heat do not go through).
This is my allinonde.yaml
http://paste.openstack.org/show/204899/

this is the route -n of a failed deployment machine(i manually altered the default gateway prior to deployment)
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 p3p1
10.155.216.0 0.0.0.0 255.255.252.0 U 0 0 0 em1
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 p3p1
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0

this is from other machine (unaltered)
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.155.216.1 0.0.0.0 UG 0 0 0 eth0
10.155.216.0 0.0.0.0 255.255.252.0 U 0 0 0 eth0
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0

EDIT-i forgot to mention, my puppet is master is communicating on 10.155.216.0

Contributor

cmurphy commented Apr 27, 2015

Can you clarify what is losing connectivity? Is it a nova instance, or the network node itself? What operating system are you running on?

bunnis commented Apr 27, 2015

Im running ubuntu 14.04. The network node itself will lose the connection
on the public interface which is connected to the internet and will fail
the deployment. Also It can not contact puppet if puppet is on the public
side. Puppet has to be on the management side.
BTW, I've been reinstalling nearly everyday and since friday you guys broke
the openstack module. Installing it on a fresh install (and i tried 4
times) will break the console classification module every single time after
puppet installs the openstack module.

THanks

On Mon, Apr 27, 2015 at 11:08 AM, Colleen Murphy notifications@github.com
wrote:

Can you clarify what is losing connectivity? Is it a nova instance, or the
network node itself? What operating system are you running on?


Reply to this email directly or view it on GitHub
#176 (comment)
.

Contributor

cmurphy commented Apr 27, 2015

I'm sorry you're experiencing this issue, I'm sure it's very frustrating.

I've been reinstalling nearly everyday and since friday you guys broke the openstack module.

We haven't made a forge release in a while so I assume you must be installing the openstack module from github. Could you help us debug this issue by trying to deploy from earlier commits, so that we can try to pinpoint the change that broke the module?

Installing it on a fresh install (and i tried 4 times) will break the console classification module every single time after puppet installs the openstack module.

What symptoms are you experiencing with the console classification module? This is on Puppet Enterprise, correct?

bunnis commented Apr 28, 2015

Im using the default command to install the openstack module. >sudo puppet
module install puppetlabs-openstack. Isnt this the correct one?

Yes, its on puppet enterprise. I can commit nodes, install ntp (as per
example on your documentation). And before friday I was able to install the
openstack module and deploy it on the nodes. After friday, as soon as I
installed the module, the classification tab presented me "an error occured
check the logs". As i said before, this is always on fresh installs,
installed the same way, without any OS updates, with the same network
configuration and everytihing.

attached image -> http://imgur.com/HQexB6i
the logs give something about java:
root@dopey:/var/log/pe-console-services# cat console-services.log
2015-04-27 20:05:37,370 INFO [p.c.class-updater] Requesting environment
list from "https://dopey:8140/v2.0/environments"
2015-04-27 20:05:37,588 INFO [p.c.class-updater] 200 response received for
request for environments from "https://dopey:8140/v2.0/environments"
2015-04-27 20:05:37,588 INFO [p.c.class-updater] Requesting classes in
production from "https://dopey.fog.poc:8140/production/resource_types/"
2015-04-27 20:05:39,820 INFO [p.c.class-updater] 200 response received for
request for classes in production from "
https://dopey:8140/production/resource_types/
".
2015-04-27 20:05:39,888 INFO [p.c.class-updater] Synchronized 104 classes
from the Puppet Master in 2 seconds
2015-04-27 20:08:24,835 WARN [p.r.h.middleware] GET
/node_groups/classifier-api/v1/environments java.net.ConnectException:
Connection refused
(Unknown Source)
sun.nio.ch.SocketChannelImpl.checkConnect
SocketChannelImpl.java:739
sun.nio.ch.SocketChannelImpl.finishConnect
DefaultConnectingIOReactor.java:173
org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent
DefaultConnectingIOReactor.java:147
org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents
AbstractMultiworkerIOReactor.java:348
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute
PoolingNHttpClientConnectionManager.java:189
org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute
CloseableHttpAsyncClientBase.java:67
org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.doExecute
CloseableHttpAsyncClientBase.java:38
org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.access$000
CloseableHttpAsyncClientBase.java:57
org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run
Thread.java:745 java.lang.Thread.run

2015-04-27 20:08:24,844 WARN [p.r.h.middleware] GET
/node_groups/classifier-api/v1/groups java.net.ConnectException: Connection
refused
(Unknown Source)
sun.nio.ch.SocketChannelImpl.checkConnect
SocketChannelImpl.java:739
sun.nio.ch.SocketChannelImpl.finishConnect
DefaultConnectingIOReactor.java:173
org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent
DefaultConnectingIOReactor.java:147
org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents
AbstractMultiworkerIOReactor.java:348
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute
PoolingNHttpClientConnectionManager.java:189
org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute
CloseableHttpAsyncClientBase.java:67
org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.doExecute
CloseableHttpAsyncClientBase.java:38
org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.access$000
CloseableHttpAsyncClientBase.java:57
org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run
Thread.java:745 java.lang.Thread.run

2015-04-27 20:08:25,318 WARN [p.r.h.middleware] GET
/node_groups/classifier-api/v1/classes java.net.ConnectException:
Connection refused
(Unknown Source)
sun.nio.ch.SocketChannelImpl.checkConnect
SocketChannelImpl.java:739
sun.nio.ch.SocketChannelImpl.finishConnect
DefaultConnectingIOReactor.java:173
org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent
DefaultConnectingIOReactor.java:147
org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents
AbstractMultiworkerIOReactor.java:348
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute
PoolingNHttpClientConnectionManager.java:189
org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute
CloseableHttpAsyncClientBase.java:67
org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.doExecute
CloseableHttpAsyncClientBase.java:38
org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.access$000
CloseableHttpAsyncClientBase.java:57
org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run
Thread.java:745 java.lang.Thread.run

On Mon, Apr 27, 2015 at 3:48 PM, Colleen Murphy notifications@github.com
wrote:

I'm sorry you're experiencing this issue, I'm sure it's very frustrating.

I've been reinstalling nearly everyday and since friday you guys broke the
openstack module.

We haven't made a forge release in a while so I assume you must be
installing the openstack module from github. Could you help us debug this
issue by trying to deploy from earlier commits, so that we can try to
pinpoint the change that broke the module?

Installing it on a fresh install (and i tried 4 times) will break the
console classification module every single time after puppet installs the
openstack module.

What symptoms are you experiencing with the console classification module?
This is on Puppet Enterprise, correct?


Reply to this email directly or view it on GitHub
#176 (comment)
.

Contributor

cmurphy commented Apr 28, 2015

We have not made a release to this module for a while, so there is no way that this could have suddenly caused your environment to break on Friday. Simply installing the module with puppet module install puppetlabs-openstack does not make any configuration changes, so it cannot have caused your network outage. Please open a support ticket for your Puppet Enterprise installation.

@cmurphy cmurphy closed this Apr 28, 2015

bunnis commented Apr 28, 2015

the network outage happened even before friday. I just casually mentioned
friday. And yes, you guys broke it. Because i did not changed anything on
my environment and as i stated previously, everytime I installed the puppet
enterprise and its nodes, the were always on fresh AKA FORMATTED installs.

On Tue, Apr 28, 2015 at 9:18 AM, Colleen Murphy notifications@github.com
wrote:

Closed #176
#176.


Reply to this email directly or view it on GitHub
#176 (comment)
.

Contributor

cmurphy commented Apr 28, 2015

Are you applying the allinone role or the network role to the puppetmaster itself?

@cmurphy cmurphy reopened this Apr 28, 2015

bunnis commented Apr 28, 2015

I tried both. Same problem

On Tue, Apr 28, 2015 at 10:04 AM, Colleen Murphy notifications@github.com
wrote:

Are you applying the allinone role or the network role to the puppetmaster
itself?


Reply to this email directly or view it on GitHub
#176 (comment)
.

Contributor

cmurphy commented Apr 28, 2015

You should not apply any openstack configuration to the puppetmaster. You should only make sure ntp is running on the puppetmaster. Please try again using different nodes for your openstack deployment.

bunnis commented Apr 28, 2015

Sorry, my bad, i didn not read your comment properly. I have 6machines, one
is puppet master and Im NOT deploying anything there.

On Tue, Apr 28, 2015 at 10:30 AM, Colleen Murphy notifications@github.com
wrote:

You should not apply any openstack configuration to the puppetmaster. You
should only make sure ntp is running on the puppetmaster. Please try again
using different nodes for your openstack deployment.


Reply to this email directly or view it on GitHub
#176 (comment)
.

Contributor

cmurphy commented Apr 28, 2015

Okay. With regard to the issue with your puppet console, I've spoken to one of our support specialists, who recommends following these instructions to tune your puppetmaster by reducing the number of JRuby processes and resetting the memory allocated to the console. Our PE engineers recommend checking /etc/puppetlabs/console-services/conf.d/console.conf and /etc/puppetlabs/console-services/conf.d/webserver.conf on the puppet master for anything that has changed. Specifically, they think that setting an incorrect hostname for classifier-server in the console.conf file or an incorrect hostname for the api server in webserver.conf may cause the error you're seeing. If you are unable to resolve the issue with the puppet console and you have a standard support license, I recommend opening a support ticket. The support team and I are confident that installing a module but applying no additional configuration from the module to the puppet master cannot cause this issue, so it appears to be a coincidence.

With regard to the networking issue on the network node, we can still try to debug that here. Can you provide the following information:

  • You said "i manually altered the default gateway prior to deployment", can you verify that you are able to reach the external network after manually altering the gateway but before applying any puppet configuration?
  • the output of the following commands on the network node both before and after applying the puppet configuration:
    • iptables -S
    • ifconfig
    • ovs-vsctl
  • the output of puppet agent -t for applying configuration to the unconfigured network node (make sure the controller is configured first)
  • the output of puppet module list from the puppet master
  • Is there any additional configuration being applied to the network node?

bunnis commented Apr 28, 2015

Hi
I tried to alter the hostname to specific IP address and the problem still
persists. Im going to make a format and a full install again just for you.
Do you believe a BIND server might be interfering with the puppet master?
Might be the only thing i added along side it last week.

Regarding those outputs, I will give them to you if I'm able to deploy
again.

BTW nothing correlated with my problems, but I thought you would like to
know that everything you perform a reboot on the master puppet, the web
interface refuses connections and I have to issue the following command:
sudo service pe-puppetserver restart; sudo service pe-console-services
restart

On Tue, Apr 28, 2015 at 3:25 PM, Colleen Murphy notifications@github.com
wrote:

Okay. With regard to the issue with your puppet console, I've spoken to
one of our support specialists, who recommends following these
instructions http://docs.puppetlabs.com/pe/3.7/config_puppetserver.html
to tune your puppetmaster by reducing the number of JRuby processes and
resetting the memory allocated to the console. Our PE engineers recommend
checking /etc/puppetlabs/console-services/conf.d/console.conf and
/etc/puppetlabs/console-services/conf.d/webserver.conf on the puppet
master for anything that has changed. Specifically, they think that setting
an incorrect hostname for classifier-server in the console.conf file or an
incorrect hostname for the api server in webserver.conf may cause the error
you're seeing. If you are unable to resolve the issue with the puppet
console and you have a standard support license, I recommend opening a
support ticket. The support team and I are confident that installing a
module but applying no addit ional configuration from the module to the
puppet master cannot cause this issue, so it appears to be a coincidence.

With regard to the networking issue on the network node, we can still try
to debug that here. Can you provide the following information:

You said "i manually altered the default gateway prior to deployment",
can you verify that you are able to reach the external network after
manually altering the gateway but before applying any puppet configuration?

the output of the following commands on the network node both before
and after applying the puppet configuration:
- iptables -S
- ifconfig
- ovs-vsctl
-

the output of puppet agent -t for applying configuration to the
unconfigured network node (make sure the controller is configured first)

the output of puppet module list from the puppet master

Is there any additional configuration being applied to the network
node?


Reply to this email directly or view it on GitHub
#176 (comment)
.

Contributor

cmurphy commented Apr 28, 2015

A bad DNS configuration in bind could cause issues with your puppetmaster. I think opening a support ticket would be the most effective way to resolve the issue with your puppetmaster.

bunnis commented Apr 28, 2015

DNS is working good. Still i manually bypassed the DNS with /etc/hosts file
and still gave me problems

On Tue, Apr 28, 2015 at 4:19 PM, Colleen Murphy notifications@github.com
wrote:

A bad DNS configuration in bind could cause issues with your puppetmaster.
I think opening a support ticket would be the most effective way to resolve
the issue with your puppetmaster.


Reply to this email directly or view it on GitHub
#176 (comment)
.

bunnis commented Apr 29, 2015

New install, new server, still same error.

On Tue, Apr 28, 2015 at 4:21 PM, Pedro Coelho Silv
wrote:

DNS is working good. Still i manually bypassed the DNS with /etc/hosts
file and still gave me problems

On Tue, Apr 28, 2015 at 4:19 PM, Colleen Murphy notifications@github.com
wrote:

A bad DNS configuration in bind could cause issues with your
puppetmaster. I think opening a support ticket would be the most effective
way to resolve the issue with your puppetmaster.


Reply to this email directly or view it on GitHub
#176 (comment)
.

Oyabi commented Jun 3, 2015

Hi,
I have also a lost connection with my configuration when I try to install a network node. Like mentionned above ovs-vsctl del-br brex resolve the connection lost problem. Clear iptables doesn't change anything.
I can't ping anything external or internal. My node IP : 10.0.249.149
You can see my error log of puppet agent -t here :

Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install neutron-fwaas' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package neutron-fwaas
Error: /Stage[main]/Neutron::Services::Fwaas/Package[neutron-fwaas]/ensure: change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install neutron-fwaas' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package neutron-fwaas
Warning: /Stage[main]/Neutron::Services::Fwaas/Neutron_fwaas_service_config[fwaas/driver]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Services::Fwaas/Neutron_fwaas_service_config[fwaas/enabled]: Skipping because of failed dependencies
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install neutron-plugin-openvswitch-agent' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  neutron-plugin-openvswitch-agent
0 upgraded, 1 newly installed, 0 to remove and 84 not upgraded.
Need to get 3872 B of archives.
After this operation, 76.8 kB of additional disk space will be used.
Err http://ubuntu-cloud.archive.canonical.com/ubuntu/ trusty-updates/juno/main neutron-plugin-openvswitch-agent all 1:2014.2.3-0ubuntu1~cloud0
  Could not resolve 'ubuntu-cloud.archive.canonical.com'
E: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/n/neutron/neutron-plugin-openvswitch-agent_2014.2.3-0ubuntu1~cloud0_all.deb  Could not resolve 'ubuntu-cloud.archive.canonical.com'

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Error: /Stage[main]/Neutron::Agents::Ml2::Ovs/Package[neutron-ovs-agent]/ensure: change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install neutron-plugin-openvswitch-agent' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  neutron-plugin-openvswitch-agent
0 upgraded, 1 newly installed, 0 to remove and 84 not upgraded.
Need to get 3872 B of archives.
After this operation, 76.8 kB of additional disk space will be used.
Err http://ubuntu-cloud.archive.canonical.com/ubuntu/ trusty-updates/juno/main neutron-plugin-openvswitch-agent all 1:2014.2.3-0ubuntu1~cloud0
  Could not resolve 'ubuntu-cloud.archive.canonical.com'
E: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/n/neutron/neutron-plugin-openvswitch-agent_2014.2.3-0ubuntu1~cloud0_all.deb  Could not resolve 'ubuntu-cloud.archive.canonical.com'

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Warning: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[ovs/local_ip]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/enable_distributed_routing]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[ovs/tunnel_bridge]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[ovs/enable_tunneling]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[ovs/integration_bridge]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[securitygroup/firewall_driver]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/arp_responder]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/polling_interval]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/l2_population]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/tunnel_types]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Agents::Ml2::Ovs/Service[neutron-ovs-agent-service]: Skipping because of failed dependencies
Error: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]: Failed to call refresh: keystone-manage db_sync returned 1 instead of one of [0]
Error: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]: keystone-manage db_sync returned 1 instead of one of [0]
Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_service[neutron]: Could not evaluate: undefined method `collect' for nil:NilClass
Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]: Could not evaluate: undefined method `collect' for nil:NilClass
Warning: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_endpoint[openstack/neutron]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user_role[neutron@services]: Skipping because of failed dependencies
Warning: /Stage[main]/Neutron::Server/Service[neutron-server]: Skipping because of failed dependencies
Error: Could not send report: No route to host - connect(2)

Do you want a full trace ?
My config file :

---
classes:
# - openstack::role::controller
# - openstack::role::storage
 - openstack::role::network
# - openstack::role::compute
# - openstack::role::allinone
 - openstack


openstack::region: 'openstack'

######## Networks
openstack::network::api: '10.0.249.0/16'
openstack::network::external: '10.0.249.0/16'

openstack::network::management: '10.0.249.0/16'
openstack::network::data: '10.0.249.0/16'

######## Fixed IPs (controllers)

openstack::controller::address::api: '10.0.249.119'
openstack::controller::address::management: '10.0.249.119'
openstack::storage::address::api: '10.0.1.222'
openstack::storage::address::management: '10.0.1.222'

######## Database

openstack::mysql::root_password: 'spam-gak'
openstack::mysql::service_password: 'fuva-wax'
openstack::mysql::allowed_hosts: ['localhost', '127.0.0.1', '10.0.249.%', '10.0.1.%']

openstack::mysql::keystone::user: 'keystone'
openstack::mysql::keystone::pass: 'fuva-wax'

openstack::mysql::cinder::user: 'cinder'
openstack::mysql::cinder::pass: 'fuva-wax'

openstack::mysql::glance::user: 'glance'
openstack::mysql::glance::pass: 'fuva-wax'
openstack::glance::api_servers: ['10.0.1.222:9292']

openstack::mysql::nova::user: 'nova'
openstack::mysql::nova::pass: 'fuva-wax'

openstack::mysql::neutron::user: 'neutron'
openstack::mysql::neutron::pass: 'fuva-wax'

openstack::mysql::heat::user: 'heat'
openstack::mysql::heat::pass: 'fuva-wax'

######## RabbitMQ

openstack::rabbitmq::user: 'openstack'
openstack::rabbitmq::password: 'pose-vix'
openstack::rabbitmq::hosts: ['10.0.249.119:5672']

######## Keystone

openstack::keystone::admin_token: 'sosp-kyl'
openstack::keystone::admin_email: 'chris.hoge@puppetlabs.com'
openstack::keystone::admin_password: 'fyby-tet'

openstack::keystone::tenants:
    "test":
        description: "Test tenant"
    "test2":
        description: "Test tenant"

openstack::keystone::users:
    "test":
        password: "abc123"
        tenant: "test"
        email: "test@example.com"
        admin: true
    "demo":
        password: "abc123"
        tenant: "test"
        email: "demo@example.com"
        admin: false
    "demo2":
        password: "abc123"
        tenant: "test2"
        email: "demo2@example.com"
        admin: false

######## Glance

openstack::images:
  Cirros:
    container_format: 'bare'
    disk_format: 'qcow2'
    source: 'http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img'

openstack::glance::password: 'na-mu-va'

######## Cinder

openstack::cinder::password: 'zi-co-se'
openstack::cinder::volume_size: '8G'

######## Swift

openstack::swift::password: 'dexc-flo'
openstack::swift::hash_suffix: 'pop-bang'

######## Nova

openstack::nova::libvirt_type: 'kvm'
openstack::nova::password: 'quuk-paj'

######## Neutron

openstack::neutron::password: 'whi-rtuz'
openstack::neutron::shared_secret: 'by-sa-bo'
openstack::neutron::core_plugin: 'ml2'
openstack::neutron::service_plugins: ['router', 'firewall', 'lbaas', 'vpnaas', 'metering']

######## Ceilometer
openstack::ceilometer::address::management: '10.0.249.119'
openstack::ceilometer::mongo::username: 'mongo'
openstack::ceilometer::mongo::password: 'mongosecretkey123'
openstack::ceilometer::password: 'whi-truz'
openstack::ceilometer::meteringsecret: 'ceilometersecretkey'

######## Heat
openstack::heat::password: 'zap-bang'
openstack::heat::encryption_key: 'heatsecretkey123'

######## Horizon

openstack::horizon::secret_key: 'whu-ghuk'
openstack::horizon::allowed_hosts: ['localhost', '127.0.0.1', '*']


######## Tempest

openstack::tempest::configure_images    : true
openstack::tempest::image_name          : 'Cirros'
openstack::tempest::image_name_alt      : 'Cirros'
openstack::tempest::username            : 'demo'
openstack::tempest::username_alt        : 'demo2'
openstack::tempest::username_admin      : 'test'
openstack::tempest::configure_network   : true
openstack::tempest::public_network_name : 'public'
openstack::tempest::cinder_available    : true
openstack::tempest::glance_available    : true
openstack::tempest::horizon_available   : true
openstack::tempest::nova_available      : true
openstack::tempest::neutron_available   : true
openstack::tempest::heat_available      : false
openstack::tempest::swift_available     : false

######## Log levels
openstack::verbose: 'True'
openstack::debug: 'True'

List of module installed with r10k:

/etc/puppet/modules
├── duritong-sysctl (v0.0.8)  invalid
├── erlang (???)
├── memcached (???)
├── nanliu-staging (v1.0.2)
├── puppetlabs-apache (v1.2.0)
├── puppetlabs-apt (v1.4.1)  invalid
├── puppetlabs-concat (v1.1.2)
├── puppetlabs-firewall (v1.6.0)
├── puppetlabs-inifile (v1.1.4)
├── puppetlabs-mongodb (v0.10.0)
├── puppetlabs-mysql (v3.3.0)
├── puppetlabs-ntp (v3.0.4)
├── puppetlabs-openstack (v5.0.2)
├── puppetlabs-postgresql (v4.0.0)  invalid
├── puppetlabs-puppetdb (v4.0.0)
├── puppetlabs-rabbitmq (v5.2.1)  invalid
├── puppetlabs-stdlib (v4.3.2)
├── qpid (???)
├── rsync (???)
├── ssh (???)
├── stackforge-ceilometer (v5.0.0)
├── stackforge-cinder (v5.0.0)
├── stackforge-glance (v5.0.0)
├── stackforge-heat (v5.0.0)
├── stackforge-horizon (v5.0.0)
├── stackforge-keystone (v5.0.0)
├── stackforge-neutron (v5.0.0)
├── stackforge-nova (v5.0.0)
├── stackforge-openstack_extras (v5.0.0)
├── stackforge-openstacklib (v5.0.0)
├── stackforge-swift (v5.0.0)
├── stackforge-tempest (v5.0.0)
├── stackforge-vswitch (v1.0.0)
├── stahnma-epel (v1.0.2)
├── vcsrepo (???)
└── xinetd (???)

My neutron node run Ubuntu 14.04 and have only one interface (eth0).
Regards.

To fix the connectivity issue stop and uninstall firewalld and NetworkManager packages/servers during kickstart your server. Seems like OpenStack juno neutron module doesn't like with NetworkManage .

I put following in my kickstart file under %post.

systemctl mask firewalld
systemctl stop firewalld
systemctl mask NetworkManager.service
systemctl stop NetworkManager.service
yum remove -y firewalld
yum remove -y NetworkManager

Make sure iptables-services is installed . The minimum installation doesn't add this package.

Contributor

cmurphy commented Jul 15, 2015

@sakibsyl: thanks for your input. Since @bunnis and @Oyabi are both running Ubuntu, I don't think firewalld or NetworkManager are related to this particular issue, but your instructions might be able to help someone else.

@Oyabi this module is expected to run on a machine with at least two interfaces and up to five interfaces. The reason this is failing for you is because your default route goes through your only interface. When you add the brex bridge to it, network traffic will stop working. You can read about this here. I would recommend running this on a machine with at least two interfaces, one to be used as a management interface. If this is not possible, you can fix the issue manually with this guide.

@bunnis I believe your issue is similar to @Oyabi's but you haven't provided the information I requested earlier. If your puppet master is repaired, could you provide that information, especially the output of ifconfig and ovs-vsctl, as well as more information on your manual alterations to the default gateway?

bunnis commented Jul 15, 2015

Hi @cmurphy
Sorry I havent payed much attention to this as I have been busy with other projects. I stopped using your all-in-one solution as I couldnt control all the steps. Right now I deployed everything using stackforge modules. I'm still having some issues with neutron but all the other services are up and running. I believe both puppet-all in one and stackforge-neutron dont create/add ports to the bridges br-ext br-int br-tun and Im trying to configure that right now. Ill be checking the guides you posted. Some info from my network node : (Although I've manually added br-ex to my internet port i still cant ping)

root@fing:/home/fing# ifconfig
br-ex     Link encap:Ethernet  HWaddr 00:22:19:d6:82:cd
          inet addr:10.155.216.205  Bcast:10.155.219.255  Mask:255.255.252.0
          inet6 addr: fe80::442b:c4ff:fec9:71c4/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:985225 errors:0 dropped:6348 overruns:0 frame:0
          TX packets:684 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:164436815 (164.4 MB)  TX bytes:59348 (59.3 KB)

br-int    Link encap:Ethernet  HWaddr 02:f5:f9:f9:af:41
          inet6 addr: fe80::c69:49ff:feb8:978b/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:17 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1400 (1.4 KB)  TX bytes:648 (648.0 B)

br-tun    Link encap:Ethernet  HWaddr 5a:84:d7:76:8e:43
          inet6 addr: fe80::34f0:4ff:fea3:fd9/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)

eth0      Link encap:Ethernet  HWaddr 00:22:19:d6:82:cc
          inet addr:192.168.0.7  Bcast:192.168.1.255  Mask:255.255.254.0
          inet6 addr: fe80::222:19ff:fed6:82cc/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:236992 errors:0 dropped:0 overruns:0 frame:0
          TX packets:255333 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:97384574 (97.3 MB)  TX bytes:71563381 (71.5 MB)
          Interrupt:16

eth1      Link encap:Ethernet  HWaddr 00:22:19:d6:82:cd
          inet6 addr: fe80::222:19ff:fed6:82cd/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:1901449 errors:593 dropped:4 overruns:0 frame:5
          TX packets:1319 errors:0 dropped:0 overruns:0 carrier:0
          collisions:1 txqueuelen:1000
          RX bytes:586724803 (586.7 MB)  TX bytes:134972 (134.9 KB)
          Interrupt:17

eth2      Link encap:Ethernet  HWaddr 00:15:17:b9:6d:76
          inet addr:192.168.10.7  Bcast:192.168.10.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:16 Memory:dfc80000-dfca0000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:19 errors:0 dropped:0 overruns:0 frame:0
          TX packets:19 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2272 (2.2 KB)  TX bytes:2272 (2.2 KB)

root@fing:/home/fing# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    0      0        0 eth0
10.155.216.0    0.0.0.0         255.255.252.0   U     0      0        0 br-ex
192.168.0.0     0.0.0.0         255.255.254.0   U     0      0        0 eth0
192.168.10.0    0.0.0.0         255.255.255.0   U     0      0        0 eth2
root@fing:/home/fing#           options: {peer=patch-int}
options:: command not found
root@fing:/home/fing#     ovs_version: "2.0.2"
ovs_version:: command not found
root@fing:/home/fing# ovs-vsctl show
fce3de98-b573-48a8-a060-b89d02abe907
    Bridge br-tun
        fail_mode: secure
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    Bridge br-ex
        Port "eth1"
            Interface "eth1"
        Port "qg-3bf41e31-77"
            Interface "qg-3bf41e31-77"
                type: internal
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port br-ex
            Interface br-ex
                type: internal
    Bridge br-int
        fail_mode: secure
        Port "tapb27fc70a-a1"
            tag: 4095
            Interface "tapb27fc70a-a1"
                type: internal
        Port "qr-a1ffa3ae-e1"
            tag: 4095
            Interface "qr-a1ffa3ae-e1"
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    ovs_version: "2.0.2"
root@fing:/home/fing# cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
        address 192.168.0.7
        netmask 255.255.254.0
        network 192.168.0.0
        broadcast 192.168.1.255
        gateway 192.168.0.1
        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 192.168.0.1

#up route add -net 192.168.0.0 netmask 255.255.254.0 gw 192.168.0.1 eth0

auto eth1
iface eth1 inet manual
        up ifconfig $IFACE 0.0.0.0 up
        up ip link set $IFACE promisc on
        down ip link set $IFACE promisc off
        down ifconfig $IFACE down

auto br-ex
iface br-ex inet static
       address 10.155.216.205
       netmask 255.255.252.0
       #network 192.168.0.0
       #broadcast 192.168.1.255
       gateway 10.155.216.1
       # dns-* options are implemented by the resolvconf package, i$
       dns-nameservers 8.8.8.8
 #      up route add -net 10.155.216.0 netmask 255.255.252.0 gw 10.155.216.1 br-ex

auto eth2
iface eth2 inet static
address 192.168.10.7
netmask 255.255.255.0

Contributor

cmurphy commented Jul 15, 2015

@bunnis,

I stopped using your all-in-one solution as I couldnt control all the steps. Right now I deployed everything using stackforge modules.

I think that is actually a great approach, as this module is not flexible enough for many deployment scenarios. At this point, if you're still having trouble with neutron or OpenVSwitch I recommend getting help from the puppet-openstack team or seeking help with OpenStack itself rather than with the puppet modules.

May I close this issue for now? If you find this module needs to be modified we can reopen this issue or you can open a pull request.

bunnis commented Jul 15, 2015

Sure, close it

@bunnis bunnis closed this Jul 15, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment