New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ephemeral port range #1

Closed
maddenpj opened this Issue May 1, 2013 · 4 comments

Comments

Projects
None yet
3 participants
@maddenpj

maddenpj commented May 1, 2013

Hi,
I am trying to upgrade our hadoop cluster to MRv2 (Yarn) and I am some issues in constraining the ephemeral ports. I noticed you tackled this problem. I opened the ports in iptables (For now I'm just sticking with the same 8036-8040 range) and I set the yarn.app.mapreduce.am.job.client.port-range property in my yarn-site but the resource manager is still trying to connect on ephemeral ports. I was wondering if this configuration works for you, maybe I'm missing something?

@maddenpj

This comment has been minimized.

Show comment
Hide comment
@maddenpj

maddenpj May 1, 2013

Actually I managed to set the port range as configured. The problem I'm having now is that as a map reduce job is running, more and more tasks fail until eventually the whole job fails. I wonder if its related to the supposed number of number of ports you need open?

maddenpj commented May 1, 2013

Actually I managed to set the port range as configured. The problem I'm having now is that as a map reduce job is running, more and more tasks fail until eventually the whole job fails. I wonder if its related to the supposed number of number of ports you need open?

@jkleint

This comment has been minimized.

Show comment
Hide comment
@jkleint

jkleint May 8, 2013

Owner

Sorry you're having trouble; what kind of errors do you see in the logs for the failed tasks? If you think it is related to the ephemeral ports, I would try a few things:

  1. Disable IPv6
  2. Verify the right ports are actually open on all nodes (including master) with sudo iptables -L -n
  3. Have a look at the logs to see what ports things are trying to listen on
  4. Check that the right things are actually listening on the right ports and the right IP (i.e., not localhost) with sudo netstat --inet -pln

Hope that helps!

Owner

jkleint commented May 8, 2013

Sorry you're having trouble; what kind of errors do you see in the logs for the failed tasks? If you think it is related to the ephemeral ports, I would try a few things:

  1. Disable IPv6
  2. Verify the right ports are actually open on all nodes (including master) with sudo iptables -L -n
  3. Have a look at the logs to see what ports things are trying to listen on
  4. Check that the right things are actually listening on the right ports and the right IP (i.e., not localhost) with sudo netstat --inet -pln

Hope that helps!

@sumit-gupta-sgt

This comment has been minimized.

Show comment
Hide comment
@sumit-gupta-sgt

sumit-gupta-sgt Nov 7, 2015

After disabling IPv6 I am getting some java.lang.reflect.InvocationTargetException.
IPv6 is disabled properly ..i am not able to see in ifconfig command.

Here is full stack trace:
log4j:WARN Failed to set property [conversionPattern] to value "%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n".
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.log4j.config.PropertySetter.setProperty(PropertySetter.java:245)
at org.apache.log4j.config.PropertySetter.setProperty(PropertySetter.java:204)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:169)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:805)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
at org.apache.log4j.LogManager.(LogManager.java:127)
at org.apache.log4j.Logger.getLogger(Logger.java:104)
at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:262)
at org.apache.commons.logging.impl.Log4JLogger.(Log4JLogger.java:108)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at org.apache.commons.logging.impl.LogFactoryImpl.createLogFromClass(LogFactoryImpl.java:1025)
at org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:844)
at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:541)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:292)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:269)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
at org.apache.hadoop.util.ShutdownHookManager.(ShutdownHookManager.java:44)
at org.apache.hadoop.util.RunJar.run(RunJar.java:200)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ExceptionInInitializerError
at java.nio.file.FileSystems.getDefault(FileSystems.java:176)
at sun.util.calendar.ZoneInfoFile$1.run(ZoneInfoFile.java:490)
at sun.util.calendar.ZoneInfoFile$1.run(ZoneInfoFile.java:481)
at java.security.AccessController.doPrivileged(Native Method)
at sun.util.calendar.ZoneInfoFile.(ZoneInfoFile.java:480)
at sun.util.calendar.ZoneInfo.getTimeZone(ZoneInfo.java:663)
at java.util.TimeZone.getTimeZone(TimeZone.java:560)
at java.util.TimeZone.setDefaultZone(TimeZone.java:657)
at java.util.TimeZone.getDefaultRef(TimeZone.java:624)
at java.util.TimeZone.getDefault(TimeZone.java:611)
at java.text.SimpleDateFormat.initializeCalendar(SimpleDateFormat.java:682)
at java.text.SimpleDateFormat.(SimpleDateFormat.java:581)
at java.text.SimpleDateFormat.(SimpleDateFormat.java:560)
at org.apache.log4j.helpers.PatternParser.finalizeConverter(PatternParser.java:265)
at org.apache.log4j.helpers.PatternParser.parse(PatternParser.java:186)
at org.apache.log4j.PatternLayout.setConversionPattern(PatternLayout.java:450)
... 31 more
Caused by: java.lang.RuntimeException: default directory must be absolute
at sun.nio.fs.UnixFileSystem.(UnixFileSystem.java:54)
at sun.nio.fs.LinuxFileSystem.(LinuxFileSystem.java:39)
at sun.nio.fs.LinuxFileSystemProvider.newFileSystem(LinuxFileSystemProvider.java:43)
at sun.nio.fs.LinuxFileSystemProvider.newFileSystem(LinuxFileSystemProvider.java:36)
at sun.nio.fs.UnixFileSystemProvider.(UnixFileSystemProvider.java:55)
at sun.nio.fs.LinuxFileSystemProvider.(LinuxFileSystemProvider.java:38)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at java.lang.Class.newInstance0(Class.java:372)
at java.lang.Class.newInstance(Class.java:325)
at sun.nio.fs.DefaultFileSystemProvider$1.run(DefaultFileSystemProvider.java:52)
at sun.nio.fs.DefaultFileSystemProvider$1.run(DefaultFileSystemProvider.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at sun.nio.fs.DefaultFileSystemProvider.createProvider(DefaultFileSystemProvider.java:42)
at sun.nio.fs.DefaultFileSystemProvider.create(DefaultFileSystemProvider.java:70)
at java.nio.file.FileSystems$DefaultFileSystemHolder.getDefaultProvider(FileSystems.java:108)
at java.nio.file.FileSystems$DefaultFileSystemHolder.access$000(FileSystems.java:89)
at java.nio.file.FileSystems$DefaultFileSystemHolder$1.run(FileSystems.java:98)
at java.nio.file.FileSystems$DefaultFileSystemHolder$1.run(FileSystems.java:96)
at java.security.AccessController.doPrivileged(Native Method)
at java.nio.file.FileSystems$DefaultFileSystemHolder.defaultFileSystem(FileSystems.java:95)
at java.nio.file.FileSystems$DefaultFileSystemHolder.(FileSystems.java:90)
... 47 more
Exception in thread "main" java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.LocalFileSystem could not be instantiated: java.lang.NullPointerException
at java.util.ServiceLoader.fail(ServiceLoader.java:224)
at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2659)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2670)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2687)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2723)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2705)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:407)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172)
at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:656)
at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:440)
at com.sumit.WordCount.main(WordCount.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.fs.Path.(Path.java:108)
at org.apache.hadoop.fs.Path.makeQualified(Path.java:467)
at org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:503)
at org.apache.hadoop.fs.RawLocalFileSystem.getInitialWorkingDirectory(RawLocalFileSystem.java:553)
at org.apache.hadoop.fs.RawLocalFileSystem.(RawLocalFileSystem.java:70)
at org.apache.hadoop.fs.LocalFileSystem.(LocalFileSystem.java:41)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at java.lang.Class.newInstance0(Class.java:372)
at java.lang.Class.newInstance(Class.java:325)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
... 18 more

Thanks,
Sumit

sumit-gupta-sgt commented Nov 7, 2015

After disabling IPv6 I am getting some java.lang.reflect.InvocationTargetException.
IPv6 is disabled properly ..i am not able to see in ifconfig command.

Here is full stack trace:
log4j:WARN Failed to set property [conversionPattern] to value "%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n".
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.log4j.config.PropertySetter.setProperty(PropertySetter.java:245)
at org.apache.log4j.config.PropertySetter.setProperty(PropertySetter.java:204)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:169)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:805)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
at org.apache.log4j.LogManager.(LogManager.java:127)
at org.apache.log4j.Logger.getLogger(Logger.java:104)
at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:262)
at org.apache.commons.logging.impl.Log4JLogger.(Log4JLogger.java:108)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at org.apache.commons.logging.impl.LogFactoryImpl.createLogFromClass(LogFactoryImpl.java:1025)
at org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:844)
at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:541)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:292)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:269)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
at org.apache.hadoop.util.ShutdownHookManager.(ShutdownHookManager.java:44)
at org.apache.hadoop.util.RunJar.run(RunJar.java:200)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ExceptionInInitializerError
at java.nio.file.FileSystems.getDefault(FileSystems.java:176)
at sun.util.calendar.ZoneInfoFile$1.run(ZoneInfoFile.java:490)
at sun.util.calendar.ZoneInfoFile$1.run(ZoneInfoFile.java:481)
at java.security.AccessController.doPrivileged(Native Method)
at sun.util.calendar.ZoneInfoFile.(ZoneInfoFile.java:480)
at sun.util.calendar.ZoneInfo.getTimeZone(ZoneInfo.java:663)
at java.util.TimeZone.getTimeZone(TimeZone.java:560)
at java.util.TimeZone.setDefaultZone(TimeZone.java:657)
at java.util.TimeZone.getDefaultRef(TimeZone.java:624)
at java.util.TimeZone.getDefault(TimeZone.java:611)
at java.text.SimpleDateFormat.initializeCalendar(SimpleDateFormat.java:682)
at java.text.SimpleDateFormat.(SimpleDateFormat.java:581)
at java.text.SimpleDateFormat.(SimpleDateFormat.java:560)
at org.apache.log4j.helpers.PatternParser.finalizeConverter(PatternParser.java:265)
at org.apache.log4j.helpers.PatternParser.parse(PatternParser.java:186)
at org.apache.log4j.PatternLayout.setConversionPattern(PatternLayout.java:450)
... 31 more
Caused by: java.lang.RuntimeException: default directory must be absolute
at sun.nio.fs.UnixFileSystem.(UnixFileSystem.java:54)
at sun.nio.fs.LinuxFileSystem.(LinuxFileSystem.java:39)
at sun.nio.fs.LinuxFileSystemProvider.newFileSystem(LinuxFileSystemProvider.java:43)
at sun.nio.fs.LinuxFileSystemProvider.newFileSystem(LinuxFileSystemProvider.java:36)
at sun.nio.fs.UnixFileSystemProvider.(UnixFileSystemProvider.java:55)
at sun.nio.fs.LinuxFileSystemProvider.(LinuxFileSystemProvider.java:38)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at java.lang.Class.newInstance0(Class.java:372)
at java.lang.Class.newInstance(Class.java:325)
at sun.nio.fs.DefaultFileSystemProvider$1.run(DefaultFileSystemProvider.java:52)
at sun.nio.fs.DefaultFileSystemProvider$1.run(DefaultFileSystemProvider.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at sun.nio.fs.DefaultFileSystemProvider.createProvider(DefaultFileSystemProvider.java:42)
at sun.nio.fs.DefaultFileSystemProvider.create(DefaultFileSystemProvider.java:70)
at java.nio.file.FileSystems$DefaultFileSystemHolder.getDefaultProvider(FileSystems.java:108)
at java.nio.file.FileSystems$DefaultFileSystemHolder.access$000(FileSystems.java:89)
at java.nio.file.FileSystems$DefaultFileSystemHolder$1.run(FileSystems.java:98)
at java.nio.file.FileSystems$DefaultFileSystemHolder$1.run(FileSystems.java:96)
at java.security.AccessController.doPrivileged(Native Method)
at java.nio.file.FileSystems$DefaultFileSystemHolder.defaultFileSystem(FileSystems.java:95)
at java.nio.file.FileSystems$DefaultFileSystemHolder.(FileSystems.java:90)
... 47 more
Exception in thread "main" java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.LocalFileSystem could not be instantiated: java.lang.NullPointerException
at java.util.ServiceLoader.fail(ServiceLoader.java:224)
at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2659)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2670)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2687)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2723)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2705)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:407)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172)
at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:656)
at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:440)
at com.sumit.WordCount.main(WordCount.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.fs.Path.(Path.java:108)
at org.apache.hadoop.fs.Path.makeQualified(Path.java:467)
at org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:503)
at org.apache.hadoop.fs.RawLocalFileSystem.getInitialWorkingDirectory(RawLocalFileSystem.java:553)
at org.apache.hadoop.fs.RawLocalFileSystem.(RawLocalFileSystem.java:70)
at org.apache.hadoop.fs.LocalFileSystem.(LocalFileSystem.java:41)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at java.lang.Class.newInstance0(Class.java:372)
at java.lang.Class.newInstance(Class.java:325)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
... 18 more

Thanks,
Sumit

@jkleint

This comment has been minimized.

Show comment
Hide comment
@jkleint

jkleint Nov 16, 2015

Owner

That looks like an unrelated problem in logging, but I'm not sure. I'm not able to maintain/test these playbooks any more; you may want to check out the hadoop roles on Ansible Galaxy:

https://galaxy.ansible.com/detail#/role/1026
https://galaxy.ansible.com/detail#/role/1263

Owner

jkleint commented Nov 16, 2015

That looks like an unrelated problem in logging, but I'm not sure. I'm not able to maintain/test these playbooks any more; you may want to check out the hadoop roles on Ansible Galaxy:

https://galaxy.ansible.com/detail#/role/1026
https://galaxy.ansible.com/detail#/role/1263

@jkleint jkleint closed this Nov 16, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment