Skip to content
This repository was archived by the owner on Jul 7, 2023. It is now read-only.
This repository was archived by the owner on Jul 7, 2023. It is now read-only.

how to use hdfs in distributed t2t #1641

@colmantse

Description

@colmantse

I am working on distributed t2t and i managed to set up to run in nfs environment but when i change to hdfs path, it gives the log as attached. I would like to confirm if setting environment variables and changing to hdfs is enough for t2t to use hdfs or if i am missing anything more.

referenced:
tensorflow/tensorflow#30981

util.NativeCodeLoader: Unable to load native-hadoop library for your platform
info retry.RetryInvocationHandler org.apache.hadoop.net.ConnectTimeoutException: Call From ... failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=...] while invoking ClientNameNodeProtocolTranslatorPB.getFileinfo over ... after 1 failover attempts. Trying to failover after sleeping for 781ms

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions