why is hdfs throws unexpected EOF error using golang?

[kof02guy]

package main
import (
    "fmt"
    "github.com/colinmarc/hdfs"
)
func main() { 
    client, err := hdfs.New("192.168.0.38:50070")
    fs, err := client.ReadDir("/")
    fmt.Println(err)
    fmt.Println(fs)
}

err is unexpected EOF And I found that the error occurs in

func (c *NamenodeConnection) readResponse(method string, resp proto.Message) error {
...
    _, err = io.ReadFull(c.conn, packet)
...
}

I'm using hadoop 2.7.3

[colinmarc]

Hi, That's the default port for the web UI. You probably want 8020: http://blog.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/

[kof02guy]

Hi Colinmark,
I've changed this but got the following error
dial tcp 192.168.0.38:8020: getsockopt: connection refused
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x467f5e]

goroutine 1 [running]:
panic(0x565260, 0xc42000c100)
/usr/local/go/src/runtime/panic.go:500 +0x1a1
audience/vendor/github.com/colinmarc/hdfs.(*Client).getFileInfo(0x0, 0x5960f9, 0x8, 0xc420067e70, 0x45c807, 0x64eac0, 0xc420046010)
/export/bzhang/devCodeForPrd/common/src/go/src/audience/vendor/github.com/colinmarc/hdfs/stat.go:35 +0xfe
audience/vendor/github.com/colinmarc/hdfs.(*Client).Open(0x0, 0x5960f9, 0x8, 0x3b, 0x0, 0x0)
/export/bzhang/devCodeForPrd/common/src/go/src/audience/vendor/github.com/colinmarc/hdfs/file_reader.go:34 +0x49
main.main()
/export/bzhang/devCodeForPrd/common/src/go/src/audience/new_audience/file_translator/test2.go:11 +0xe2
It seems that the port is not opened.

I can contact hdfs using github.com/vladimirvivien/gowfs

[colinmarc]

github.com/vladimirvivien/gowfs uses WebHDFS, so that makes sense. I'd guess either you're using a nonstandard port (I've seen 9000 used elsewhere) or you have a firewall blocking access.

That panic doesn't look great, though, so I'll leave this open for now.

[colinmarc]

Hm, can you post more of the traceback, actually? It looks like you're calling Open on a nil Client.

[kof02guy]

I made some changes to this piece of code:
`package main

import (
"fmt"
"github.com/colinmarc/hdfs"
)

func main() {
client, err := hdfs.New("192.168.0.38:9000")
fmt.Printf("client:%v err:%v\n", client, err)
file, err := client.Open("testfile")
fmt.Printf("file:%v err:%v\n", file, err)
}`

and its output:

client:<nil> err:dial tcp 192.168.0.38:9000: getsockopt: connection refused
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x467c3e]

goroutine 1 [running]:
panic(0x565260, 0xc42000c100)
        /usr/local/go/src/runtime/panic.go:500 +0x1a1
audience/vendor/github.com/colinmarc/hdfs.(*Client).getFileInfo(0x0, 0x5960f9, 0x8, 0x64ea80, 0xc420048010, 0x597f0f, 0x11)
        /export/bzhang/devCodeForPrd/common/src/go/src/audience/vendor/github.com/colinmarc/hdfs/stat.go:35 +0xfe
audience/vendor/github.com/colinmarc/hdfs.(*Client).Open(0x0, 0x5960f9, 0x8, 0x2, 0x2, 0x4c)
        /export/bzhang/devCodeForPrd/common/src/go/src/audience/vendor/github.com/colinmarc/hdfs/file_reader.go:34 +0x49
main.main()
        /export/bzhang/devCodeForPrd/common/src/go/src/audience/new_audience/file_translator/test2.go:11 +0x10b

when I use 50070 as its port to open:

client:&{0xc4200ec000 <nil>} err:<nil>
1213486160 &{{0xc4200740e0}}
unexpected EOF
file:<nil> err:open testfile: unexpected EOF

I added one line in namenode.go

func (c *NamenodeConnection) readResponse(method string, resp proto.Message) error {
    var packetLength uint32
    err := binary.Read(c.conn, binary.BigEndian, &packetLength)
    if err != nil {
        return err
    }

    packet := make([]byte, packetLength)
    fmt.Println(packetLength, c.conn)
    ...
}

and the Println outputs "1213486160 &{{0xc4200740e0}}"

So I think 50070 is ok, but the response decode failed. And I've tried 8020 and 9000, but both these 2 ports can not be connected to generate a valid client as you can see.

[kof02guy]

Thank you very much. I've found the reason for the problem. I should use 127.0.0.1 as the address to open because HDFS opens its IPC port on 127.0.0.1 instead of 0.0.0.0
I use 127.0.0.1:9000 and the error is gone

[colinmarc]

Great - I'm going to close this, but feel free to reopen if you find any new issues.