Getting error "Can not start filer" after restarting the server

[mosic]

As the previous issue (#273) got closed after the crashes were fixed, I'm making a new one specifically about the problem with the filer.

In short, if I upload 1 million files using filer and then restart the server this is the output I get:

...
I0331 01:07:34 28195 files_in_leveldb.go:41] directory 333 fileName 16_45_260240.jpg fid 41,0f421cc377e0a5
I0331 01:07:34 28195 filer_server_handlers.go:209] post result {"name":"gpocam.jpg","size":187580}
I0331 01:07:34 28195 filer_server_handlers.go:234] saving /gpocam/snapshots/recordings/2016/04/11/12/16_56_260240.jpg => 41,0f422758b73c5a
I0331 01:07:34 28195 files_in_leveldb.go:41] directory 333 fileName 16_56_260240.jpg fid 41,0f422758b73c5a
^CI0331 01:07:37 28195 volume_server.go:117] Shutting down volume server...
I0331 01:07:37 28195 volume_server.go:119] Shut down successfully!


$ ./weed -v=4 server -filer=true -volume.max 50 -dir /mnt/seaweedfs
I0331 01:07:38 13324 file_util.go:20] Folder /mnt/seaweedfs Permission: -rwxrwxr-x
I0331 01:07:38 13324 file_util.go:20] Folder /mnt/seaweedfs/filer Permission: -rwx------
I0331 01:07:38 13324 file_util.go:20] Folder /mnt/seaweedfs Permission: -rwxrwxr-x
I0331 01:07:38 13324 topology.go:86] Using default configurations.
I0331 01:07:38 13324 master_server.go:59] Volume Size Limit is 30000 MB
I0331 01:07:38 13324 server.go:205] Start Seaweed Master 0.70 beta at localhost:9333
F0331 01:07:38 13324 filer_server.go:53] Can not start filer in dir /mnt/seaweedfs/filer : /gpocam/snapshots/recordings/2016/04/05/09 should be have id 145 instead of 156
goroutine 36 [running]:
github.com/chrislusf/seaweedfs/go/glog.stacks(0x11ed100, 0x0, 0x0, 0x0)
  /home/mosic/go/src/github.com/chrislusf/seaweedfs/go/glog/glog.go:767 +0xb8
github.com/chrislusf/seaweedfs/go/glog.(*loggingT).output(0x11ce5a0, 0xc800000003, 0xc8201a6c00, 0x11a4806, 0xf, 0x35, 0x0)
  /home/mosic/go/src/github.com/chrislusf/seaweedfs/go/glog/glog.go:718 +0x259
github.com/chrislusf/seaweedfs/go/glog.(*loggingT).printf(0x11ce5a0, 0xc800000003, 0xe3c600, 0x22, 0xc82004dd20, 0x2, 0x2)
  /home/mosic/go/src/github.com/chrislusf/seaweedfs/go/glog/glog.go:656 +0x1d4
github.com/chrislusf/seaweedfs/go/glog.Fatalf(0xe3c600, 0x22, 0xc82004dd20, 0x2, 0x2)
  /home/mosic/go/src/github.com/chrislusf/seaweedfs/go/glog/glog.go:1149 +0x5d
github.com/chrislusf/seaweedfs/go/weed/weed_server.NewFilerServer(0xc8200e3860, 0x22b8, 0xc820171370, 0xe, 0xc820172580, 0x1a, 0x0, 0x0, 0xd723e8, 0x3, ...)
  /home/mosic/go/src/github.com/chrislusf/seaweedfs/go/weed/weed_server/filer_server.go:53 +0xb9e
main.runServer.func1()
  /home/mosic/go/src/github.com/chrislusf/seaweedfs/go/weed/server.go:174 +0x206
created by main.runServer
  /home/mosic/go/src/github.com/chrislusf/seaweedfs/go/weed/server.go:189 +0xdf9

[chrislusf]

Can you please attach the "dir.log" file if you can find it?

btw, it's not recommended to have deep directories.
Just a few directories with many many files under each directory is better.

[mosic]

These are the contents of dir.log: https://gist.github.com/mosic/d3db68cc12e985327954cc0cbb41da1b

I'm also attaching the whole filer directory, in case it's useful: filer.zip

What would you recommend to use as the directory structure? I've picked the current one because it makes it easy to go back and delete directories that represent a certain day. Slightly more about that here: https://github.com/evercam/evercam-api/wiki/File-Storage

Would {camera_id}/{source}/2016-12-31/23-59_59_999.jpg be noticeably better for seaweed?

[chrislusf]

The dir.log is very helpful! I am going to fix it.
Please keep current tests so that we can confirm we fix this issue.

Under one folder, this would work better:
{camera_id}-{source}-2016-12-31-23-59_59_999.jpg

This is because there would not be many directories at all, avoiding many concurrent directory tree traversing.

[mosic]

That's great news!

If I was using a single directory, how could I delete e.g. all files saved today? In the current structure, I would simply use:

curl -X DELETE "http://localhost:8888/{camera_id}/snapshots/{source}/2016/04/05/?recursive=true"

It would be great if wildcards were supported, so I could do something like:

curl -X DELETE "http://localhost:8888/{camera_id}-{source}-2016-04-05-*?recursive=true"

[chrislusf]

ok. Maybe use the directory for now. The less number of directory, the better.

We may add wildcard support later. But just don't count on it.