Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

User level fs api

macan edited this page · 7 revisions

User Level File System API

Hard work for almost one week, we finally have a user level file system access interface now. We only implement library in C, while implementing by other languages is possible if you can follow the internal communication standard (named XNET).

We have implemented several important file system routines such as file create, stat, unlink, mkdir, rmdir, and readdir. The complete API reference is described in Section 2.

1. Quick Start with a Python Client

Using the C library, we create a simple file system client prototype in Python. You can find the source code at 'test/python/client.py'. To make fun with it, you have to boot Pomegranate in FS mode as describe in Getting-Start-fs or using our new file system boot script.

  • If you have configured your ssh authentication on your box, then you can use the following steps to boot the file system servers
$ MODE=fs bin/hvfs.sh start
...
$ bin/hvfs.sh stat
... # If you have seen all the servers are running, it is ok to go to next step.
  • If you have not configured your ssh authentication yet, we provide you an *except *script. Following the next steps:
$ USERNAME=your_user_name PASSWD=your_pass_word MODE=fs bin/hvfs.sh start
...
$ USERNAME=your_user_name PASSWD=your_pass_word bin/hvfs.sh stat
...

Next, create the file system 0

$ type=1 fsid=0 op=1 test/xnet/r2cli.ut 0
...

Then do as follows:

$ cd test/python
$ python client.py
FS Client 0 Running w/ (1 threads)...
[INFO] Client Self id 0 port 8412
[INFO] create DCONF cmd channel: /tmp/.MDS.DCONF.30065 w/ 0
[INFO] memlimit 0
[INFO] Begin parse the reg reply message
b01d016855b285f00fb1edb5751af494 CHRING
aa876e5d7c1a228c6ea54543d7929db1 CHRING
[INFO] fsid 0 gdt_uuid 0 gdt_salt 82b9ec3 root_uuid 1 root_salt f8f7f9c
[WARN] HVFS (/home/macan/hvfs/hvfs.macan/xnet/xnet_simple.c,  2113): xnet_replace_ipaddr[2a95588f80]: site 20000 addr 127.0.0.1 8412
[WARN] HVFS (/home/macan/hvfs/hvfs.macan/xnet/xnet_simple.c,  2113): xnet_replace_ipaddr[2a95588f80]: site 20001 addr 127.0.0.1 8413
[WARN] HVFS (/home/macan/hvfs/hvfs.macan/xnet/xnet_simple.c,  2113): xnet_replace_ipaddr[2a95588f80]: site 20002 addr 127.0.0.1 8414
[WARN] HVFS (/home/macan/hvfs/hvfs.macan/xnet/xnet_simple.c,  2113): xnet_replace_ipaddr[2a95588f80]: site 20003 addr 127.0.0.1 8415
[WARN] HVFS (/home/macan/hvfs/hvfs.macan/xnet/xnet_simple.c,  2113): xnet_replace_ipaddr[2a95588f80]: site 40000 addr 127.0.0.1 8210
[WARN] HVFS (/home/macan/hvfs/hvfs.macan/xnet/xnet_simple.c,  2113): xnet_replace_ipaddr[2a95588f80]: site 40001 addr 127.0.0.1 8211
[WARN] HVFS (/home/macan/hvfs/hvfs.macan/xnet/xnet_simple.c,  2113): xnet_replace_ipaddr[2a95588f80]: site 40002 addr 127.0.0.1 8212
[WARN] HVFS (/home/macan/hvfs/hvfs.macan/xnet/xnet_simple.c,  2113): xnet_replace_ipaddr[2a95588f80]: site 40003 addr 127.0.0.1 8213
[WARN] HVFS (/home/macan/hvfs/hvfs.macan/xnet/xnet_simple.c,  2113): xnet_replace_ipaddr[2a95588f80]: site 60000 addr 127.0.0.1 8810
[WARN] HVFS (/home/macan/hvfs/hvfs.macan/xnet/xnet_simple.c,  2113): xnet_replace_ipaddr[2a95588f80]: site 60001 addr 127.0.0.1 8811
[WARN] HVFS (/home/macan/hvfs/hvfs.macan/xnet/xnet_simple.c,  2113): xnet_replace_ipaddr[2a95588f80]: site 80000 addr 127.0.0.1 8710
[WARN] HVFS (/home/macan/hvfs/hvfs.macan/xnet/xnet_simple.c,  2113): xnet_replace_ipaddr[2a95588f80]: site c0000 addr 127.0.0.1 9001
[WARN] HVFS (/home/macan/hvfs/hvfs.macan/xnet/xnet_simple.c,  2113): xnet_replace_ipaddr[2a95588f80]: site c0001 addr 127.0.0.1 9002
[INFO] AMI gdt uuid 0 salt 82b9ec3 root uuid 1 salt f8f7f9c for site client 20000
Welcome to Python FS Client Shell, for help please input ? or help
(Cmd)
EOF            commit         cpout          getactivesite  help           mkdir          online         rmdir          touch
cat            cpin           delete         getcluster     ls             offline        quit           stat           
(Cmd) 

The simple shell is self-documented, and we just give you a short demo :) In the following steps 1. check whether there are entries in the root directory. 'None' means there is no active entry.

(cmd) ls /
[ERR ] HVFS (/home/macan/hvfs/hvfs.macan/mds/dh.c,   343): mds_dh_load[2a95588f80]: Load DH (remote): uuid 1 itbid 7, site 40002
[INFO] refresh uuid 1 salt 127188722 bitmaps
[WARN] HVFS (/home/macan/hvfs/hvfs.macan/mds/xtable.c,   742): mds_bitmap_load[2a95588f80]: bitmap f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
None
Time elasped: 0.003186s
  1. make a new directory 'dir' in the root directory. I will describe the result string in detail in Section 2.
(cmd) mkdir /dir
1 794bef2 8000400000000003 8000000 0 0 40755 2 0 0 1288103578 1288103578 1288103578 0 0 $0$0$ [0 0 0 0]

Time elasped: 0.000983s
  1. create a new file 'first' in the root directory.
(cmd) touch /first
1 794bef2 400000000004 8000000 0 0 100644 1 0 0 1288103641 1288103641 1288103641 0 0 $0$0$ [0 0 0 0]

Time elasped: 0.000176s
  1. copy a local file to Pomegranate file '/first'.
(cmd) cpin ./client.py /first
[WARN] HVFS (/home/macan/hvfs/hvfs.macan/api/api.c,  5207): __hvfs_fwrite[2a95588f80]: puuid 1 uuid 400000000004 to 60000 C 0 L @ 0 len 19777
+OK
Time elasped: 0.000750s
  1. check the content of file '/first'.
(cmd) cat /first
....
  1. copy a local file to Pomegranate file '/second', the new file will be create automatically.
(cmd) cpin ./client.py /second
[ERR ] HVFS (/home/macan/hvfs/hvfs.macan/api/api.c,  4131): __hvfs_stat[2a95588f80]: LOOKUP failed @ MDS site 40003 w/ -2
[WARN] HVFS (/home/macan/hvfs/hvfs.macan/api/api.c,  5207): __hvfs_fwrite[2a95588f80]: puuid 1 uuid 600000000003 to 60001 C 0 L @ 0 len 19777
+OK
Time elasped: 0.001803s
  1. copy Pomegranate file '/second' to a local file './client.py.bak'.
(cmd) cpout /second ./client.py.bak
Time elasped: 0.000470s
  1. switch to another console and check if the file 'client.py.bak' is the same as 'client.py'.
$ md5sum client.py client.py.bak
93d697adc2f5aa39d6121eb3278a9b9e  client.py
93d697adc2f5aa39d6121eb3278a9b9e  client.py.bak
  1. list the root directory. The first character 'd' means this entry is a directory, while '-' means it is a regular file. The second field is flags and permission. The third field is the file system inode number. And the last field is the entry name.
(Cmd) ls /
[INFO] refresh uuid 1 salt 127188722 bitmaps
d 040755     8000400000000003 dir
- 100644         400000000004 first
- 100644         600000000003 second

Time elasped: 0.002027s
  1. delete file '/first'.
(Cmd) delete /first
1 794bef2 400000000004 8000000 0 0 100644 0 19777 0 1288103641 1288103641 1288103641 0 0 $0$0$ [0 0 0 0]

Time elasped: 0.000234s
  1. remove the directory '/dir'. (Note that, at this moment, we do not check the subdirs and entries. These entries will be leaked!)
(Cmd) rmdir /dir
1 794bef2 8000400000000003 8000000 0 0 40755 0 0 0 1288103578 1288103578 1288103578 0 0 $0$0$ [0 0 0 0]

Time elasped: 0.000257s
(Cmd) ls /
[INFO] refresh uuid 1 salt 127188722 bitmaps
- 100644         600000000003 second

Time elasped: 0.001870s

2. API Reference (C Implementation)

  • Initiate a client framework
int __core_main(int argc, char *argv[])

There are some basic information you have to provide to init the client framework.

-d, --id number     self client/amc id. (amc is Another Metadata Client.)
-p, --port number  self client/amc port.
-r, --root ip_addr   ip address of root server.
-f, --fsid number    file system ID. (for file system, you should set it to 0; the default value is 1, you have overwrite to 0.)
-y, --type string    client type: client/amc. (for file system, you should set client type to 'client'; otherwise, set it to 'amc'.)
-h, --help
  • Finalize the client framework
void __core_exit(void)
  • Stat a file
int hvfs_stat(char *path, char *name, void **data_back)

Stat a file 'path/name' to get the file attributes such as file mode, size, and name. On success, this function return 0 with a valid string contains file attributes. The are 20 fields in the returned string pointed by 'data-back'.

/* @data_back: It is a string return back to caller.
 *
 * The format of data_back is:
 * F1: puuid # for dir it is the self uuid
 * F2: psalt # for dir it is the self salt
 * F3: uuid # self uuid
 * F4: mdu.flags # mdu flags
 * F5: mdu.uid # mdu user id
 * F6: mdu.gid # mdu group id
 * F7: mdu.mode # mdu mode
 * F8: mdu.nlink # mdu nlink
 * F9: mdu.size # mdu saved file length
 * F10: mdu.dev # mdu saved dev
 * F11: mdu.atime # mdu saved access time
 * F12: mdu.ctime # mdu saved change time
 * F13: mdu.mtime # mdu saved modify time
 * F14: mdu.dtime # mdu saved delete time
 * F15: mdu.version # mdu saved version
 * F16: mdu.symname or llfs_ref # $REF_COLUMN$ or $fsid$rfino$ or symname
 * F17: column_no
 * F18: c->saved_itbid
 * F19: c->len
 * F20: c->offset
 */
  • File or directory Creation
int hvfs_create(char *path, char *name, void **data, u32 is_dir)

Create a file or directory 'path/name'. The file or directory attributes are returned by 'data'.

  • Update the file attributes with the new one
int hvfs_fupdate(char *path, char *name, void **data)

Using a new file attributes pointer by 'data' to update the file's old attributes. The declaration of the updater structure is shown as follows:

/*
 * used for setattr and create
 *
 * Layout of mdu_update:
 *
 * |---mdu_update---|---symname---| or
 * |---mdu_update---|--llfs_ref--|---mu_column---|
 */
struct mdu_update 
{
    /* flags for valid */
#define MU_MODE         (1 << 0)
#define MU_UID          (1 << 1)
#define MU_GID          (1 << 2)
#define MU_FLAG_ADD     (1 << 3)
#define MU_ATIME        (1 << 4)
#define MU_MTIME        (1 << 5)
#define MU_CTIME        (1 << 6)
#define MU_VERSION      (1 << 7)
#define MU_SIZE         (1 << 8) /* for truncate? */
#define MU_FLAG_CLR     (1 << 9)
#define MU_COLUMN       (1 << 10) /* update column infomation? */
#define MU_NLINK        (1 << 11)
#define MU_SYMNAME      (1 << 12) /* symbol name */
#define MU_LLFS         (1 << 13) /* llfs_ref */

    u64 atime;
    u64 mtime;
    u64 ctime;
    u64 size;                   /* for truncate? */

    u32 valid;

    u32 uid;
    u32 gid;

    u32 flags;

    u32 version;
    s32 nlink;
    u16 mode;
    u16 column_no;              /* # of columns */
    u16 namelen;                /* name length of symbol name */
};

struct column                   /* 24B */
{
    u64 stored_itbid;           /* for computing the location of dfile */
    u64 len;
    u64 offset;
};

struct mu_column
{
    u64 cno;
    struct column c;
};

Thus, you can construct a struct mdu_update to update the file attributes. Note that, you have to mask the corresponding flags in mdu_update.valid.

  • Delete a regular file
int hvfs_fdel(char *path, char *name, void **data_back)

Delete a regular file is trival :) We return the just deleted file's attributes to you if you have special needs.

  • Read the file content
int hvfs_fread(char *path, char *name, int column, void **data, u64 *len)

At this moment, we do not supply the read interface to read from a file offset. But, there is no restriction to forbid us to read from offset other 0 :) We will add a routine looks like libc.pread().

  • Write the data region to a file
int hvfs_fwrite(char *path, char *name, int column, void *data, u64 len)

At this moment, we do not supply the write interface to write at a file offset. You should note that any file in Pomegranate is internally immutable. Thus, writes from a offset should be transformed to 'read to memory, modify, and write to new position'. It is a little inefficient, however, most little files are not modified frequently or they would not be directly written by pread (say they had already been read in memory).

  • Trigger a memory snapshot in MDS
int hvfs_fcommit(int id)

Trigger a memory snapshot in MDS server 'id'. All the dirty metadata is written back to the storage layer.

  • Free the memory buffer allocated by the library
void hvfs_free(void *p)
Something went wrong with that request. Please try again.