-
Notifications
You must be signed in to change notification settings - Fork 0
/
notes.txt
125 lines (98 loc) · 5.48 KB
/
notes.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
Some more notes
Need mime type on FileNodes so you can choose how to index out data
Snap Tag command - creates named tag for existing or current version
For a tag should also have a timestamp (including versions)
So a snap time could also work - call, recursively a snap for a tag given a time
Perhaps the tree node entry is a block node, not the list itself,
then you would only need to persist the tree when you add a new node, at the expense
of perhaps slower lookups of the values because of the node indirection
Some more notes
DONE Do block add command
DONE Block range command (though how to make that work with update data on pure file)
then...
Attributes of node (commands to add, listall, get)
Then...
Reverse search index (btree of words to url + version) (so rebuild for each update)
Search index stored in repo. So how to search all repos (mounted etc.?) Maybe you have to
provide the urls of the mount points in search
BTree search notes
We have a big list of words
Maybe we have a list of list of words (a key is the first set, and one key is "text")
Each word has associated with it a list of paths and versions
so
text -> one -> /alan/one v1
text -> one -> /alan/one v2
given a search for "text = one" it could return all of those versions for /alan/one
(maybe we should have a latest version tag as well)
textLatest -> one -> /alan/one
Maybe text -> one is a btree (GoLLRB)
and the value is the list of data
Some more notes (added here at the top)
Fix console by putting in lock between notification updates and console updates
Do web front end next
Web API (kind of like api.github.com)
1. Most responses are in json form, except for getting content from a regular file (the file contents is returned
the mime type is set in the response).
2. The general form is:
[url]?cmd=[command on that url]&[any other parameters required for that command]
3. For uploading new data a POST form to a uri will also work (saves putting content in query parameters)
4. The default command is "get" so it can be omitted. So a simply uri corresponds to "return the contents of that file" or "return a
json document with information about this directory".
5. Other commands
stat = stat of uri (returned as json)
append = append data to that uri
tags = tags of that uri
gettag = retrieve the contents given a tag
blocks = retrieve a list of the block names for the tag
getrange = retrieve contents based on a range (start point, end point, each can be open) (with optional version)
delete = remove content
mount = create a mount point
link = create a link
diff = difference between two tags (TBD)
Some notes on what needs to be done for pmfs
(DONE for now) Append to File
Copy File (do this at the block level really - and between file systems potentially - it's really a series of append file calls for each block)
Create file of a given type (which can also be used to create a mountable file)
Handling traversal of mounted filesystems
(DONE) Upper side caching of nodes, channel to sync updates, non-removal of cached items that are not synced
Caching of data nodes (treat data nodes separately)
Series files (pass inode and key/value to underlying)
Tags (filesystem based tagname -> key/value (and path or inode?), with the ability to search, order, filter etc.)
e.g. Color -> Red -> [ /one/two, /three/four]
Find paths where color = 'Red'
Updating tags on deletion (so perhaps a reverse index needs to also be stored)
Tag data saved as nodes of course
Versioning
Create new version instead of just overwriting. Version links could be perhaps tags pointing to inodes of the original file node
instead of paths
E.g. Version -> /one/two/three [ 1 -> 12223, 2 -> 22211, 3 -> 23324 ]
Functions to page through a file or directory listing (byte point to byte point, series start to end, etc.)
MongoDB implementation
Cassandra implementation
Postgresql implementation
(INPROGRESS) Client side shell https://github.com/nsf/termbox-go
Thrift api + client side application (using that console based library?) (like a shell prompt with some screen stuff going on)
Web front end (browse structure, search using tags, browse versions, update data, append data, delete data, create mount points, archive versions)
Symbolic links (hard and soft)
Channel on shell to show output from filesystem execution (messages basically)
Sample commands through the "shell"
cd /alan ; do a stat on the folder
ls ; dir listing
cat /alan/1 ; print out contents
append /alan/1 content ; add to file
append 1 content ; add to file, using the current directory
rm 1 ; delete a file
copy 1 2 ; copy one file to another
stat 1 ; print out information on a file
mount /fred/mongo mongodb:/127.0.0.1:1222/db collectionName
append /fred/mongo/one Some content ; Would now be writing to mongodb
cat /fred/mongo ; Would write out some information about the mount point
rm /fred/mongo ; Would remove that mount point (maybe not removing the data?)
(DONE) delete command in shell
When you create a file (add a file) you need to specify the mime type (it defaults to text/plain). The mime type
is stored in the File node.
A directory node can be a mount type. If we traverse through that we pass on the remaining path to the cached filesystem
associated with that node id, if the cached fs does not exist we load it from the contents of the extended Block node attributes
of the directory node (which should contain some structure that was passed during an "add mount" call)
Some bugs
(DONE) appending data goes into continuation continually once it starts