considered-elements.tex

% STATUS: Complete & Checked
Several key elements of a storage network were identified and different design decisions based around these elements have been considered. 
		
		\subsubsection{Network Connectivity}
			In order to join the network, a node must have some knowledge of where to find other nodes. This can be achieved through the use of some central authority that holds a list of nodes, enabling anyone to acquire the knowledge to use the network, or each node can act as a part of a decentralised directory, where as long as one node can be connected to, the remaining participants of the network can be discovered.
			
			Neither solution is without issues: the central authority could reduce security by providing a more important class of node that could be seen as a target by an adversary, thereby reducing the security, while the acquisition of the knowledge needed to connect to the initial node would have to occur out-of-band and any information from a decentralised directory may be subject to abuse as an element of trust is required.
			
			Although the initial connection node may seem to be much the same as the central authority, it would be possible to have several of these initial connection nodes, and as long as they are members of the network, this is acceptable. This presents any application the freedom to use any nodes as initial connection nodes, as opposed to limiting it to a single central authority.
			
			% ISSUES: This may be better suited in the API design section
			As the introduction of a central authority presents too much risk and lowers the amount of freedom compared to a distributed directory, the API should provide a mechanism to allow users of the API to set initial connection nodes in order to discover the rest of the network.
			
			Alternatively, if all users have access to some information stream where steganography could be used, it may be possible to use this for the purpose of advertising nodes. The Bitcoin Message Service uses a similar mechanism for the distribution of messages in the Bitcoin blockchain \cite{btcmsg}. As all Tor users have access to Tor directory nodes, it was hoped that hidden service descriptors could be used for steganography, however, due to the nature of Tor's directory nodes, this method is very unreliable, although possible, as some malicious nodes have used this mechanism to harvest hidden service descriptors \cite{crawl}.
			
			% Complete & Checked
			Given the nature of anonymous services, such as Tor's hidden services, the initiator of a connection is aware of the destination network identifier (or, pseudoidentity), as this is required to make the connection, although the target of the connection has no knowledge of the initiator's psuedoidentity. This has the result that connections can be considered unidirectional, although the transfer of data is bidirectional once the connection is established. In some cases, determining the pseudoidentity of the initator is desired and so this presents an interesting challenge. The initator could tell the target their pseudoidentity, which the target could either check in some way, or trust. Alternatively, as the pseudoidentity is likely to have asymmetric key backing, a digital signature could be appended to a message, proving the pseudoidentity of the initator.
			
			\input{internodal-connections}
			
			% Complete & Checked
			The topology of the network describes how nodes organise themselves and connect to each other. In some networks, such as Freenet, nodes organise themselves into a small-world topology, where any node can communicate with any other node through some number of other nodes. This is done as a mechanism for implementing anonymity, but creates an attack vector where an adversary may be able to manipulate the topology to allow for attacks \cite{pretre2005attacks}. As an existing anonymity layer will be used, there is no need to use the topology to keep nodes anonymous. Rather, it will be considered mainly for purpose of information distribution.
			
			% Complete & Checked
			Since any topology that allows node connections to be directly manipulated presents a potential attack vector, a purely logical topology, where nodes each decide who they connect to unidirectionally should be used. Since this method relies on no external influence to any node, there is a chance that some nodes may not be selected for a connection by any other node, thereby excluding them. While this is only an issue if a large number of nodes are not selected, as this effectively reduces the potential resources in the network, this number should be kept to a minimum. Formula \ref{eq:internodalconnections} shows the probability of any node not being selected by any other for a unidirectional connection. Figure \ref{fig:internodalconnections} shows this probability where each node maintains 20 unidirectional connections out. As shown, this probability is very low and even when the network is large, under the condition of periodic connection renewal, so that newer nodes are included, it is likely that all nodes will always be selected for a connection. Figure \ref{fig:internodalexample} shows an example network with the internodal connections.			
			\begin{figure}
				\centering
				\begin{tikzpicture}[auto, thick]
				\tikzstyle{vertex}=[circle,fill=black!25,minimum size=17pt,inner sep=0pt]
				\node[vertex] (Node-1) at (-2,0) {$1$};
				\node[vertex] (Node-2) at (2,0) {$2$};
				\node[vertex] (Node-3) at (0,-2) {$3$};
				\node[vertex] (Node-4) at (0,2) {$4$};
				\node[vertex] (Node-5) at (0,0) {$5$};
				
				\draw[-latex, thick, dashed, black] (Node-1) .. controls (0,1) .. (Node-2);
				\draw[-latex, thick, dashed, black] (Node-1) -- (Node-5);
				
				\draw[-latex, thick, dashed, black] (Node-2) .. controls (0,-1) .. (Node-1);
				\draw[-latex, thick, dashed, black] (Node-2) .. controls (1.1,-1.1) .. (Node-3);
				
				\draw[-latex, thick, dashed, black] (Node-3) .. controls (0.9,-0.9) .. (Node-2);
				\draw[-latex, thick, dashed, black] (Node-3) -- (Node-1);
				
				\draw[-latex, thick, dashed, black] (Node-4) -- (Node-2);
				\draw[-latex, thick, dashed, black] (Node-4) -- (Node-5);
				
				\draw[-latex, thick, dashed, black] (Node-5) -- (Node-3);
				\draw[-latex, thick, dashed, black] (Node-5) -- (Node-2);
				
				\node[text width=5cm] at (6,0) {This example network consists of 5 nodes, each with 2 connections. In this case, node 4 has not been selected for a connection, while node 2 is connected by every other node.};
				
				%\draw[-latex, thick, dashed, black] (Node-2) -- (Node-1);
				\end{tikzpicture}	
				
				\caption{High-level internodal connection diagram example}
				\label{fig:internodalexample}
				%\end{center}
			\end{figure}
			
			
			
			
			
			
			
			
			
			
			
			
			
			
			
			
			
			
			
		\subsubsection{Packet Classes}
			\begin{description}[topsep=-5pt,itemsep=-1ex,partopsep=2ex,parsep=1.5ex]
				\item[Data as Notifications] \hfill \\
				A single class of data packet is used, where the intended message and notification of the existence of said message are combined. This means that if a notification is successful, the data will always be available, resulting in lower delivery latency. However, as the size of some packets could be large, and the majority of this information is destined for most nodes, this is likely to be wasteful and could cause scalability issues.
				\item[Notifications Referencing Data] \hfill \\
				Two classes of packet are used: a data packet that is just for storing data and a notification packet that is only for notifying a node about the existence of a data packet. While all nodes still need to check all the notifications, only a subset of all nodes need to be in possession of the data packets. In addition, the notifications may also include information such as some known locations of the desired data packet (rendezvous nodes). This method has the potential to reduce the availability of data packets that are not distributed well throughout the network.
				\item[Notifications with Small Payload] \hfill \\
				Each notification packet is a union\footnote{A union type is one that provides an either-or contents structure.} of a small payload or a data packet reference. The total size of this is still comparable to that of a typical notification packet. The result of this structure would be very high availability for any small payload.
			\end{description}
			Given that notifications with small payloads have the potential to provide maximal flexibility while also reducing the load on the network, these will be used.
			
		\subsubsection{Information Distribution and Delivery}
			Distributing the information across the network must occur in order to keep the availability of information high. In order to receive a message, everyone must either check every message, or there must be some form of delegation to a third party, or parties, that must perform the same role.
			
			Systems, such as Bitmessage, take the approach that everyone can attempt to determine if a message is for them, whereas other systems, such as email, take advantage that the recipient of a message can be determined and use this to place the message in a predictable location. Other networks, such as Freenet, move the obligation of notification to out-of-band communications, meaning they do not address this issue.
			
			As the Bitmessage network grows, it splits into ``streams''~\cite{bitmsg}, effectively creating smaller subnetworks. Membership of a stream can be a function of some identifying element of the user, such as a public key (figure \ref{fig:stream-normal}). Although this method is promising in terms of scalability, it has some disadvantages for the anonymity of the network. This type of optimisation is only possible where each stream has enough members such that if the stream was isolated, the property of anonymity would hold. Under the assumption that one user controls exactly one node, it is enough for the node membership of any stream for be sufficiently high. However, if this assumption cannot be made, then it opens the possibility to deanonymisation attacks. The worst-case for this attack is that it may be possible to determine the public key, and therefore the identity that a node is interested in retrieving messages for. This would occur where an adversary is able to isolate a node in a stream (figure \ref{fig:stream-malicious}) and could happen where an adversary creates a large number of nodes with similar public keys (through trial and error, or otherwise) to the target node. Although this may only reveal the recipient's identity for new packets in the stream where the isolation attack is taking place, it is still undesirable.
			
			\input{streams-1}
			
			While all nodes need to check all messages to a degree, this mechanism is only required to notify of a message. Hence, instead of combining the notification and data functionality in a single packet, and having everyone acquire all of these, a split can be made. This would still require everyone to acquire and check all the notification packets, but these notifications could point to a data packet elsewhere on the network, along with a collection of rendezvous nodes that are known to be in possession of these data packets.
			
			\subsubsection{Broadcast Handling}
			Notifications will be broadcast to at least all nodes that are members of a stream, this is likely to be the bottleneck of the system. The handling of broadcasts should therefore be subject to optimisation where possible.

			\begin{description}[topsep=-5pt,itemsep=-1ex,partopsep=2ex,parsep=1.5ex]
				\item[Universal Polling (UP)] \hfill \\
				Nodes query other nodes for a list of their stored broadcasts, acting like a distributed hash table. The querying node can then request any broadcast it does not have stored. If the rate of broadcasts inserted into the network is greater than the rate they expire, then the size of the query response will grow unsustainably.
				\item[Active Forwarding (AF)] \hfill \\
				Upon receiving a broadcast for the first time, the node forwards it to all of its connections. This does not require any list of broadcasts to be distributed, saving on some overhead. However, as the node initiating the forwarding is unaware if the nodes are in possession of the broadcast, the receiving node will be responsible for signalling that further transfer is not required, probably in the form of terminating the connection.
				\item[Selective Polling (SP)] \hfill \\
				When a node receives a broadcast, it also stores the time of receipt. When a node polls another node, it stores the time of polling for that node. In a similar way to UP, polling is used to gather a list of broadcasts, but a parameter specifying the last time of polling is passed too. The polled node returns the list of all broadcasts that it received after this time. This has potential anonymity implications where it may reveal the time at which a node received a broadcast. However, this can be negated by having the receiving node add a small random time to the real time of receipt.
			\end{description}
			\input{broadcast-handling-scatter}
			Simulations for each of the 3 broadcast handling methods were run for a set period of time. For the interval up to this time, nodes are stimulated, adding tasks for them to perform, such as querying another node or pushing data into the network. After the allowed simulation time, no more stimulations occur and the remaining tasks are allowed to complete. The interval from stopping stimulations to the end of the simulation is referred to as the simulation lag. More detail on the simulator can be found in appendix \ref{simulator}. It should be noted that time, in this context, is measured in simulation ticks, not the real time the simulation takes to complete. Small simulation lag values are what is expected to result from a sustainable system where tasks do not accumulate.
			
			Simulation results are not absolute, and real scalability and performance results will be dependant on real network usage. The simulations presented here, subject to minor variation in the random nature of simulating usage, have been conducted under the same usage conditions.
			
			Running the simulation for UP for a set time, while varying the size of the network results in the graph shown in figure \ref{fig:simulation-lag}. The point where the method fails for a simulation interval of $150\cdot10^6$ ticks, can be seen to occur at 80 nodes. For UP, the points of failure for different simulation times were plotted, as shown in figure \ref{fig:critical-point}. This downward trend indicates that the number of nodes needed for network failure decreases the longer the network is active.
			
			Simulations for SP and AF were also conducted, but points of failure could not be determined.
			
			The simulations were run for $300\cdot10^6$ ticks, to achieve near steady-state behaviour in the network. The bandwidth used over each of the simulations was compared, as shown in figure \ref{fig:bandwidth-required}. A possible reason for AF having worse performance than SP is that when multiple nodes forward the same packet to the same node, a race condition is effectively created (figure \ref{fig:activefail}).
			
			While optimisations for AF can prevent this race condition by informing any receiving nodes of the forwarded packet ID before the packet contents and allowing only one incoming connection for each unique packet, this has the potential to open up an attack vector for a denial of service where a malicious node could easily forward a genuine packet at a very slow speed, therefore allowing an attacker to potentially deny a node of new packets.
			
			\begin{figure}
				\centering
				\begin{tikzpicture}[auto, thick]
				\tikzstyle{vertex}=[circle,fill=black!25,minimum size=17pt,inner sep=0pt]
				\node[vertex] (Node-1) at (-2,0) {1};
				\node[vertex] (Node-2) at (0,0.75) {2};
				\node[vertex] (Node-3) at (0,-0.75) {3};
				\node[vertex] (Node-4) at (2,1.5) {4};
				\node[vertex] (Node-5) at (2,0) {5};
				\node[vertex] (Node-6) at (2,-1.5) {6};
				
				\draw[->, thick, solid, black] (-3,0) -- (Node-1);
				
				\draw[->, thick, solid, black] (Node-1) -- (Node-2);
				\draw[->, thick, solid, black] (Node-1) -- (Node-3);
				
				\draw[->, thick, solid, black] (Node-2) -- (Node-4);
				\draw[-), thick, solid, red] (Node-2) -- (Node-5);
				
				\draw[->, thick, solid, black] (Node-3) -- (Node-6);
				\draw[-), thick, solid, red] (Node-3) -- (Node-5);
				
				\node[text width=5cm] at (6,0) {Node 1 receives a packet, which it forwards to nodes 2 and 3. These both forward to node 5, creating a race condition that wastes bandwidth.};
				
				%\draw[-latex, thick, dashed, black] (Node-2) -- (Node-1);
				\end{tikzpicture}	
				
				\caption{Active forwarding wasted bandwidth caused by race condition}
				\label{fig:activefail}
				%\end{center}
			\end{figure}
			
			SP has the potential to require the least bandwidth for broadcasts. AF may also be useful to support for the purpose of initial packet distribution, to both increase availability faster and to increase anonymity by hiding the original source of packets.