web-app.nw

\chapter{Descot Web Application}%
The Descot Web Application provides a web-based interface for accessing 
the Descot System for search, browsing, and submission.  It is designed 
to permit easy use for those who do not wish to have a special client 
that integrates with their system. 

\section{Pages}%
There are two main dynamic pages defined for Descot at the moment: 
Browsing and Searching. Browsing includes the display of normal 
library descriptions and information in HTML form.  Submitting is 
also handled dynamically, so we should take care of that here as 
well.

<<Pages>>=
<<Browsing>>

<<Searching>>

<<Submitting>>
@

\noindent Pages that should be added are submissions and RDF Library Retreival, 
where each library is exported with its actual properties.  These 
are a major TODO.

Browsing includes displaying the library, showing a navigatible 
category list, and showing the libraries in a given category. 
We give three separate page procedures to handle each of these 
functions individually.

<<Browsing>>=
<<Library>>

<<Category Browse>>

<<Category List>>
@

\noindent The simplest library page to do, and the most used, is the library 
rendering page, which will simply display the results of querying 
for a specific library URL.

<<Library>>=
(define (library-page url)
	(descot-wrapper "Browse" descot-stylesheet (library->html (current-store) url)))
@

<<Imports>>=
(rnrs base)
(arcfide descot web generators)
(arcfide descot web parameters)
(arcfide descot rdf rdf-to-sxml)
@

\noindent [Notice the use of [[descot-wrapper]] to help everything go along.]

Next, we use our iterators and the like to get the listing of 
libraries for each category. We always try to make sure that category 
names are capitalized when they are displayed, but lower-cased when 
they are stored. 

<<Category Browse>>=
(define (category-page category)
	(descot-wrapper (string-append "Category " (capitalize-string category))
		descot-stylesheet
		(collect-list (for lib-name (in-list (libraries-in-category category)))
			(library->short-html (string-append descot-browse-path "?id=")
				(current-store) 
				lib-name))))
@

<<Imports>>=
(arcfide descot web utilities)
(riastradh foof-loop)
@

\noindent Here we do have a little bit of a hack since we know the browsing 
interface, and we pass a parameter ``?id='' explicitly.  This will be 
okay, and I want to avoid over engineering something before it even 
gets off the ground. 

When someone first goes to the browse link, if they do not enter any 
kind of query parameter in the URL, they will be shown a list of the 
available package categories, on which they can click and navigate. 
We use [[make-column-table-with-size]] to layout the table for us, 
expecting that it will lay them out in downward alphabetical order 
according to the columns we specify.  This column width is set 
externally to help with different layouts.

<<Category List>>=
(define (browse-page)
	(descot-wrapper "Browse" descot-stylesheet
		(let ([cats (store-categories (current-store))])
			(if (null? cats)
				`(p "No Categories are available.")
				(make-column-table-with-size 
					(browser-column-width)
					(categories->links cats))))))
@

<<Imports>>=
(arcfide descot rdf utilities)
@

\noindent Now, we progress to searching, which isn't nearly so complicated 
in its overall layout, even though it does use some procedures in 
the back to hide some complexity. Overall, there is the basic 
search results page, and the Advanced Search page, which is the 
landing page for search.

<<Searching>>=
<<Search Results>>

<<Advanced Search>>
@

\noindent Advanced search is not yet implemented, 
so we just use a place holder there. 

<<Advanced Search>>=
(define (advanced-search-page)
	(descot-wrapper "Advanced Search" descot-stylesheet
		`(p "Sorry, advanced search is not yet implemented.")))
@

\noindent To implement the search results, we have to take the [[match?]] 
procedure that we are given and use it on every single library 
that we can find.  We expect the [[match?]] procedure to be of the kind 
that we can apply just to a library node/ID. 

We use [[libary->short-html]] to print our results in a more succinct 
fashion from the main library browsing page.

<<Search Results>>=
(define (search-page match?)
	(define (search)
		(collect-list 
			(for name (in-list (matching-libraries match? (library-ids (current-store)))))
			(library->short-html 
				(string-append descot-browse-path "?id=")
				(current-store) 
				name)))
	(descot-wrapper "Search Results" descot-stylesheet
		(let ([results (search)])
			(if (null? results)
				'(p "Sorry, no results found.")
				results))))
@

\section{Library Submission}%
When someone wishes to submit a new library, at the moment, we will 
allow them to do so without much help. Right now, the submission on 
the DWA is merely a form with fields that are not checked or anything. 
The submissions will then be saved in a special folder that is given 
by the web parameters module. We will use gensym printing for handling 
the unique names for each submission. However, normal [[write]] 
form gensyms come out with funny characters in them, so we will 
need to convert them to something nicer:

<<Submitting>>=
(define (format-gensym id)
	(define cleaner (make-char-quotator '((#\space . ":"))))
	(let ([id-string (with-output-to-string (lambda () (write id)))])
		(apply string-append 
			(cleaner (substring id-string 2 (1- (string-length id-string)))))))
@

<<Imports>>=
(oleg util)
(only (scheme) with-output-to-string)
(rnrs io simple)
@

\noindent [[submit-page]] will return out the basic form used to accept submissions.

<<Submitting>>=
(define (submit-page)
	(descot-wrapper "Submit a New Library" descot-stylesheet
		`(form 
			(@ 
				(method "POST") 
				(enctype "multipart/form-data") 
				(action ,descot-submit-path))
	 		(p 
				(input (@ (name "code") (type "file")))
				(input (@ (name "file-type") (type "radio") (value "srdf") (checked "checked")))
				"SRDF"
				(input (@ (name "file-type") (type "radio") (value "turtle")))
				"Turtle")
	 		(p (button (@ (type "submit")) "Submit Library")))))

@

\noindent On a successful submission, we need to let them know they succeeded.

<<Submitting>>=
(define (successful-submit-page)
	(descot-wrapper "Success!" descot-stylesheet
		'(p 
			"Thank you for submitting a library to Descot and helping "
			"us to grow! At the moment, submissions are not updated "
			"automatically in the database; they must first be reviewed "
			"and if they are approved, they will show up in the database "
			"within a couple of days [hopefully].  Please bear with us "
			"as we continue to improve Descot!")))

@

\noindent On failure, we should print out an error, and if debugging is enabled,
we should print out the error message.

<<Submitting>>=
(define (submit-failure-page name fmts . vals)
	(descot-wrapper "Failure!" descot-stylesheet
		`(p 
			"An error occured when submitting your library. "
			"It is possible that the Library is not formatted "
			"correctly, or that there was a server error. "
			"Please examine your library for errors and try again. "
			"If this error persists, please contact "
			,(descot-maintainer)
			".")))

@

\noindent The maintainer comes from the web parameters [[descot-maintainer-email]] 
and [[descot-maintainer-name]].

<<Submitting>>=
(define (descot-maintainer)
	`(a (@ (href ,descot-maintainer-email)) ,descot-maintainer-name))
@

\noindent See the next section for information about how we actually handle the 
submission when it comes in.

\section{Page Handlers}%
Page handlers in MIT Scheme mod\_lisp handle the urls and mime-types that 
are requested from the server. Here we use mostly URL based dispatch 
with default mime types.  Generally, other things should be served 
statically by the Apache Web Server if it can for various reasons.  

There are really only two search URLs defined, the browse and the 
search path.  Each one is expected to take some parameters to 
help determine what page should be returned. 

Additionally, we have a general RDF handler for those URLs that should 
return some kind of Turtle representation of the sentences containing 
that URL as the root.

<<Handlers>>=
<<Browse Handler>>

<<Search Handler>>

<<RDF Handler>>

<<Mirror Handler>>

<<Submission Handler>>
@

\noindent For the browse URL the two mutually exclusive parameter [[id]] 
and [[cat]] tell us whether we should be displaying a library 
or a category listing.  If neither of those are used, then we 
use the basic category listing. 

<<Browse Handler>>=
(register-request-handler! (modlisp-subtree-matcher descot-browse-path)
  (lambda (req)
		(let (
				[id (get-modlisp-parameter 'id req)]
				[cat (get-modlisp-parameter 'cat req)])
			(make-html-response
				(write-html-page
					(cond
						[id => library-page]
						[cat => category-page]
						[else (browse-page)]))))))
@

<<Imports>>=
(arcfide modlisp)
@

\noindent For search, we use the Advanced Search page as our landing page, 
or, if we are given a query, we try to list the results. 

<<Search Handler>>=
(register-request-handler! (modlisp-subtree-matcher descot-search-path)
	(lambda (req)
		(let ([query (get-modlisp-parameter 'q req)])
			(make-html-response
				(write-html-page
					(cond
						[query (search-page (make-search-proc query))]
						[else (advanced-search-page)]))))))
@

\section{Serving RDF Sentences for a Library}%
When a user tries to retrieve a URL which corresponds to the 
URL of a given RDF entity, they should be presented with the 
set of RDF sentences that make up that library.  If they want 
this information in another form, they can use the browse 
pages to get that.  For now, this is the paradigm I want to use, 
but this may change, since technically it is possible to serve 
the HTML with the RDF depending on what the client requests. 

At any rate, right now, the easiest way to do this is to 
take the url request and reconsitute the entire URL from the 
pieces in mod\_lisp, and check to see whether such a thing 
exists as a subject in the current store.  If it does, then 
we just print out all the sentences that have it as a subject. 

We need to get around some other ``features'' of Apache and 
mod\_lisp in order to serve all the right URLs, since we 
can't assign the lisp-handler to take everything and then 
leave out one specific file set; root lisp-handlers don't 
seem to be working the way they should.  However, since 
all of our RDF files are in libs/, impls/, \&c., we can 
workaround this problem by explicitly listing these in the 
httpd.conf file. 

This won't make a difference in our default handler here, 
except that it will never have to serve static content. 

<<RDF Handler>>=
(register-request-handler! (modlisp-subtree-matcher descot-rdf-path)
	(lambda (req)
		(make-typed-response 
			(make-mime-type 'text 'plain '())
			(with-output-to-string 
				(lambda ()
					(write-descot-request
						(string-append "http://" 
							(get-modlisp-header req 'host "") 
							(let ([res (get-modlisp-header req 'url #f)])
								(if res (format "~{~a~^/~}" (uri-path res)) "")))))))))
@

<<Imports>>=
(arcfide descot server)
(arcfide mime)
(riastradh schemantic-web uri)
@

It should be noted that RDF URLs coming in with fragments in them 
must be encoded.  That is, the \# character in the URL must be 
encoded using the \%23 string, or else the URL will come through 
without the fragment.  This appears to be an artifact of mod\_lisp, 
but I have not confirmed this yet.

\section{Serving Updates/Mirroring}%
Descot servers provide a mirroring mechanism. Using the API, we define a handler 
here.

<<Mirror Handler>>=
(register-request-handler! (modlisp-subtree-matcher descot-mirror-path)
	(lambda (req)
		(make-typed-response
			(make-mime-type 'text 'plain '())
			(with-output-to-string
				(lambda ()
					(write-descot-updates (current-output-port)))))))
@


\section{Accepting Submissions}%
When handling Submissions, we will grab the post parameters and 
check if they are null. If it isn't, then we have a submission and 
we can go ahead and write out the library.  Otherwise, we'll just 
display the main submit landing page.

<<Imports>>=
(oleg sxml-to-html)
(only (srfi :13) string-index)
(riastradh parscheme partext)
@

When handling submissions, we have to take care of the mime contents, which 
requires us to parse the [[Content-Disposition]] header, and the following Utility 
helps to handle that.

<<Utilities>>=
(define (parse-mime-parameter-list x)
	(let ([start (string-index x #\;)])
		(if start
			(parse-string mime-parser:parameter-list
				(substring x start (string-length x))
				#f (lambda (g c s) g) 
				(lambda (e c s) 
					(error 'parse-mime-parameter-list
						"Failed to parse parameter list" e)))
			'())))
@

When we have received a file to put in the submission queue, we want to 
put the actual file in one place, and associate a time stamp with it, which 
we do with the two following chunks, assuming that our generic location 
is [[fname]].

<<Write Received File>>=
(let (
		[port
			(open-file-output-port fname 
				(file-options no-fail) 
				(buffer-mode block)
				modlisp-transcoder)]
		[data (mime-entity-contents code)])
	((cond [(string? data) put-string] [(bytevector? data) put-bytevector]
			[else (error 'submit-code "Unknown entity content type" data)])
		port
		data)
	(close-port port))
@

<<Write Timestamp>>=
(call-with-output-file (string-append fname ".timestamp")
	(lambda (port) 
		(put-string port (date-and-time))
		(put-char port #\newline)
		(put-string port (get-modlisp-header req 'remote-ip-addr "Unknown"))
		(put-char port #\newline)
		(put-datum port (mime-entity-fields code))
		(put-char port #\newline)))
@

If we cannot tell that we have received a file for submission, the default is to display 
the submission form page. We assume that we have a file submission if we 
have a [[multipart/form-data]] [[Content-Type]] as well as have a submitted 
element with the name [[code]].

<<Submission Handler>>=
(register-request-handler! (modlisp-subtree-matcher descot-submit-path)
	(lambda (req)
		(define (success)
			(make-html-response (write-html-page (successful-submit-page))))
		(define (form-page)
			(make-html-response (write-html-page (submit-page))))
		(define (code-element? x)
			(let ([disp (assq 'content-disposition (mime-entity-fields x))])
				(and disp
					(let ([res (assq 'name (parse-mime-parameter-list (cdr disp)))])
						(and res (string-ci=? "code" (cdr res)))))))
		(let (
				[t (get-modlisp-header req 'content-type (make-mime-type #f #f '()))]
				[contents (modlisp-packet-contents req)])
			(if (and contents (eq? 'multipart (mime-type-main t)) (eq? 'form-data (mime-type-subtype t)))
				(let (
						[code (find code-element? contents)]
						[fname (format "~a~a" descot-submit-root (format-gensym (gensym "submission")))])
					(if code
						(begin 
							<<Write Received File>>
							<<Write Timestamp>>
							(success))
						(form-page)))
				(form-page)))))

@

\section{Optimization/Initialization/Running}

DWA is configured with standard unsafe optimizations, and the entire 
program is run from one nested LET expression. All definitions must 
come before the handlers, since the handlers are actually expressions 
rather than definitions.

<<web-app.ss>>=
<<Web App License>>

(let ()

(import 
	<<Imports>>)

<<Utilities>>

<<Pages>>

(define (initialize-server)
	<<Initialization>>)

<<Handlers>>

(scheme-start 
	(lambda fns 
		(initialize-server)))

)

@

We need to make sure that we initialize the server api the way we 
want. The default SRDF format will be used for the store, but 
we specify the store location from our parameters:

<<Initialization>>=
(descot-store descot-store-root)
@

At the moment, we read in the entire store into memory for simplicity. 
this makes our accessor functions much easier to write, so we assume 
this. However, this could have ramifications in the future if the size 
of the stores grows too large.  Instead of doing this, it would be 
better if the stores were grabbed directly from the filesystem when 
they were needed, assuming we have a heavy memory constraint.  On 
the other hand, I have an inkling thought that this won't matter on 
today's high memory systems. At least, I hope not.

<<Initialization>>=
(current-store (read-descot-store (descot-store)))
@

Now we can safely start the mod\_lisp server.

<<Initialization>>=
(start-modlisp-server 3000)
@

Right now, the server is meant to be launched using something like 
nohup(1) with backgrounding so that the process can be thrown into the 
background and it will run with output to some file. 

\section{Licensing}

Descot's Web Application is licensed under an ISC License.

<<Web App License>>=
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; Web Application Component of Descot
;;; 
;;; Copyright (c) 2009 Aaron Hsu <arcfide@sacrideo.us>
;;; 
;;; Permission to use, copy, modify, and distribute this software for
;;; any purpose with or without fee is hereby granted, provided that the
;;; above copyright notice and this permission notice appear in all
;;; copies.
;;; 
;;; THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
;;; WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
;;; WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE
;;; AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL
;;; DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA
;;; OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
;;; TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
;;; PERFORMANCE OF THIS SOFTWARE.