-
Notifications
You must be signed in to change notification settings - Fork 1
/
8.19.2015, 8.47.54 PM Twine Archive.html
184 lines (156 loc) · 9.24 KB
/
8.19.2015, 8.47.54 PM Twine Archive.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
<tw-storydata name="Understanding Web Archiving" startnode="6" creator="Twine" creator-version="2.0.8" ifid="3F08D6C6-4B11-419C-BAE2-FDA19CBED39D" format="Harlowe" options=""><style role="stylesheet" id="twine-user-stylesheet" type="text/twine-css">.autobui {
display: block;
margin: 0.5em;
padding: 0.5em 1em;
border: 1px dotted gray;
background: #eee;
}
.site {
background: aliceblue;
display: block;
font-family: sans-serif;
padding: 1.5em;
}
tw-passage > * {
/*display: inline-block;*/
}
tw-expression {
/* display: inline;*/
}</style><script role="script" id="twine-user-script" type="text/twine-javascript">
</script><tw-passagedata pid="1" name="Actions" tags="" position="625,500"><h3>Welcome to the WARC-9000</h3>
(if: (history:) contains "Actions")[(either: "Good work.","Keep it up.","Good to see you again.","Oh, it's you.","Back so soon?","I want to help you.","Everything is going extremely well.","I am completely operational, and all my circuits are functioning perfectly.")
What do you want to do next?
](else:)[
*Instead of your usual home page, the browser brings up some kind of web archiving user interface. The system presents you with three options.*
How can I help you?
]
* [[Crawl]]
* [[Replay]]
* [[Preserve|Preserve2]]
</tw-passagedata><tw-passagedata pid="2" name="Crawl" tags="" position="475,500"><h3>Crawl</h3>
The screen fills with text, describing [[what it means to crawl a web page, and the tools you can use to do it|Understanding Crawling]].
(if: (history:) contains "Crawl")[You notice that the WARC-9000 is now displaying information about the crawling process at the bottom of the browser window. It says:
<span class="autobui">**So far, you have crawled (print: $ctot) resource(s).**
(link:"Start Crawling")[(set: $mode to "crawl")(goto:"Homepage")]</span>
[[Do something else|Actions]]
]</tw-passagedata><tw-passagedata pid="3" name="Index" tags="" position="800,375"><h3>Generate CDX</h3>
(if: $ctot is 0)[You have not crawled anything yet!](else:)[
Indexing (print: $ctot) resources...{(live:2s)[ DONE.
<br/>
<br/>
[[Back|Replay]]
(stop:)]}
(set: $itot to $ctot)
]</tw-passagedata><tw-passagedata pid="4" name="Replay" tags="" position="775,525"><h3>Replay</h3>
(if: $ctot is 0)[You've not crawled anything yet!
[[Now what?|Actions]]](else:)[{
(if: $itot is not 0)[
<span class="autobui">You have crawled (print: $ctot) resource(s), and (print: $itot) of those have been indexed for replay.
(link:"Replay Crawled Site")[(set: $mode to "replay")(goto:"Homepage")]</span>
](else:)[
When replaying an archived web site, we need to be able to take the web requests coming from the browser and work our which of the resources we've collected each request corresponds to. This means we need map that takes individual resource URLs and indicates where we should look in the WARC files. This is a type of indexing, where we process each WARC file and generate a content index or <a target="_blank" href="http://archive.org/web/researcher/cdx_file_format.php">CDX file</a>.
<span class="autobui">[[Generate CDX...|Index]]</span>
]
[[Do something else|Actions]]
}
]
</tw-passagedata><tw-passagedata pid="5" name="Homepage" tags="site" position="625,675">(if: $mode is "crawl" or $visited_home_page is true)[
<span class="site">**Welcome to SCCorp!**
From here you can:
* [[Look at something exciting|Static Page]]
* [[Look at something dynamic|Dynamic Page]]
* [[Sign our guestbook|Interactive Page]]
</span>
](else:)[
(display: "Resource Not In Archive")
]
{
<span class="autobui">
(if: $mode is "replay")[[[Stop Replaying|Replay]]](else:)[[[Stop Crawling|Crawl]]
(set: $ctot to $ctot + 1)
<br/>
<small>You have crawled (print: $ctot) resource(s) so far.</small>
(set: $visited_home_page to true)
]
</span>
}
</tw-passagedata><tw-passagedata pid="6" name="Welcome" tags="intro" position="375,300"><h2>Understanding Web Archiving:<br/><small>An interactive introduction to how web archiving works.</small></h2>We'll find out how web content is archived, preserved, and made accessible. Along the way, we'll introduce the major concepts and de-mystify the technical jargon.
Interested? Then let's [[Get Started...|The Scene]]
(set: $ctot to 0)
(set: $itot to 0)</tw-passagedata><tw-passagedata pid="7" name="Static Page" tags="site" position="475,800">(if: $mode is "crawl" or $visited_static_page is true)[
<span class="site">[[Homepage]] > Something Exciting
This is a great page.
</span>
](else:)[
(display: "Resource Not In Archive")
]
{
<span class="autobui">
(if: $mode is "replay")[[[Stop Replaying|Replay]]](else:)[[[Stop Crawling|Crawl]]
(set: $ctot to $ctot + 1)
<br/>
<small>You have crawled (print: $ctot) resource(s) so far.</small>
(set: $visited_static_page to true)
]
</span>
}
</tw-passagedata><tw-passagedata pid="8" name="Dynamic Page" tags="site" position="625,800">(if: $mode is "crawl" or $visited_dynamic_page is true)[
<span class="site">[[Homepage]] > Something Dynamic
This is a great page.
</span>
](else:)[
(display: "Resource Not In Archive")
]
{
<span class="autobui">
(if: $mode is "replay")[[[Stop Replaying|Replay]]](else:)[[[Stop Crawling|Crawl]]
(set: $ctot to $ctot + 1)
<br/>
<small>You have crawled (print: $ctot) resource(s) so far.</small>
(set: $visited_dynamic_page to true)
]
</span>
}
</tw-passagedata><tw-passagedata pid="9" name="Interactive Page" tags="site" position="775,800">(if: $mode is "crawl" or $visited_interactive_page is true)[
<span class="site">[[Homepage]] > Guestbook
This is a great page.
</span>
](else:)[
(display: "Resource Not In Archive")
]
{
<span class="autobui">
(if: $mode is "replay")[[[Stop Replaying|Replay]]](else:)[[[Stop Crawling|Crawl]]
(set: $ctot to $ctot + 1)
<br/>
<small>You have crawled (print: $ctot) resource(s) so far.</small>
(set: $visited_interactive_page to true)
]
</span>
}
</tw-passagedata><tw-passagedata pid="10" name="Resource Not In Archive" tags="" position="925,500"><h3>Resource Not In Archive!</h3>
Oh dear, it looks like you didn't manage to capture this page during the crawl.
</tw-passagedata><tw-passagedata pid="11" name="Preserve2" tags="" position="700,225"><h3>Preserve</h3>
(if: $ctot is 0)[You have not crawled anything yet!
At least it's cheap to store.
[[Now what?|Actions]]](else:)[You have:
(if: $itot is 0)[
* (print: $ctot) crawled resource(s) stored in WARC format.
](else:)[
* (print: $ctot) crawled resource(s) stored in WARC format.
* One CDX file built by indexing (print: $itot) resource(s).
]
But how will you preserve what you've got?
* [[Know Your Data]]
* [[Perform Fixity Checks]]
* [[Scan For Access Risks]]
[[Do something else|Actions]]
]
</tw-passagedata><tw-passagedata pid="12" name="Know Your Data" tags="" position="950,325">So what's in a WARC?
For more:
* <a target="_blank" href="http://fileformats.archiveteam.org/wiki/WARC">WARC on the file format wiki</a>
* <a target="_blank" href="http://en.wikipedia.org/wiki/Web_ARChive">WARC on Wikipedia</a></tw-passagedata><tw-passagedata pid="13" name="Perform Fixity Checks" tags="" position="995,198">Are our WARCs still intact? Undamaged by bad luck, bit rot, or poor handling? How would we know?
<a target="_blank" href="http://en.wikipedia.org/wiki/File_Fixity">File Fixity on Wikipedia</a></tw-passagedata><tw-passagedata pid="14" name="Scan For Access Risks" tags="" position="950,75">Double-click this passage to edit it.</tw-passagedata><tw-passagedata pid="15" name="Understanding Crawling" tags="" position="350,500">However, your organisation has decided against using an automated crawler. Instead, as you browse the site, your computer will record the conversation between the browser and the web server and built up the archive as you go. This approach means you'll get a high-quality archive, but also means you'll have to go through the pages of the web site by hand.
[[Crawl]]</tw-passagedata><tw-passagedata pid="16" name="The Scene" tags="intro" position="525,300">You are in a maze of open-plan office cubicles, all alike. Your boss has told you that you are now responsible for archiving your organisation's web site. You oozed confidence as you accepted the role. After all, how hard can it be? And anyway, sometimes the best way to learn is through on-the-job experience.
You sit at your desk, turn to the computer, and hunt around the desktop and menus as you try to work out where to start. Eventually, you give up, and [[open up the web browser|Actions]] in order to do some research, after you've checked Facebook, obviously...
</tw-passagedata></tw-storydata>