ICS-Web-Crawler

Web Crawler For UCI ICS Websites

Capabilities

• Find unique pages on the UC Irvine School of Computer Science server.
• Find the longest page in terms of word count
• Ignore common stop words such as a, the
• Find a list of top 50 most common words
• Count subdomains under ics.uci.edu

Technologies

• Python
• BeautifulSoup

Stats

Subdomains crawled:

*.ics.uci.edu/*
*.cs.uci.edu/*
*.informatics.uci.edu/*
*.stat.uci.edu/*

Unique pages found:

9167

Longest page in terms of word counts:

http://www.ics.uci.edu/~kay/wordlist.txt

Word Count: 385,020

A list of 50 most common words, in word = count format:

1. research = 37343
2. 2022 = 24874
3. computer = 24799
4. information = 20736
5. ramesh = 19455
6. students = 18664
7. 2018 = 17929
8. software = 17375
9. 2021 = 17265
10. student = 16591
11. events = 16303
12. 2020 = 16109
13. uci = 15497
14. 2019 = 15304
15. news = 15230
16. data = 15040
17. science = 14280
18. 2016 = 14030
19. 2017 = 13694
20. may = 13603
21. september = 13573
22. current = 13418
23. graduate = 13318
24. engineering = 13065
25. october = 12732
26. 2015 = 12730
27. past = 12566
28. informatics = 12417
29. undergraduate = 12288
30. courses = 12154
31. design = 12039
32. search = 11707
33. books = 11391
34. 10 = 11048
35. can = 11015
36. people = 10916
37. projects = 10881
38. computing = 10600
39. june = 10418
40. jain = 10384
41. july = 10285
42. department = 9993
43. ics = 9910
44. read = 9890
45. new = 9750
46. november = 9680
47. ph = 9620
48. march = 9590
49. us = 9583
50. time = 9490

A list of ics.uci.edu subdomain, in URL, count format:

• accessibility.ics.uci.edu, 1
• acoi.ics.uci.edu, 74
• aiclub.ics.uci.edu, 1
• archive.ics.uci.edu, 6
• asterix.ics.uci.edu, 6
• cbcl.ics.uci.edu, 3
• cert.ics.uci.edu, 3
• chenli.ics.uci.edu, 3
• cml.ics.uci.edu, 185
• code.ics.uci.edu, 13
• computableplant.ics.uci.edu, 31
• cradl.ics.uci.edu, 20
• create.ics.uci.edu, 7
• cwicsocal18.ics.uci.edu, 12
• cyberclub.ics.uci.edu, 2
• dejavu.ics.uci.edu, 1
• dgillen.ics.uci.edu, 25
• duttgroup.ics.uci.edu, 114
• dynamo.ics.uci.edu, 1
• elms.ics.uci.edu, 1
• emj.ics.uci.edu, 45
• esl.ics.uci.edu, 1
• evoke.ics.uci.edu, 8
• flamingo.ics.uci.edu, 10
• fr.ics.uci.edu, 3
• frost.ics.uci.edu, 1
• grape.ics.uci.edu, 74
• graphics.ics.uci.edu, 4
• graphmod.ics.uci.edu, 1
• hack.ics.uci.edu, 2
• hai.ics.uci.edu, 3
• hobbes.ics.uci.edu, 1
• hpi.ics.uci.edu, 3
• i-sensorium.ics.uci.edu, 1
• iasl.ics.uci.edu, 19
• informatics.ics.uci.edu, 2
• intranet.ics.uci.edu, 1
• ipf.ics.uci.edu, 2
• ipubmed.ics.uci.edu, 1
• isg.ics.uci.edu, 188
• jgarcia.ics.uci.edu, 24
• keys.ics.uci.edu, 1
• luci.ics.uci.edu, 4
• malek.ics.uci.edu, 1
• mcs.ics.uci.edu, 83
• mds.ics.uci.edu, 19
• mhcid.ics.uci.edu, 18
• mondego.ics.uci.edu, 3
• mse.ics.uci.edu, 2
• mswe.ics.uci.edu, 14
• nalini.ics.uci.edu, 7
• ngs.ics.uci.edu, 3034
• perennialpolycultures.ics.uci.edu, 1
• plrg.ics.uci.edu, 84
• redmiles.ics.uci.edu, 4
• riscit.ics.uci.edu, 1
• sdcl.ics.uci.edu, 209
• seal.ics.uci.edu, 6
• sherlock.ics.uci.edu, 1
• sli.ics.uci.edu, 338
• sourcerer.ics.uci.edu, 1
• stairs.ics.uci.edu, 1
• student-council.ics.uci.edu, 1
• studentcouncil.ics.uci.edu, 3
• support.ics.uci.edu, 1
• swiki.ics.uci.edu, 1
• tad.ics.uci.edu, 1
• tastier.ics.uci.edu, 1
• tippersweb.ics.uci.edu, 4
• transformativeplay.ics.uci.edu, 59
• tutors.ics.uci.edu, 1
• unite.ics.uci.edu, 10
• vision.ics.uci.edu, 7
• wearablegames.ics.uci.edu, 11
• wics.ics.uci.edu, 1038
• wiki.ics.uci.edu, 1
• www-db.ics.uci.edu, 12
• xtune.ics.uci.edu, 6

Contributors

• Shengyuan Lu
• Qinyu Chen