UPDATE (February 4, 2024): This is the discussion about this project on HN: here. Please specifically read @dang's comment regarding the core assumption of this project: here. On a personal note, the number of Stories removed yesterday (Saturday, February 3, 2024) was the lowest ever recorded by the service. This includes 2 duplicate Stories. As a side note, in the list always check whether a Story is a duplicate or not: this is a very reasonable reason for removal and unfortunately I have no way of automatically determining it in the service!
The purpose of this project is to try to understand the type and scale of the moderation of the Hacker News Front Page.
NOTE: I love Hacker News. I try to read it every day. In the case of OnnxStream (here for example), 95% of the comments were helpful and intelligent. I also understand that moderating a site with huge traffic and where users are basically anonymous must be a very difficult task.
Returning to the purpose of this project, from what I have been able to see, the "public" (i.e. observable from the outside) moderation of the Front Page consists of two main tools: modification of the title of a Story (voluntarily or involuntarily influencing its growth in terms of rank) or directly its removal.
Regarding the first type of moderation, an excellent site is already available that tracks changes to Story titles. Here instead I will focus on the second type.
For the reasons explained in the "Why?" section below, I have developed a small application that logs all the Stories that are removed from the Front Page, for personal use. I later discovered that there is no tool/website that provides this type of information and I decided to make it public here. It was a difficult decision but my rationale is: is it better to have more transparency or less transparency?
If you know of a tool/website similar to this, please let me know: I will archive this repo or set it to private.
A possible very positive outcome for this project could be to have a list similar to this, but available directly among the HN lists. Or even to notify a user when a Story is penalized on the Front Page, perhaps indicating the number of flags and/or the reason, for example.
Feel free to skip this part or click to expand
A friend of mine posted two Stories on Hacker News related to OnnxStream (31 days apart), the first related to SDXL Turbo support and the second related to TinyLlama and Mistral 7B support.
In the case of the first, the Story was among the first on the Front Page, until its title was changed from "Stable Diffusion Turbo on a Raspberry Pi Zero 2 generates an image in 29 minutes" to "OnnxStream: Stable Diffusion XL 1.0 Base on a Raspberry Pi Zero 2". This effectively "killed" the Story. One user pointed out that the new title didn't reflect the spirit of the Story (thanks @practice9).
In the case of the second, the Story was in third place on the Front Page, less than an hour after the submission. In this case it was simply removed from the Front Page.
Having discovered this, perplexed, I sent an email to the moderator. @dang, who was very kind and quick in his response, explained to me that the Story had been flagged by users even without being explicitly [flagged], and that he could therefore only hypothesize the causes of the flag. His hypothesis was that (some?) users might be fed up with news related to LLMs.
While I have no reason to doubt Daniel's good faith, it's hard to believe that HN users would be tired of LLM-related news.
So I decided to develop a small console application to determine the frequency of this phenomenon (actually I was also motivated by the prospect of writing some C# code, after more than 2 years of complete abstinence). I subsequently discovered that there were no tools/websites that monitored this specific phenomenon and I therefore decided to make it public here.
Using the official HN API, the service fetches 90 Top Stories every minute and makes a comparison with the first 30 Top Stories (i.e. the Front Page) fetched the previous minute. It logs all missing Stories here. The assumption is that a Story cannot go from the top 30 to a position greater than 90 in a single minute, without having been explicitly removed. If a Story reappears on the Front Page, it is removed from this log. All Stories present in the second-chance pool are excluded from the log. Title and URL are those from when the Story first appeared in the top 30. The number of points and comments and the rank are those from when the Story was removed from the Front Page. The ID points to the news.social-protocols.org page for that Story, which provides a graph of the Story's position on the Front Page over time.
NOTE: always check whether a Story is a duplicate or not: this is a very reasonable reason for removal and unfortunately I have no way of automatically determining it in the service!
- 47214708 #5 53 points 17 comments -> Math Notepad
- 47219462 #9 210 points 101 comments -> Open Camera is a FOSS Camera App for Android
- 47206990 #10 83 points 6 comments -> Apache Otava
- 47220411 #15 307 points 67 comments -> Payphone Go
- 47215112 #17 51 points 9 comments -> Triplet Superconductor
- 47284114 #13 42 points 9 comments -> AI and the Illegal War
- 47284834 #7 39 points 16 comments -> AI Error May Have Contributed to Girl's School Bombing in Iran
- 47285375 #2 25 points 11 comments -> German police probe student poster slur (Lick my balls Merz) against Merz
- 47284916 #23 34 points 40 comments -> Why New Zealand is seeing an exodus of over-30s
- 47285433 #27 9 points 1 comments -> Your AI Slop Bores Me
- 47286236 #13 15 points 17 comments -> Claude AI Helped Bomb Iran. But How Exactly?
- 47286300 #2 28 points 41 comments -> Show HN: Tanstaafl – Pay-to-inbox email on Bitcoin Lightning
- 47288046 #12 50 points 41 comments -> LLM Doesn't Write Correct Code. It Writes Plausible Code
- 47289123 #6 56 points 22 comments -> Tech jobs are getting demolished in ways not seen since 2008
- 47289914 #9 8 points 0 comments -> Track AIPAC – Follow Israel lobby spending in U.S. politics
- 47289933 #7 20 points 1 comments -> Footage shows US citizen shot dead by ICE agent in Texas traffic stop
- 47289869 #16 4 points 2 comments -> T3 Code is the best way to code with AI
- 47289646 #6 -> Show HN: Prompt Armour – Real-time PII detection for AI chatbots, 100% local
- 47290631 #8 5 points 0 comments -> The Prompt I Cannot Read – Written by an LLM, about Being an LLM
- 47291202 #15 -> Sam Altman's greed and dishonesty are finally catching up to him
- 47292849 #29 11 points 1 comments -> Patel gutted FBI counterintelligence team tasked with tracking Iranian threats
- 47292841 #17 24 points 31 comments -> Ghostmd: Ghostty but for Markdown Notes
- 47293514 #20 36 points 21 comments -> Iranian Women Graduate in Stem 3× the Rate of U.S. Women and Has 5× More PhDs
- 47294008 #22 32 points 29 comments -> Israel Strikes Oil Facilities in Iran
- 47294082 #19 8 points 2 comments -> Americans Are Now a Target for ICE
- 47296787 #10 16 points 17 comments -> You Don't Need a Vector Database
- 47261529 #18 5 points 1 comments -> We Turned Our Wireshark Wizard into a Markdown File
- 47300560 #10 7 points 1 comments -> Show HN: WhatsApp Clone – No Setup or Signup
- 47302494 #16 4 points 0 comments -> The Making and Unraveling of the American–Israeli Alliance
- 47239117 #19 117 points 62 comments -> Log messages are mostly for the people operating your software
- 47248801 #21 6 points 0 comments -> Some Lotto Math
- 47266919 #23 16 points 5 comments -> Are You Noticing This?
- 47306157 #2 55 points 13 comments -> Grammarly is using our identities without permission
- 47306692 #5 6 points 0 comments -> Every business will have AGI by 2027
- 47306447 #11 8 points 10 comments -> MiniMax Music 2.5 – AI Music Generation Model for Fast Song Creation
- 47307837 #30 6 points 3 comments -> Grok sparks outrage after chatbot makes offensive jibes about football disasters
- 47308468 #20 70 points 37 comments -> Owner of ICE detention facility sees big opportunity in AI man camps
- 47309655 #23 23 points 12 comments -> Global Stocks Sink as Oil Prices Surge
- 47309790 #15 9 points 0 comments -> India offered sanctuary to Iranian ship three days before US sank it
- 47311077 #21 71 points 31 comments -> US missile hit military base near Iran school, video analysis shows
- 47311798 #30 6 points 0 comments -> Video appears to show U.S. cruise missile striking Iranian school compound
- 47311745 #9 16 points 7 comments -> The 1979 Design Choice Breaking AI Workloads
- 47312946 #12 5 points 1 comments -> UK Cryptography Developer claims IP to protect UK National Security was stolen
- 47314036 #6 10 points 0 comments -> AI Didn't Break the Senior Engineer Pipeline. It Showed That One Never Existed
- 47313993 #29 8 points 1 comments -> Teenagers report for duty as Croatia reinstates conscription
- 47317115 #6 11 points 3 comments -> Helios: Real Real-Time Long Video Generation Model
- 47313613 #16 205 points 70 comments -> Workers report watching Ray-Ban Meta-shot footage of people using the bathroom
- 47316880 #1 210 points 50 comments -> In Memoriam, Tony Hoare
- 47321494 #19 7 points 2 comments -> OpenClaw Did Not Just Go Viral in China, It Solved a Structural Problem
- 47322002 #21 6 points 0 comments -> There are 56 ethnicities in China–and 55 are getting squashed
- 47323300 #9 3 points 2 comments -> How Do You Choose a Managed Security Service Provider (MSSP)?
- 47323828 #10 7 points 1 comments -> You Bought the AI Licenses. Why Is Only One Developer Getting 10x Results?
- 47323918 #22 10 points 3 comments -> Intensifying global heat threatens livability for younger and older adults
- 47321533 #13 396 points 214 comments -> Yann LeCun's AI startup raises $1B in Europe's largest ever seed round
- 47327143 #26 10 points 5 comments -> Maybe the G in AGI stands for Gemini
- 47327367 #30 35 points 2 comments -> DOGE member took Social Security data on a thumb drive, whistleblower alleges
- 47328122 #30 9 points 2 comments -> The Latest Republican Efforts to Make It Harder to Vote in the Midterms
- 47328532 #25 4 points 0 comments -> Air strikes cause black rain and 'unprecedented' pollution in Tehran
- 47331708 #11 13 points 1 comments -> Sergey Brin Funds Misleading Ballot Measure to Negate Wealth Tax
- 47332939 #1 29 points 41 comments -> Why does AI tell you to use Terminal so much?
- 47275743 #21 14 points 18 comments -> Docs directories are doomed
- 47290574 #26 12 points 7 comments -> CAP theorem – Partition is a verb
- 47291258 #12 7 points 2 comments -> Show HN: Custom Git-receive-pack implementation
- 47296163 #11 10 points 2 comments -> Show HN: I wrote down every expensive hardware development mistake I've seen
- 47338445 #18 23 points 20 comments -> Why Do They Want to Get Rid of Software Engineers?
- 47338960 #16 6 points 1 comments -> Medical technology company in Michigan hit by suspected Iran-linked cyberattack
- 47339888 #12 80 points 38 comments -> Hisense TVs force owners to watch intrusive ads
- 47339763 #30 11 points 2 comments -> Nvidia Releases NemoClaw – Enterprise AI Agents, Redefined
- 47343112 #15 15 points 7 comments -> NemoClaw – Nvidia's upcoming open-source AI agent platform
- 47346383 #15 26 points 8 comments -> US intelligence says Iran government is not at risk of collapse
- 47348573 #30 3 points 0 comments -> Independent Geophysical Forensic Analysis of the Nordstream Pipeline Sabotage
- 47350341 #25 7 points 0 comments -> At Least 17 U.S. Sites Damaged in War with Iran, Analysis Shows
- 47350252 #21 14 points 29 comments -> Show HN: I built an SDK that scrambles HTML so scrapers get garbage
- 47350467 #24 45 points 41 comments -> USDA is closing buildings, relocating staff, and downsizing-a lot
- 47352819 #20 42 points 12 comments -> Another DOGE staffer explaining how he flagged grants at NEH for "DEI"
- 47353293 #7 8 points 0 comments -> We will come to regret our every use of AI
- 47353981 #28 16 points 10 comments -> Show HN: LogClaw – Open-source AI SRE that auto-creates tickets from logs
- 47355213 #9 30 points 16 comments -> Show HN: Aurion OS – A 32-bit GUI operating system written from scratch in C
- 47357294 #3 33 points 11 comments -> Illinois introduces OS-level age verification law
- 47358033 #10 4 points 0 comments -> Qodo Outperforms Claude in Code Review Benchmark