Use Case for Generators: News Feeds Aggregation Using Generators #2557
Replies: 6 comments
|
While this material is useful, it's not an open issue as such, so I'm proposing to close it with no action. |
|
I actually think these use-cases demonstrate some very good real world examples of why generators are very useful (and powerful) |
Dr. Kay, I posted this here as an issue in order to have a centralized place for it, so that we can reference it. I would love any proposal for a better place for this and other use-cases that come in support for any current or future proposal. |
|
Closing this doesn't mean the information isn't useful, it means that it's not an issue that requires action. |
|
but closed issues easily get lost in the long tail of issues that are closed as action was done, elsewhere I've found gh discussions are quite good for this sort of thing, they seem not to be used here |
|
A good idea. I have converted this issue to a conversation. |
Uh oh!
There was an error while loading. Please reload this page.
In response to:
QT4CG-147-02: NW to chase up DN and LQ about follow-up to the generator discussionUse Case: News Feeds Aggregation Using Generators
Contents
Use Case: News Feeds Aggregation Using Generators
The Problem
Modern RSS/JSON aggregators must process hundreds of continuously updating feeds without excessive memory usage or latency, while supporting filtering, merging, and prioritization in real time.
Actors
Goals
End-User
“As a user, I want to get the latest, up-to-the-minute news from many important sources. I want each brief news item to be presented with a link to more detailed information from the original source.”
Editor
“As an editor, I want to be alerted to any change in the aggregated news-stream, as it happens continuously, and to have powerful ways of inserting, reordering, appending, prepending or deleting one or more news-items.”
Administrator
“As an administrator, I want to start, stop, or restart the system, manage the configured feeds, and monitor operational health and error conditions.”
Functional Requirements
Constraints / Assumptions / Preconditions
Assumptions
Constraint
Preconditions
Proposed High-Level Solution
Each feed is modeled as a generator producing yield values lazily.
The ordered set of values produced by successive, demand-driven calls to
move-next()is called the yield of the generator.A generator’s yield may be finite or infinite, and may be empty for a given generator instance without implying exhaustion of the underlying data source.
Known Approaches That Are Problematic
These approaches require full materialization in memory:
Benefits of the Generators Approach
End-to-End Flow
Remarks
G_{name}.Brief Description of the Core Processes in the Pipeline
Process 1 — Feed Fetching & Acquisition
Goal:
Continuously pull RSS / Atom / JSON-LD feeds from CNN, Fox, NBC, BBC, etc.
Includes:
Output:
A generator whose yield values are raw feed items (XML / JSON documents) → input to Process 2.
Process 2 — Parsing & Normalization
Goal:
Convert heterogeneous raw feed items into a uniform internal format.
Normalized fields include:
Output:
A generator yielding clean, normalized
NewsItemdocuments → input to Process 3.Process 3 — Content Filtering & Exclusion Rules
Goal:
Remove unwanted items early using configurable rule sets.
Examples:
Techniques:
Output:
A generator yielding allowed, filtered
NewsItemdocuments → input to Process 4.Rejected items are stored separately for auditing.
Process 4 — Topic Classification
Goal:
Assign each item to one or more topics.
Example topics:
Approaches:
Output:
A generator yielding categorized
NewsItemdocuments → input to Process 5.Process 5 — Similarity Analysis & Clustering
Goal:
Group news items from different sources describing the same event.
Techniques:
Produces:
Output:
A generator yielding clusters of related articles → input to Process 6.
Note:
To better match streaming behavior, clustering may operate within bounded windows (e.g., sliding windows) while still consuming the input generator.
Process 6 — Ranking, Urgency, and Freshness Scoring
Goal:
Prioritize which news appears on the Summary Page.
Computed scores:
Weighted formula:
FinalScore = a*Urgency + b*Freshness + c*Coverage + d*EditorRulesItems with the highest scores per topic are selected.
This stage does not require a full total ordering; instead a partial ordering (e.g., top-K per topic) preserves bounded memory.
Editor-driven operations (insert, remove, reorder) are modeled as generator transformations applied downstream of ranking.
Output:
A generator yielding ranked clusters → input to Process 7.
Process 7 — Summary Page Generation
This stage consumes the input generator and produces finite views intended for presentation.
Goal:
Build a continuously updated Summary Page (“Front Page”) containing:
The page auto-refreshes and always reflects the newest items.
Process 8 — Detailed Pages & Cross-Links
This stage consumes its input generator and produces finite presentation views.
For each cluster:
Cross-links include:
Notes on the Process Pipeline
→ produces
G_rawItemslazily (RSS, JSON APIs, DB cursors, web services)for-each,filter,append,prepend,insert-at,remove-where,concat, orfold, etc., producing a new generator derived from the previous one(e.g.,
to-array, bounded sort, pagination)take(n))Why This Fits the Generator Datatype Extremely Well
Alternative Flows
Alternative Flow 1 — Feed Temporarily Stops Producing New Items
Condition:
A feed is reachable but has no new items since the last polling cycle.
Flow:
move-next()).Result:
The pipeline continues uninterrupted; no special handling is required.
Alternative Flow 2 — Partial Consumption of the Pipeline
Condition:
Only a finite prefix of the stream is required (e.g., top N items).
Flow:
take(N).Result:
Latency and memory usage remain bounded. The pipeline supports early termination naturally.
Alternative Flow 3 — Editor Inserts or Reorders Items
Condition:
An editor manually modifies the aggregated stream.
Flow:
(
append,prepend,insert-at,remove-at,remove-where).Result:
Editorial control integrates seamlessly without breaking the pipeline.
Exception Flows
Exception Flow 1 — Feed Unreachable or Network Failure
Condition:
A feed cannot be reached during polling.
Flow:
Result:
The system continues operating with remaining feeds.
Exception Flow 2 — Malformed Feed Data
Condition:
A feed item is malformed (invalid XML/JSON or schema validation problems, e.g. missing required fields).
Flow:
Result:
Malformed data does not propagate downstream.
Exception Flow 3 — Resource Exhaustion Risk
Condition:
A downstream operation risks exceeding memory limits.
Flow:
Result:
System stability is preserved under load.
Postconditions
Upon successful execution:
Functional Outcomes
Technical Guarantees
System State
References
RSS 2.0 Specification
https://www.rssboard.org/rss-specification
Atom Publishing Protocol (RFC 5023)
https://www.rfc-editor.org/rfc/rfc5023
JSON-LD Specification
https://json-ld.org/spec/
TF-IDF, “Understanding TF-IDF (Term Frequency-Inverse Document Frequency)”,
https://www.geeksforgeeks.org/machine-learning/understanding-tf-idf-term-frequency-inverse-document-frequency/
TF-IDF + SVM, “Strengthening Fake News Detection: Leveraging SVM and Sophisticated Text Vectorization Techniques. Defying BERT?”,
https://arxiv.org/html/2411.12703v1
Sentence-BERT (SBERT)
Reimers, N. & Gurevych, I., 2019
https://arxiv.org/abs/1908.10084
Fine-tuned BERT, “Fine-tuning a BERT model”,
https://www.tensorflow.org/tfmodels/nlp/fine_tune_bert
Ada Embeddings (OpenAI)
Radford et al., 2021
https://arxiv.org/abs/2103.00020
Cosine Similarity
https://en.wikipedia.org/wiki/Cosine_similarity
Hierarchical Clustering
https://en.wikipedia.org/wiki/Hierarchical_clustering
DBSCAN
Ester et al., 1996
https://www.aaai.org/Papers/KDD/1996/KDD96-037.pdf
All reactions