Social Network Microservices

This project is a backend system for a simple social network, designed with a microservices architecture to handle various functionalities.

High-Level Design

1. Web Servers:

   • Role: Handle HTTP requests from front-end clients.
   • Function: Serve as the entry point for user interactions and implement business logic.

2. Application Servers:

   • Role: Back-end services.
   • Function: Process application logic and manage data flow between different components.
     • Use-and-Post Service: Manages posts creation, editing, and deletion, as well as interactions with post such as comment and reaction.
     • Newsfeed Generator: Aggregates posts from followed users and sorts them by relevance and time.

3. Metadata Database and Cache:

   • Role: Store user-related information.
   • Function: Maintain user profiles, authentication details, and other users data.

4. Posts Database and Cache:

   • Role: Store post-related information.
   • Function: Handle data for posts, likes, and comments.

5. Photo Storage and Cache:

   • Role: Blob storage (maybe S3).
   • Status: Not implemented yet.
   • Function: Planned to store images associated with posts, providing scalable and efficient image management.

6. Newsfeed Generation Service:

   • Current Implementation: Uses database queries to aggregate and sort posts from followed users by time.
   • Planned Improvement:
     • Pre-build Newsfeeds: Cache each user's newsfeed in Redis memory cache as a List to optimize request speed.
     • Asynchronous Updates: To reduce latency, after creating a post, a message is sent to a message queue. An offline service then processes this message and updates the newsfeed cache for users who follow the poster.

Key Features

• User Management:

  • Sign Up, Login, Edit Profile: Manage user accounts and personal information.

• Follow System:

  • Get User Follow List, Follow, Unfollow: Manage and view user connections.

• User Interaction:

  • Get User Posts, Create Post, Edit Post, Delete Post, Get Post, Comment on Post, Like Post: Interact with and manage posts.

• Newsfeed:

  • Newsfeed: Generate a personalized newsfeed for user with posts from followed users, sorted by relevance and time.

Technologies Used

• Go Gin Web Framework: For building RESTful APIs and managing microservices.
• MySQL: For relational data storage and management.
• Redis: For caching and improving system performance.
• Docker: For containerizing the application to ensure scalability and ease of deployment.

Issues and Discussion

1. System Enhancement: Login

Problem 1: Password Security

When users log in, they provide their user_name and password. If passwords are stored in the database as plain text, a security breach could expose these passwords to unauthorized individuals, including hackers or even insiders with access to the database. This approach is highly insecure because stolen passwords can be used immediately.

Solution:

• Hashing: Instead of storing passwords in plain text, use one-way hashing functions to convert the password into a fixed-size string of characters. Common hashing algorithms include bcrypt and SHA. This method ensures that even if the hashed passwords are exposed, the original passwords cannot be easily retrieved. The hashing process is irreversible, meaning it’s computationally infeasible to convert the hash back into the original password.

• Salting: To further enhance security, add a unique, random string known as a "salt" to each password before hashing. Salting prevents attackers from using precomputed tables (rainbow tables) to crack passwords by ensuring that identical passwords generate different hashes. Each user has a different salt, making it much harder for attackers to use generic methods to guess passwords.

By implementing these techniques, we can improve password security and protect user credentials from being compromised even if an attacker gains access to the database.

Problem 2: Brute Force Attacks

Brute force attacks involve trying all possible password combinations until the correct one is found. Without safeguards, attackers can exploit weak or common passwords by automating this process.

Solution:

• Account Lockout: Implement an account lockout mechanism that temporarily disables accounts after a certain number of failed login attempts.

• Rate Limiting: Apply rate limiting to login attempts, restricting the number of login requests from a single IP address within a specified time frame.

• CAPTCHA: Incorporate CAPTCHA challenges during the login process to distinguish between human users and automated scripts. CAPTCHA requires users to complete tasks that are easy for humans but difficult for bots, adding an extra layer of security.

These approaches help mitigate the risk of brute force attacks and enhance protection for user accounts against unauthorized access.

Problem 3: Checking user_name Availability

To check if a user_name is available during account creation, querying the database directly can be inefficient due to high query volume. Instead, use a caching mechanism to reduce database load:

Redis Cache: Store used user_names in Redis for fast lookups. This avoids frequent database queries.

• Bitmap: Hash user_name to an index and set a corresponding bit. This approach can result in collision (false positive) but it is space efficiency.

• Bloom Filter: Hash user_name multiple times to check if a user_name might be present. It minimizes collision (false positive) and is space-efficient.

Cache Update Strategy:

• After Database Insert: Update the cache after user_name has been successfully added to the database. This ensures that the cache remains consistent with the database. However, if 2 requests with the same username arrive simultaneously, the username may not yet be maked unavailable in Redis, causing both requests could be recorded in the database. To prevent this, a unique index should be set for the username."

• Before Database Query: Update the cache immediately upon receiving the request to create a user_name. This approach requires handling transaction failures case by deleting username hash on Redis, but both Bitmap and Bloom Filter are not support deletion because of false negatives collision.

2. System Enhancement: Post

Problem 1: Media Files Storage

To efficiently store media files (images, videos, ...) for posts:

1. Storage Solution:
   • Images are uploaded to a storage server, such as AWS S3.
   • Each media file is identified by path.
2. Database:
   • The database stores only the path of the media files associated with the post.
3. Image Retrieval:
   • When a request for an image is made, an URL is generated by S3. This URL provides temporary access to the image.
   • The server returns this URL to FE, then FE uses it to display the image.
   • Images are also uploaded to a CDN to enhance global accessibility and speed. First time access can use S3 URL and uploade file to the CDN. When upload to the CDN is complete, replace S3 URL by CDN URL.

3. System Enhancement: Newsfeed

Problem 1: Optimizing Newsfeed Generation

To efficiently deliver newsfeeds to users, the following strategy is employed:

• Use Query to aggregate posts:

  • The basic SQL query for retrieving a user's newsfeed is:

    -- Retrieve all posts from users followed by a specific user
    SELECT p.*
    FROM post p
    -- Join the 'post' table with the 'user' table to link posts with users
    JOIN user u ON p.user_id = u.id
    -- Join the 'user' table with 'user_user' to determine which users are being followed
    JOIN user_user uu ON u.id = uu.following_id
    -- Filter results to include only posts from users followed by the specified user (user_id = 19283923)
    WHERE uu.user_id = 19283923;

  • With this approach performance may be impacted with large posts data because of joining tables.

• Pre-Building Newsfeeds:

  • Newsfeeds are pre-built for each user and stored in Redis.
  • Updating the cache is when post creation requests. When a new post is created:
    • A message is sent to a message queue.
    • Service reads this message and processes it, then updates the post lists for newfeed in Redis for all users who follow post owner.