BatchHTTP

A Go library for performing a batch of HTTP requests sequentially, with an optional delay between each request. This library focuses on simplicity and reliability for development-time scripting and debugging, such as batch-creating test data, triggering admin APIs, or performing backend maintenance tasks. It is not intended for production use or high-performance workloads.

Features

• Sequential Execution: All requests are performed one at a time, with no concurrency
• Configurable Delay: Add an optional delay between requests to avoid overwhelming servers
• Convenient Methods: Includes Get, PostJson, and PostForm methods
• Flexible Response Handling: Save responses as separate files per ID or combine them into a single output file
• Json Field Extraction: Extract specific fields from Json responses using gjson
• Custom ID Generation: Use the built-in GenIds function or provide your own custom ID generator
• Proxy Support: Configure HTTP/SOCKS5 proxy for all requests
• Header Support: Add custom headers to your requests
• Debug Mode: Enable detailed request/response logging powered by go-resty
• Powered by go-resty: Built on top of the popular go-resty HTTP client for reliability and features

Installation

go get github.com/dharper7051/batchhttp

Usage

Basic GET Requests

package main

import (
    "fmt"
    "time"
    "github.com/dharper7051/batchhttp"
)

func main() {
    // Create a client without proxy
    client, err := batchhttp.NewClient("")
    if err != nil {
        panic(err)
    }
    
    // Perform GET requests
    err = client.Get(batchhttp.Options{
        GenIdsFunc: func() []string {
            return batchhttp.GenIds(1, 100)
        },
        UrlFunc: func(id string) string {
            return fmt.Sprintf("https://api.example.com/users/%s", id)
        },
        SeparateFile: true,
        OutputFile:   "./responses/user.json",
        Delay:        1 * time.Second,
        Headers: map[string]string{
            "Authorization": "Bearer your-token",
        },
    })
    
    if err != nil {
        panic(err)
    }
}

POST Json Requests

client, _ := batchhttp.NewClient("")

err := client.PostJson(batchhttp.Options{
    GenIdsFunc: func() []string {
        return batchhttp.GenIds(1, 50)
    },
    UrlFunc: func(id string) string {
        return "https://api.example.com/data"
    },
    BodyFunc: func(id string) []byte {
        return batchhttp.JsonBody(map[string]interface{}{
            "id":   id,
            "name": fmt.Sprintf("User %s", id),
        })
    },
    OutputFile: "all_responses.json",
    Delay:      500 * time.Millisecond,
})

POST Form Requests

client, _ := batchhttp.NewClient("")

err := client.PostForm(batchhttp.Options{
    GenIdsFunc: func() []string {
        return []string{"10", "20", "30", "40", "50"}
    },
    UrlFunc: func(id string) string {
        return "https://api.example.com/submit"
    },
    BodyFunc: func(id string) []byte {
        values := url.Values{}
        values.Set("user_id", id)
        values.Set("action", "update")
        return batchhttp.FormBody(values)
    },
    SeparateFile: true,
    OutputFile:   "./form_responses/response.txt",
    Delay:        200 * time.Millisecond,
})

Using Proxy

Configure a proxy for all requests:

// SOCKS5 proxy
client, err := batchhttp.NewClient("socks5://127.0.0.1:1080")
if err != nil {
    panic(err)
}

// HTTP proxy
client, err := batchhttp.NewClient("http://proxy.example.com:8080")
if err != nil {
    panic(err)
}

err = client.Get(batchhttp.Options{
    GenIdsFunc: func() []string {
        return batchhttp.GenIds(1, 10)
    },
    UrlFunc: func(id string) string {
        return fmt.Sprintf("https://api.example.com/users/%s", id)
    },
    OutputFile: "responses.json",
})

Debug Mode

Enable debug mode to see detailed request and response information:

client, _ := batchhttp.NewClient("")

err := client.Get(batchhttp.Options{
    GenIdsFunc: func() []string {
        return batchhttp.GenIds(1, 5)
    },
    UrlFunc: func(id string) string {
        return fmt.Sprintf("https://api.example.com/users/%s", id)
    },
    Debug:      true,
    OutputFile: "responses.json",
    Delay:      500 * time.Millisecond,
})

When Debug: true is set, the library will output detailed information about each request and response, including:

• Request URL, method, headers, and body
• Response status, headers, and body
• Request/response timing information

This is powered by go-resty's built-in debug functionality and is useful for troubleshooting API issues.

Extracting Json Fields

Use the ExtractFields option to extract specific fields from Json responses:

client, _ := batchhttp.NewClient("")

err := client.Get(batchhttp.Options{
    GenIdsFunc: func() []string {
        return batchhttp.GenIds(1, 100)
    },
    UrlFunc: func(id string) string {
        return fmt.Sprintf("https://api.example.com/users/%s", id)
    },
    ExtractFields:   "name,email,user.id",
    ExtractFieldSep: "\t",
    OutputFile:      "extracted_data.tsv",
    Delay:           500 * time.Millisecond,
})

This will extract the specified fields from each Json response and save them with the id as the first field. The default separator is tab (\t), but you can customize it using ExtractFieldSep. Nested fields are supported using dot notation (e.g., user.profile.age).

Output format: The first line contains the header with field names, followed by data lines:

id	name	email	user.id
1	John Doe	john@example.com	123
2	Jane Smith	jane@example.com	456

Saving Response Bodies

You can save all response bodies to a file with separators for debugging or logging:

client, _ := batchhttp.NewClient("")

err := client.Get(batchhttp.Options{
    GenIdsFunc: func() []string {
        return batchhttp.GenIds(1, 5)
    },
    UrlFunc: func(id string) string {
        return fmt.Sprintf("https://api.example.com/users/%s", id)
    },
    ResponseBodyFile: "response_bodies.txt",
    OutputFile:       "extracted_data.csv",
    ExtractFields:    "name,email",
    Delay:            500 * time.Millisecond,
})

The ResponseBodyFile will contain all response bodies with separators:

=== 1 ===
{"id":1,"name":"User 1","email":"user1@example.com"}
=== 2 ===
{"id":2,"name":"User 2","email":"user2@example.com"}
=== 3 ===
{"id":3,"name":"User 3","email":"user3@example.com"}

POST with Dynamic URLs

You can use UrlFunc with POST requests to send to different URLs per id:

client, _ := batchhttp.NewClient("")

err := client.PostJson(batchhttp.Options{
    GenIdsFunc: func() []string {
        return batchhttp.GenIds(1, 10)
    },
    UrlFunc: func(id string) string {
        return fmt.Sprintf("https://api.example.com/users/%s/update", id)
    },
    BodyFunc: func(id string) []byte {
        return batchhttp.JsonBody(map[string]interface{}{
            "status": "active",
        })
    },
    OutputFile: "update_responses.json",
    Delay:      500 * time.Millisecond,
})

Custom ID Generation

You can provide a custom id generation function for any use case:

client, _ := batchhttp.NewClient("")

err := client.Get(batchhttp.Options{
    GenIdsFunc: func() []string {
        // Generate custom ids (e.g., UUIDs, database ids, etc.)
        return []string{"user-001", "user-002", "user-003"}
    },
    UrlFunc: func(id string) string {
        return fmt.Sprintf("https://api.example.com/users/%s", id)
    },
    OutputFile: "responses.json",
    Delay:      500 * time.Millisecond,
})

Or use the built-in GenIds helper for sequential numeric IDs:

err := client.Get(batchhttp.Options{
    GenIdsFunc: func() []string {
        return batchhttp.GenIds(1, 100) // Generates "1" through "100"
    },
    UrlFunc: func(id string) string {
        return fmt.Sprintf("https://api.example.com/users/%s", id)
    },
    OutputFile: "responses.json",
})

API Reference

Client

NewClient(proxyUrl string) (*Client, error)

Creates a new batch HTTP client with optional proxy support.

• proxyUrl: Proxy URL in the format "socks5://127.0.0.1:1080" or "http://proxy.example.com:8080". Use empty string for no proxy.

Functions

GenIds(start, count int) []string

Generates a slice of sequential string ids.

• start: The starting id number
• count: The number of ids to generate

Example: GenIds(1, 100) generates string ids from "1" to "100".

Methods

Get(opts Options) error

Performs sequential GET requests for each id.

PostJson(opts Options) error

Performs sequential POST requests with Json bodies.

PostForm(opts Options) error

Performs sequential POST requests with form-encoded bodies.

Options

All methods use the unified Options struct:

• GenIdsFunc: Function to generate ids (required)
• UrlFunc: Function that generates a URL based on an id (required)
• BodyFunc: Function that generates request body based on an id (for POST requests, optional)
• OutputFile: File path for saving responses (optional)
• Headers: Custom HTTP headers (optional)
• SeparateFile: If true, appends id suffix to OutputFile for each response (e.g., response_1.json, response_2.json)
• ExtractFields: Comma-separated list of Json field paths to extract (optional)
• ExtractFieldSep: Separator for extracted fields (default: "\t")
• ResponseBodyFile: File path for saving all response bodies with separators (optional)
• Debug: Enable debug mode for detailed request/response logging (default: false)
• Delay: Duration to wait between consecutive requests (optional, use 0 for no delay)

Helper Functions

JsonBody(data map[string]interface{}) []byte

Converts a map to Json bytes for use in BodyFunc. Returns an empty byte slice if marshaling fails.

FormBody(values url.Values) []byte

Converts URL values to form-encoded bytes for use in BodyFunc.

HTTP Client

This library is built on top of go-resty, a popular and feature-rich HTTP client for Go. This provides:

• Robust HTTP/HTTPS support
• Automatic retry capabilities
• Flexible proxy configuration
• Built-in debug mode
• Clean and simple API

Json Field Extraction

The library uses gjson for efficient Json parsing and field extraction. When you specify ExtractFields, the library will:

1. Parse each Json response
2. Extract the specified fields using gjson path syntax
3. Prepend the id as the first field
4. Output the values separated by ExtractFieldSep (default: tab)

Supported gjson path syntax:

• Simple fields: name, email
• Nested fields: user.profile.age, data.items.0.id
• Array elements: users.0.name, items.#.price
• Wildcards and queries: See gjson documentation

If a field doesn't exist in the response, an empty string is used in its place.

Example output with default tab separator:

id	name	email
1	John Doe	john@example.com
2	Jane Smith	jane@example.com
3	Bob Wilson	bob@example.com

Example with custom separator (e.g., ExtractFieldSep: ","):

id,name,email
1,John Doe,john@example.com
2,Jane Smith,jane@example.com
3,Bob Wilson,bob@example.com

The header line is automatically generated from the field names specified in ExtractFields, with "id" prepended as the first column.

Response Handling

You have two options for saving responses:

1. Separate Files: Set SeparateFile: true and provide an OutputFile. Each response will be saved with the id appended to the filename (e.g., response_1.json, response_2.json).

2. Single File: Set SeparateFile: false (or omit it) and provide an OutputFile path. All responses will be concatenated into a single file, separated by newlines.

Licence

MIT Licence - see LICENSE file for details.