Unexpected chunking behaviour

[gduteaud]

On attempting to chunk a roughly 12,000 character text using the default settings, I get a total of three chunks. The first two have around 900 characters each, and the last one has over 10,000.

Tested with another, different but similarly structured text, and obtained similar results.

This is particularly surprising as these examples are not that different from the provided "alice.txt" example, which seems to work find.

Here is the relevant part of my code:

const rawContent = document.content;

const text = rawContent.map(page => page.content).join(' ');

console.log(text);
  
// Using llm-chunk https://github.com/golbin/llm-chunk/tree/main

// Default options
const chunks = chunk(text, {
  minLength: 0,          // number of minimum characters into chunk (default: 0)
  maxLength: 1000,       // number of maximum characters into chunk (default: 1000)
  splitter: "paragraph", // paragraph | sentence (default: "paragraph")
  overlap: 0,            // number of overlap chracters (default: 0)
  delimiters: ""        // regex for base split method (default: "")
});

console.log(chunks);

Here's what document.content looks like, before I concatenate just the "content" into a single long string:

[
  {
    "content": "By-Laws - 1  BYLAWS FOR  “NAME”   NEIGHBORHOOD ASSOCIATION  ARTICLE I   PURPOSE  Section 1   NAME   OF   ORGANIZATION:   The   name   of   the   organization   shall be:______________________________.   Here   in   after   referred   to   as   the Association.  Section 2   PURPOSE:   The purposes for which the Association is organized are:  a.   To enhance the livability of the area by establishing and maintaining an open   line   of   communication   and   liaison   between   the   neighborhood, government agencies and other neighborhoods.  b.   To provide an open process by which all members of the neighborhood may involve themselves in the affairs of the neighborhood.  c.   To do and perform all of the activities related to said purposes, to have and enjoy all of the powers granted and engage in any lawful activity for corporations that may be organized under California non-profit corporation law.  d.   To be organized exclusively for educational, scientific and charitable purposes.   Notwithstanding any statement of purpose or powers aforesaid, this association shall not, except to an insubstantial degree, engage in any activities or exercise any powers that are not in furtherance of its specific and primary purpose.",
    "pageNumber": 1
  },
  {
    "content": "By-Laws - 2  ARTICLE II MEMBERSHIP  Section 1   MEMBERSHIP:   Membership in the Association shall be open to all residents, property owners and business licensees located within the boundaries of the Association as defined in Article XII of these bylaws.   To become a member, a membership   application   must   be   completed   and   filed   with   the   Association Secretary and annual dues are paid.  ARTICLE III   FUNDING  Section 1   DUES:   Annual dues will be set forth by a resolution of the Board of Directors.  Section 2   CONTRIBUTIONS:   Voluntary contributions will be accepted.  ARTICLE IV   MEETINGS  Section 1   GENERAL MEMBERSHIP MEETINGS:   There shall be at least two general membership meetings yearly held on days decided upon by the majority vote of the Board of Directors.   Notification for all meetings shall require seven days advance written or telephone notice to all members of the Association.  Section 2   ANNUAL MEETING:   One membership meeting shall be designated as the annual meeting. At this meeting:  a.   The President shall report on the state of the association  b.   The Treasurer shall give an annual financial report  c.   Elections for Members of the Board shall be held  Section 3   SPECIAL MEETING:   A special meeting of the membership may be called by the President or by a majority of the Board of Directors as deemed necessary. Notification and purpose of the special meeting shall require seven days advance written or telephone notice to all members of the Association.",
    "pageNumber": 2
  },
  {
    "content": "By-Laws - 3  Section 4   AGENDA:   The President shall prepare the agenda for general and special meetings of the membership.   Any member may suggest an item to be added to the agenda by submitting the item in writing to the Board of Directors at least seven   days   in   advance   of   the   membership   meeting.   Any member   of   the Association may make a motion to add an item to the board, general or special agendas at those respective meetings.   Adoption of that motion requires a second and majority vote.  Section 5   QUORUM:   A quorum for any general or special meeting of the Association shall be the number of members in attendance.   Unless otherwise specified in these bylaws, decisions of the Association shall be made by a majority vote of those members present at any meeting.  Section 6   PARTICIPATION:   Any general, special, or committee meeting is open to any person and all that wish to may be heard. However, only members are entitled to vote. All actions or recommendations of the general or special meetings shall be communicated to all affected parties, including minority reports.  Section 7   VOTING:   All members, 18 years of age or older, shall have one vote each to be cast during attendance at any general or special meeting.  Section 8   PROCEDURES:   The Association shall follow Roberts Rules of Order (revised) in all areas not covered by the bylaws.  ARTICLE V   BOARD OF DIRECTORS  Section 1   COMPOSITION   OF   THE   BOARD:   The   Board   of   Directors   hereinafter referred to, as the Board shall be composed of nine members.  Section 2   DUTIES OF THE BOARD:  a.   MANAGEMENT:   The Board shall manage the affairs of the Association in the interim between general meetings.   The Board shall be accountable to the membership; shall seek the views of those affected by any proposed policies or reactions before adopting any recommendation on behalf of the Association; and shall strictly comply with these bylaws.",
    "pageNumber": 3
  },
  {
    "content": "By-Laws - 4  b.   VACANCIES:   The Board may fill any vacancy on the Board or a committee by majority vote of the Board.   A member appointed to fill a vacancy shall serve the remainder of the unexpired term.  c.   BOARD MEETINGS:   The Board shall meet four times a year and at least fourteen days prior to any general or special meeting or at any other time the President may designate.   These meetings shall be open session; however only Board members may vote.   A quorum for board meetings shall be fifty percent plus one of the Board members; decisions shall be made by majority vote.   Directors shall be notified of board meetings in writing or by telephone in advance.   A majority of Board members may call a board, general or special meeting as deemed necessary  d.   EMERGENCY POWERS:   In such cases where the Board is required to provide neighborhood response before the question is presented to the membership, the Board must indicate to the questioner that this is the case, and shall present the action taken at a special or general meeting with 30 days, or within a reasonable time for ratification by the membership where circumstances dictate.  ARTICLE VI   ELECTION OF BOARD MEMBERS  Section 1   ELIGIBILITY:   Only members shall be qualified to hold an elected position.  Section 2   TERM OF THE BOARD:   Members of the Board shall be elected to serve for two years.  Section 3   NUMBERING OF SEATS:   The Board seats shall be numbered from one through nine and these number designations shall remain and determine year of election.   Even numbered seats will be elected in even numbered years and odd numbered seats will be elected in odd numbered years.  SPECIAL CIRCUMSTANCE:   The first election of the Board if in an even numbered year will elect odd numbered seats for three years or if in an odd numbered will elect even numbered seats for three years. Subsequent to this election, all seats will be elected to two-year terms.",
    "pageNumber": 4
  },
  {
    "content": "By-Laws - 5  Section 4   METHOD OF ELECTION:   Elections shall be held at the annual meeting. Candidates shall be nominated from the floor.   If two candidates are nominated, election requires a majority vote of the membership present.   If more than two candidates are nominated, a plurality of the vote will be sufficient.  Section 5   IMPEACHMENT:   Any holder of an   elected   office may be removed and replaced by a two-thirds vote of a general or special meeting of the membership. Removal does not require cause.  ARTICLE VII   OFFICERS  Section 1   TITLES:   The Board shall elect from among its members a President, Vice- President, Secretary and Treasurer at the first board meeting following the annual election.  Section 2   DUTIES OF THE OFFICERS:  a.   PRESIDENT:   The President shall prepare the agenda and preside at all meetings of the board and membership; shall appoint chairs of committees with majority approval of the Board.  b.   VICE-PRESIDENT:   The   Vice   President   shall   assist   the   President   and preside at meetings in the absence of the President.  c.   SECRETARY:   The Secretary shall keep minutes and written records of majority and minority opinions expressed at all meetings; shall be responsible for   all   correspondence   for   the   Association,   shall   make   records   of   the Association available for inspection for any proper purpose at a reasonable time.  d.   TREASURER:   The Treasurer shall be held accountable for all funds and shall give an accounting at each general meeting; shall receive, safekeep and disburse the Association   funds, but   such disbursement shall require   the signature of one other Board member.",
    "pageNumber": 5
  },
  {
    "content": "By-Laws - 6  ARTICLE VIII   COMMITTEES  Section 1   STANDING COMM1TTEES:   The following Standing Committees shall be instituted to be responsible for programs and activities of the Association of a long-standing nature.   The chairman of the committees shall be appointed by the President with the agreement of the Board of Directors and may be dismissed without cause by the President with the agreement of the Board. Standing Committees of the Association shall be:  Code and Crime Enforcement  Grievance  Membership  Public Relations/Newletters  Section 2   AD HOC COMMITTEES:   The President shall institute with the agreement of the Board of Directors such other committees as are deemed necessary to transact the business of the association. The chairman of the committee shall be appointed by the President with the agreement of the Board and may be dismissed without cause by the President with the agreement of the Board.  ARTICLE IX   GRIEVANCE PROCEDURE  Section 1   PERSON   OR   GROUP   ADVERSELY   AFFECTED:   A   person   or   group adversely affected by a decision or policy of the Association may submit in writing a complaint to any member of the Grievance Committee.  Section 2   RECEIPT OF COMPLAINT:   With in seven days of receipt of the complaint, the committee shall arrange with the petitioner a mutually acceptable place, day and hour for a review of the complaint, and will in writing, within thirty days recommend a resolution of the grievance to the Board.  Section 3   FINAL RESOLUTION:   The committee shall attempt to resolve the complaint and   shall   submit   a   report   of   their   recommendation   and/or   action   to   the complainant, Board and membership.   If the committee, Board and petitioner cannot reach agreement, final resolution of the complaint shall be by vote of   a majority of the membership at a general or special meeting.",
    "pageNumber": 6
  },
  {
    "content": "By-Laws - 7  ARTICLE X   PROCEDURE FOR CONSIDERATION OF PROPOSALS  Section 1   EXECUTION:   The Board shall be responsible for the execution of this article.  Section 2   SUBMISSION OF PROPOSALS:   Any person or group, inside or outside the boundaries of the Association and any city agency may propose in writing items for consideration and/or recommendation to the Board.   The Board shall decide whether proposed items will appear on the agenda of either the Board, standing or special committees of general or special meetings.  Section 3   NOTIFICATION:   The   proponent   and   members   directly   affected   by   such proposals shall be notified in writing of the place, day and hour the proposal; shall be reviewed not less than seven days in advance.  Section 4   ATTENDANCE:   The proponent may attend this meeting to make a presentation and answer questions concerning the proposal.  Section 5   DISSEMINATION:   The   Association   shall   submit   recommendations   and dissenting views as recorded from the meeting to the proponent and other appropriate parties.  ARTICLE XI   ADOPTION AND AMENDMENT OF THE BYLAWS  Section 1   ADOPTION OF THE BYLAWS:   Adoption of these bylaws shall require a two- thirds vote of the members present at a general meeting.  Section 2   AMENDMENT OF THE BYLAWS:   Amendments to the bylaws shall be made at any general or special meeting after the members have been notified of all proposed amendments to the bylaws by a minimum of at least seven days advance written or telephone notice.   Adoption of amendments shall require a two-thirds vote of the members present at the meeting.",
    "pageNumber": 7
  },
  {
    "content": "By-Laws - 8  ARTICLE XII   BOUNDARIES  Section 1   BOUNDARIES:   The boundaries of the Association shall be defined as follows:  North:  South:  East:  The boundary of the Association shall run along the centerline of all right-of-ways and projections mentioned above.",
    "pageNumber": 8
  },
  {
    "content": "By-Laws - 9  Certificate of Secretary  I, the undersigned, certify that I am the presently elected and acting Secretary of _____________________________, a California Corporation, and the above bylaws, consisting of eight pages, are the bylaws of this corporation as adopted by the members of the Association on the ___ day of ________.  By:____________________  _____________, Secretary",
    "pageNumber": 9
  }
]

And here's what the logged chunks look like:

[
  "By-Laws - 1  BYLAWS FOR  “NAME”   NEIGHBORHOOD ASSOCIATION  ARTICLE I   PURPOSE  Section 1   NAME   "... 900 more characters,
  "   Notwithstanding any statement of purpose or powers aforesaid, this association shall not, except "... 906 more characters,
  " vote of the Board of Directors.   Notification for all meetings shall require seven days advance wr"... 10442 more characters
]

Any help troubleshooting this would be greatly appreciated!

[michaelzoidl]

I've the same issue! Similar settings (minLength: 100, maxLength: 400, overlap: 200) - the first 4 chunks have the exact length - but the last one has a few thousand chars.

[gduteaud]

Yeah I managed to get this to work decently by tinkering with the settings but it's still annoying. It doesn't seem the author is interested in maintaining this repo though so I'd love to fork this/submit a PR to fix it but I just don't have the bandwidth at the moment, happy to collaborate with you if you want to get the ball rolling though!