1.1+ - Roadmap (bug fixes and more) #17

realcoloride · 2023-03-14T15:47:47Z

realcoloride · 2023-03-14T15:50:02Z

@realcoloride After contacting w/ user K3YOMI in #1, it turns out that he fixed with adding the user agent. He sent me a copy with the patched package and it was working, but today I checked for it and it returns nothing. Here's his modified version and I have granted his permission. I hope this helps you to confirm what's going on in here.

node_characterai.zip

According to his instructions, there are two step to make it work:

fill in the the token (not access token) in chat.js, by replacing (TOKEN HERE) to the token (Keep the TOKEN word)
The token can be found in the network tab of dev tool, by searching recent:

Only authenticateWithToken will work (with access token), cannot login as guest

I have confirmed that it was working yesterday night, but somehow it stopped working in today. See if it works for you.

This will be my priority as for now, because it seems like it seems to be a root of other issues right now... Thank you for the help though, I will keep you updated

realcoloride · 2023-03-14T16:25:30Z

Hello!

I located some issues regarding logging in:

Continuing the chat returns 403, so no chat instance is returned because of a cloudflare error
Trying to log in with a guest account will cause a cloudflare error
Trying to log in with an account will fail because for some reason the tokens were reset

So far, I really need a way to go around the cloudflare issues at the moment.
EDIT: I'm considering using puppeteer to go around cloudflare.

realcoloride · 2023-03-14T17:20:27Z

Hello, I've made a puppeteer implentation and it turns out that even with plugins such as the stealth plugin, the request seem to work only when not headless. (so chrome needs to appear, this may be annoying and more resource eating on lower end machines)

Headless Broswer:

Non headless Browser:

I will check out selenium as recommended by some people and friends of mine.
In case of last scenario, I might try hosting a proxy that uses puppeteer to intercept requests but that might be bloated and if the server goes offline, many uses can go down too.

I will keep you updated!

realcoloride · 2023-03-14T18:26:52Z

Alright for now, I will use a temporary fix that uses headful chrome.
Until any solution is found, I will port the requests to use pupeteer.
This might make requests slower & consume more resources but I don't have a choice at the moment.

realcoloride · 2023-03-14T18:33:34Z

For 1.1.1, the temporary solution will be to use the pupeteer instance or a temporary proxy I will try to make in order to forward requests.
Hopefully if after 1.1.1 a way is found to be headless & performant, 1.1.2 could be pushed.

I will keep you updated!

moon91210 · 2023-03-14T18:45:25Z

What about linux users? like myself. i only have access to a terminal.

realcoloride · 2023-03-14T19:10:07Z

Yes I had thought of this part too, I work on Windows and don't know how to use pupeteer on Linux.
For now, I might add proxy support and pupeteer support until i find a viable solution.

marcosathanasoulis · 2023-03-15T03:59:19Z

Getting undefined when logging in with a token (guest login did not work).

const response = await chat.sendAndAwaitResponse('Hi', true)
TypeError: Cannot read properties of undefined (reading 'sendAndAwaitResponse')

Happy to help testing, I have a proxy service I can try and also have some experience with puppeteer. Thanks for the helpful project!

feelinSleepy · 2023-03-15T04:02:13Z

Hey! Sorry if this is a redundant question but I just wanted to ask, will 1.1.1 have a fix for the Status 403 error(At least I think that's the error from reading through the old issues, its the same error as the comment above) on Linux? Its what I run my server on, so hopefully you can figure something out haha. Also, thanks for picking up this project!

realcoloride · 2023-03-15T05:53:29Z

I'm aware of the issues. To answer it all: cloudflare. Cloudflare blocks the requests at the moment, see up and other issues. Which breaks the lib completely. At the moment, I'm doing my best to find the best way to go around cloudflare protection. For such, the best at the moment is a chromium browser using puppeteer. Le mer. 15 mars 2023 à 05:02, feelin__sleepy ***@***.***> a écrit :

…

Hey! Sorry if this is a redundant question but I just wanted to ask, will 1.1.1 have a fix for the Status 403 error(At least I think that's the error from reading through the old issues, its the same error as the comment above) on Linux? Its what I run my server on, so hopefully you can figure something out haha. Also, thanks for picking up this project! — Reply to this email directly, view it on GitHub <#17 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZ4WO5LNDLLA75V3HIXPVKDW4E5NBANCNFSM6AAAAAAV2UBEL4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Naozumi520 · 2023-03-15T10:50:41Z

@realcoloride The way to fix the issue is simple:

const userAgent = 'CharacterAI/1.0.0 (iPhone; iOS 14.4.2; Scale/3.00)';
 page.setUserAgent(userAgent);

Naozumi520 · 2023-03-15T10:50:52Z

Naozumi520 · 2023-03-15T10:51:05Z

It's now working with headless mode.

feelinSleepy · 2023-03-15T11:51:32Z

@realcoloride The way to fix the issue is simple:
const userAgent = 'CharacterAI/1.0.0 (iPhone; iOS 14.4.2; Scale/3.00)';
 page.setUserAgent(userAgent);

Decided to try this real quick before I went to work, no luck, just says "page" in "page.setUserAgent(userAgent);" is undefined. Could definitely be an issue on my end though with my code being wonky.

Naozumi520 · 2023-03-15T12:11:15Z

@realcoloride The way to fix the issue is simple:
const userAgent = 'CharacterAI/1.0.0 (iPhone; iOS 14.4.2; Scale/3.00)';
 page.setUserAgent(userAgent);
Decided to try this real quick before I went to work, no luck, just says "page" in "page.setUserAgent(userAgent);" is undefined. Could definitely be an issue on my end though with my code being wonky.

This isn't mean for the original package. OP used the puppeteer tricks to bypass the Cloudflare service. And I'm currently using my own code to get the response with puppeteer. So you cannot just add it to the package.

feelinSleepy · 2023-03-15T12:14:23Z

@realcoloride The way to fix the issue is simple:
const userAgent = 'CharacterAI/1.0.0 (iPhone; iOS 14.4.2; Scale/3.00)';
 page.setUserAgent(userAgent);
Decided to try this real quick before I went to work, no luck, just says "page" in "page.setUserAgent(userAgent);" is undefined. Could definitely be an issue on my end though with my code being wonky.
This isn't mean for the original package. OP used the puppeteer tricks to bypass the Cloudflare service. And I'm currently using my own code to get the response with puppeteer. So you cannot just add it to the package.

Ohhh. Gotcha, sorry about that

Naozumi520 · 2023-03-15T12:15:34Z

But as soon as OP updated the code with my trick, it should be just working fine!

realcoloride · 2023-03-15T13:06:22Z

Ok after fiddling around a little more, I've found out how to get around it succesfully.

Thank you for the help @Naozumi520!

realcoloride · 2023-03-15T13:36:35Z

So far i've been doing great converting all requests:

Except for sending messages which listen for streaming. I need to check if there is a workaround around this.

Naozumi520 · 2023-03-15T14:09:23Z

This is how I'm currently doing:

page.on('response', async (response) => {
                try {
                    if (response.url() == "https://beta.character.ai/chat/streaming/") {
                        const text = await response.text()
                        // the last choke, which is "lastchoke": true
                        let raw = text.trim().split('"replies": [{"text":')[text.trim().split('"replies": [{"text":').length - 1];
                        raw = raw.split(', "id":')[0].trim()
                        const emojiRegex = /\\u([\dA-Fa-f]{4})/g;
                        const reply = raw.substring(1, raw.length - 1).replaceAll('↵', ', ').replaceAll('\\n', '').replaceAll('\\"', '"')
                        const decodedText = reply.replaceAll(emojiRegex, (match, grp) => String.fromCharCode(parseInt(grp, 16)));
                        console.log(decodedText)
                    }
                } catch (e) {
                    console.log(e)
                }
            });

It's very dirty indeed 😓

realcoloride · 2023-03-15T14:33:21Z

@Naozumi520 I might use your code and modify some stuff until I can find a viable better solution. Then probably push an experimental update after im done doing a few stuff.
Thank you again for your help & contribution.

realcoloride · 2023-03-15T18:12:04Z

@Naozumi520 Hello, Here is my current sample code, could you be more precise please?
This code is currently experimental though, but hopefully if it works, I might push it as a temporary update until the cloudflare issue is fixed. (this code does not work for /streaming/)

async request(url, options) {
        const page = this.page;

        const method = 
        (options.method == 'POST' || options.method == undefined || options.method == null
         ? 'POST' : 'GET');
        
        const body = (method == 'GET' ? {} : options.body);
        const headers = options.headers;

        page.once('request', interceptedRequest => {
            var data = {
                'method': method,
                'postData': body,
                'headers': headers
            };
        
            try {
                interceptedRequest.continue(data);
            } catch (error) {
                console.log("[node_characterai] Puppeteer - Non fatal error: " + error);
            }
        });
        page.on('response', async (response) => {
            // thank you Naozumi520 for this piece of code
            try {
                if (response.url() == "https://beta.character.ai/chat/streaming/") {
                    const text = await response.text()
                    // the last choke, which is "lastchoke": true
                    let raw = text.trim().split('"replies": [{"text":')[text.trim().split('"replies": [{"text":').length - 1];
                    raw = raw.split(', "id":')[0].trim()
                    const emojiRegex = /\\u([\dA-Fa-f]{4})/g;
                    const reply = raw.substring(1, raw.length - 1).replaceAll('↵', ', ').replaceAll('\\n', '').replaceAll('\\"', '"')
                    const decodedText = reply.replaceAll(emojiRegex, (match, grp) => String.fromCharCode(parseInt(grp, 16)));
                    console.log(decodedText)
                }
            } catch (e) {
                console.log(e)
            }
        });

        let response

        try {
            response = await page.goto(url, { waitUntil: 'networkidle0' });
        } catch (error) {
            console.log("[node_characterai] Puppeteer - " + error)
        }
        return response;
    }

Naozumi520 · 2023-03-16T00:03:57Z

Sure I will check for it! But I have lessons soon, I will report the result later.

Naozumi520 · 2023-03-16T00:07:43Z

Hi! Would you mind sending me your currently testing code, so I can make a test with it?

Edit: My discord is "Naozumi#0233"

realcoloride · 2023-04-11T09:25:24Z

Hello everyone,
An experimental update 1.1.2 was published. Please go check it out and tell me if it works properly.

I am looking forward to polish the next updates that come for 1.1+.
Thank you all for your patience and help.

The only limitations for now is that you must use a headful puppeteer.

saki64 · 2023-04-11T18:37:06Z

I got your code to work in headless by adding '--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.3' to args.

Naozumi520 · 2023-04-12T09:19:24Z

I got your code to work in headless by adding '--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.3' to args.

Yes, it works! 🤯

realcoloride · 2023-04-12T09:25:43Z

I got your code to work in headless by adding '--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.3' to args.

I'll check this out! Also have to fix and implement #22 ✅ & #23 ✅.

realcoloride · 2023-04-12T09:53:55Z

Can confirm headless works and will be pushed to the main branch now. Many thanks to @saki64

realcoloride · 2023-04-12T10:13:48Z

Experimental update 1.1.3 has been published.

Please let me know how this update goes!

onnowhere · 2023-04-13T18:24:20Z

It's working really well so far! Great work on figuring this out. One thing I noticed was it seems like search for characters is now allowed for guests (worked just by removing the guest check). Wondering if anything else is now guest enabled that wasn't before as well.

(also this is a bit more of a feature request but thought it could be useful if some additional fields could be returned for example sendAndAwaitResponse which also has name and avatar from src_char)

realcoloride · 2023-04-13T20:59:14Z

Hello and thanks for the feedback!

I will take in consideration that. Because beforehand, it was a logged in only feature. Also, I'm pretty sure the Reply class already contains the information about the character, otherwise, to get more information, a getMessage() should do the trick.

realcoloride · 2023-05-11T09:00:45Z

Hello!
Brand new features coming in like image generation and upload (thanks to @creepycats and his excellent work).

Also, POLL: Do you think a documentation is needed?

Joabutt · 2023-06-04T15:45:11Z

i fixed it by going into the node modules and setting the executable path for puppeteer. in ./node_modules/node_characterai/requester.js at line 39:

            headless: this.#headless,
            args: [
                '--fast-start',
                '--disable-extensions',
                '--no-sandbox',
                '--disable-setuid-sandbox',
                '--no-gpu',
                '--disable-background-timer-throttling',
                '--disable-renderer-backgrounding',
                '--override-plugin-power-saver-for-testing=never',
                '--disable-extensions-http-throttling',
                '--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.3'
            ],
            executablePath: "/usr/bin/chromium-browser",
        });

added the executable path, works for me on my raspberrypi 4b, with wwebjs as a chatbot

realcoloride · 2023-06-06T12:33:40Z

Hello, I am planning to push the update 1.1.4, an experimental update to add new features like the new puppeteer headless version, executable path, cloudflare waiting room, and image utilities. I will make a roadmap to keep you updated and link issues.

realcoloride · 2023-06-06T12:57:48Z

1.1.4+ roadmap:

Done	Title	Issues/PRs	Commits	State
✅	Add waiting room	#45 #30	`63667d2` `e741efe`	Untested
✅	Add new markdown information			Done
✅	Switch to new headless mode		`06aca01`	Done
✅	Be able to change puppeteer's executable path	#53	`06aca01`	Untested
✅	Image features and generation	#33 #38		Untested
✅	Support CharacterAI's Plus (cai+) subscription version	#47	`ef4aaa8`	Done
✅	Fixed requires (require files instead of package, and removing the fs require)			Done
✅	Cleanup chromium instances upon process exit and unauthentification		`5e3e7fc` `1ec54ff`	Done
✅	Fix issues relating to authenticateAsGuest()	#30	`d1b6272` `25a541d`	Done
❌	Fix potential Linux users issues	#31 #53		Waiting

realcoloride · 2023-06-22T21:08:24Z

Hello everyone, I've spotted an issue where chromium tabs would be left open when closing the program. I will investigate the issue and resolve it.

realcoloride · 2023-06-25T11:57:10Z

The issue has been fixed with commits 5e3e7fc and 1ec54ff

realcoloride · 2023-06-25T13:26:18Z

Experimental version 1.1.4 has been published. If you have any bugs or issues or feedback let me part of it!
In a wrap, here are all the new features:

Cloudflare waiting room
New efficient headless mode & auto chromium instances cleanup
Be able to change puppeteer's executable path
Image features and generation
CharacterAI's Plus (cai+) API endpoints
Fix issues relating to authenticateAsGuest()
A lot of bug fixes and more

Happy coding!

feelinSleepy · 2023-07-13T08:13:41Z

Just to make sure, is it possible to get this working on a Raspberry Pi? It seems like it won't due to, according to Puppeteer, the architecture being different and unsupported by Chromium. Though, at the same time, I've heard about Puppeteer running fine on a RPi with a few tweaks. Do you have any insight you can give?

realcoloride · 2023-07-13T18:20:27Z

Just to make sure, is it possible to get this working on a Raspberry Pi? It seems like it won't due to, according to Puppeteer, the architecture being different and unsupported by Chromium. Though, at the same time, I've heard about Puppeteer running fine on a RPi with a few tweaks. Do you have any insight you can give?

Hello, I don't really know as I don't use Linux neither raspberry pi, but I think that it should be fine because chromium is supported on most platforms. Maybe try tweaking things but I can't help much sadly.

feelinSleepy · 2023-07-13T21:06:26Z

Just to make sure, is it possible to get this working on a Raspberry Pi? It seems like it won't due to, according to Puppeteer, the architecture being different and unsupported by Chromium. Though, at the same time, I've heard about Puppeteer running fine on a RPi with a few tweaks. Do you have any insight you can give?

Hello, I don't really know as I don't use Linux neither raspberry pi, but I think that it should be fine because chromium is supported on most platforms. Maybe try tweaking things but I can't help much sadly.

Ah, I suppose "Here be dragons" haha. I'll definitely keep tweaking, it should be possible, I suppose the issue is I just don't know where to make the tweaks at.

I'm looking for this API's equivalent of this code example :

"const puppeteer = require('puppeteer');

(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://example.com');
await page.screenshot({path: 'example.png'});

await browser.close();
})();"

Basically, where Puppeteer gets initiated it seems. I need to change some settings about where it runs Chromium from as far as I can tell. Do you know where I might be able to find it? Many thanks for any help.

realcoloride · 2023-07-13T21:22:46Z

Basically, where Puppeteer gets initiated it seems. I need to change some settings about where it runs Chromium from as far as I can tell. Do you know where I might be able to find it? Many thanks for any help.

see README.md for information, "in depth troubleshooting".

feelinSleepy · 2023-07-13T21:33:35Z

Basically, where Puppeteer gets initiated it seems. I need to change some settings about where it runs Chromium from as far as I can tell. Do you know where I might be able to find it? Many thanks for any help.

see README.md for information, "in depth troubleshooting".

Perfect! I'm not sure how I missed it- thanks for the help!

realcoloride · 2023-09-10T20:34:12Z

Closed in relevance for 1.2+ versions, do not hesitate to open an issue if something is needed.

This was referenced Mar 14, 2023

Error, fetch is not defined. #15

Closed

Tokens don't work #13

Closed

403 on createOrContinueChat #16

Closed

Chat.deleteMessage() doesn't seem to work #14

Closed

realcoloride pinned this issue Mar 14, 2023

realcoloride changed the title ~~1.1.1 - Roadmap (bug fixes and more)~~ 1.1+ - Roadmap (bug fixes and more) Mar 15, 2023

realcoloride mentioned this issue Apr 12, 2023

Error: Failed to launch the browser process puppeteer #25

Closed

realcoloride mentioned this issue Jun 5, 2023

Cannot find module “fs/promises” #50

Closed

This was referenced Jun 6, 2023

Adding image support #33

Closed

.createOrContinueChat hangs #47

Closed

realcoloride mentioned this issue Jun 6, 2023

[New Pull Request] added waiting time when web character.ai is busy #45

Closed

Parking-Master mentioned this issue Jun 7, 2023

Help #53

Closed

realcoloride mentioned this issue Jun 25, 2023

Its load? #36

Closed

realcoloride closed this as completed Sep 10, 2023

realcoloride unpinned this issue Sep 11, 2023

1.1+ - Roadmap (bug fixes and more) #17

1.1+ - Roadmap (bug fixes and more) #17

Comments

realcoloride commented Mar 14, 2023 • edited Loading

[Roadmap] Sum up of update goals: (from issue #14, merged here)

realcoloride commented Mar 14, 2023

realcoloride commented Mar 14, 2023 • edited Loading

realcoloride commented Mar 14, 2023

realcoloride commented Mar 14, 2023

realcoloride commented Mar 14, 2023

moon91210 commented Mar 14, 2023

realcoloride commented Mar 14, 2023

marcosathanasoulis commented Mar 15, 2023

feelinSleepy commented Mar 15, 2023

realcoloride commented Mar 15, 2023 via email

Naozumi520 commented Mar 15, 2023

Naozumi520 commented Mar 15, 2023

Naozumi520 commented Mar 15, 2023

feelinSleepy commented Mar 15, 2023

Naozumi520 commented Mar 15, 2023

feelinSleepy commented Mar 15, 2023

Naozumi520 commented Mar 15, 2023

realcoloride commented Mar 15, 2023

realcoloride commented Mar 15, 2023

Naozumi520 commented Mar 15, 2023 • edited Loading

realcoloride commented Mar 15, 2023 • edited Loading

realcoloride commented Mar 15, 2023 • edited Loading

Naozumi520 commented Mar 16, 2023

Naozumi520 commented Mar 16, 2023 • edited Loading

realcoloride commented Apr 11, 2023

saki64 commented Apr 11, 2023

Naozumi520 commented Apr 12, 2023

realcoloride commented Apr 12, 2023 • edited Loading

realcoloride commented Apr 12, 2023 • edited Loading

realcoloride commented Apr 12, 2023

onnowhere commented Apr 13, 2023 • edited Loading

realcoloride commented Apr 13, 2023

realcoloride commented May 11, 2023

Joabutt commented Jun 4, 2023 • edited Loading

realcoloride commented Jun 6, 2023

realcoloride commented Jun 6, 2023 • edited Loading

realcoloride commented Jun 22, 2023

realcoloride commented Jun 25, 2023

realcoloride commented Jun 25, 2023

feelinSleepy commented Jul 13, 2023

realcoloride commented Jul 13, 2023

feelinSleepy commented Jul 13, 2023

realcoloride commented Jul 13, 2023

feelinSleepy commented Jul 13, 2023

realcoloride commented Sep 10, 2023

realcoloride commented Mar 14, 2023 •

edited

Loading

realcoloride commented Mar 14, 2023 •

edited

Loading

Naozumi520 commented Mar 15, 2023 •

edited

Loading

realcoloride commented Mar 15, 2023 •

edited

Loading

realcoloride commented Mar 15, 2023 •

edited

Loading

Naozumi520 commented Mar 16, 2023 •

edited

Loading

realcoloride commented Apr 12, 2023 •

edited

Loading

realcoloride commented Apr 12, 2023 •

edited

Loading

onnowhere commented Apr 13, 2023 •

edited

Loading

Joabutt commented Jun 4, 2023 •

edited

Loading

realcoloride commented Jun 6, 2023 •

edited

Loading