Integrate fact checking action

[danisztls]

Fix #14

[danisztls]

GitGuardian warning is a false positive

[marina-chibizova]

@alxarno I will need your help with review

[github-actions]

2gether, a European cryptocurrency platform, was hacked, resulting in the loss of €1.183 million worth of cryptocurrencies. ✅
Multiple sources confirm that 2gether was hacked and that €1.183 million worth of cryptocurrency was stolen, which equates to 26.79% of overall funds.

The attackers behind the 2gether hack remain unidentified. ✅
Multiple articles report that the hackers behind the 2gether hack remain unidentified, including a statement from the 2gether CEO on Twitter.

Fact-check failed due to errors

[danisztls]

The action is triggered by the review_requested/ready_for_review pull request events. That's why I'm converting it back and forth between draft & ready so to trigger the action. In the future it might make sense to implement the possibility of triggering it manually using workflow_dispatch.

So to demo the fact checking script, I copied one of the posts and modified the following statement to be false:

**August 31, 2020:** 2gether DIDN'T resumes normal operations.

The fact checking isn't exhaustive and will check a few statements chosen by the model and not all of them. In this case it ignored the falsificated statement.

Note: Revert the test data before merging.

[github-actions]

2gether, a European cryptocurrency platform, was hacked ✅
Multiple sources confirm that 2gether was hacked and approximately €1.2 million in cryptocurrency was stolen from cryptocurrency investment accounts.

2gether lost €1.183 million worth of cryptocurrencies ✅
The claim is true. According to multiple sources, including The Cryptonomist and Crypto trading platform 2gether, 2gether suffered a hack and lost 27% of its funds, equal to 1.1 million euros. Hackers stole 114 BTC and 281 ETH worth a combined €1.18 million from its users' investment accounts.

Fact-check failed due to errors

[danisztls]

The script is failing because of:

Rate limit reached for default-gpt-3.5-turbo in organization org-qifYVoOOQvrz24XJJ6GjJmpb on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.

[marina-chibizova]

@danisztls do you need help with this?

[danisztls]

@danisztls do you need help with this?

Yes. Need to add a payment method to the account.

[marina-chibizova]

@danisztls done

[github-actions]

2gether, a European cryptocurrency platform, was hacked, resulting in the loss of €1.183 million worth of cryptocurrencies. ✅
Multiple sources confirm that 2gether was hacked and €1.183 million worth of cryptocurrencies were stolen.

A combination of poor security practices and system vulnerabilities allowed the attackers to access user funds. ✅
The search results mention weak security controls and practices, unpatched software, and poor cyber hygiene as common causes for system vulnerabilities that can be exploited by attackers to gain access to sensitive information or take control of a system. Therefore, it is likely that a combination of poor security practices and system vulnerabilities allowed the attackers to access user funds.

2gether lost €1.183 million worth of cryptocurrencies, which included various types of crypto assets. ✅
Multiple sources confirm that 2gether suffered a security breach and lost €1.183 million worth of cryptocurrency from investment accounts, which equates to 26.79% of overall funds.

2gether has been working on compensating affected customers, albeit with some limitations and delays. ✅
According to the search result '2gether hack: Crypto trading platform struggles to reimburse stolen ...', the company has been considering ways to repay affected customers adequately, indicating that they have been working on compensating affected customers. However, the same result also mentions that the company wasn't capable of reimbursing the funds lost during the 2gether hack immediately, which suggests that there have been some limitations and delays.

[github-actions]

2gether, a European cryptocurrency platform, was hacked, resulting in the loss of €1.183 million worth of cryptocurrencies. ✅
Multiple sources confirm that 2gether was hacked on July 31, 2020 and lost €1.183 million worth of cryptocurrencies, including Bitcoin and Ethereum.

A combination of poor security practices and system vulnerabilities allowed the attackers to access user funds. ✅
The search results mention poor security practices such as weak controls, unpatched software, and default configurations and passwords that can be exploited by attackers. These practices can lead to system vulnerabilities that allow attackers to gain access to sensitive information and take control of a system, which could potentially result in the theft of user funds.

2gether lost €1.183 million worth of cryptocurrencies, which included various types of crypto assets. ✅
Multiple sources confirm that 2gether suffered a hack in July 2021 and lost €1.183 million worth of cryptocurrencies, including Bitcoin and Ethereum.

2gether has been working on compensating affected customers, albeit with some limitations and delays. ✅
The search result shows that 2gether has been considering ways to repay affected customers adequately, although it was not capable of reimbursing the funds lost during the 2gether hack immediately. However, there is no evidence to suggest that the compensation process has not been ongoing.

26.79% of users were affected by the attack and that they are working on a compensation plan. ❌
There is no evidence to support the claim that 26.79% of users were affected by the attack. However, it is true that 2gether compensated 91% of its users in full in BTC and ETH following the crowdfunding campaign after the hack. They were unable to reimburse the remaining 9% of users affected by the hack even after a successful fundraising effort.

[danisztls]

@marina-chibizova This is the action output. Is this what is expected?

[marina-chibizova]

yes, let's merge this one. you can also add some caching to not process each commit every time - something like this https://github.com/probot/metadata. but let's leave it for next pr

[danisztls]

@marina-chibizova

let's merge this one

A nitpick but I recommend squashing instead of merging, the result will be the same but the Git log will make more sense. 😄

you can also add some caching to not process each commit every time - something like this https://github.com/probot/metadata.

It doesn't process each commit every time. It process files modified by the PR when it's marked as ready for review.

I would cache the workflow metadata in the repo itself instead of in the PR as a comment like probot/metadata do. It might be useful to have a fact checking metadata file for each content file. Though caching might not be that useful as the model isn't so deterministic and it can extract/generate similar statements with different wordings in each run.

Currently there's kind of a bug where in the first run it extracts and verifies a few statements and in subsequent runs it verify a bit more. Last time it was exceptionally triggered twice due to the CODEOWNERS change and in the second run it verified an additional statement which was identified as false but yet that is covering ~half of the content only. What's is happening is that the model is timing out as it normally does. I should debug and fix this, probably by chunking the prompts in smaller bits so it doesn't time out.