137 arbitrage agent #511
137 arbitrage agent #511
Conversation
…trage-agent # Conflicts: # poetry.lock # pyproject.toml
WalkthroughThis pull request introduces significant updates to the prediction market agent's architecture, including the addition of new data models for handling correlations and arbitrage betting strategies. It defines classes such as Changes
Possibly related issues
Possibly related PRs
Suggested reviewers
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Outside diff range and nitpick comments (2)
tests/agents/arbitrage_agent/test_arbitrage_agent.py (2)
13-45: LGTM: Well-designed fixtures with a minor suggestion.The fixtures are well-structured and effectively isolate components for testing. The use of mocking in the
arbitrage_agentfixture and the creation of distinct market scenarios are particularly good practices.Consider adding type hints to the mock objects in the market fixtures for improved code clarity:
m1 = Mock(spec=OmenAgentMarket)This change would make the intended type of the mock object more explicit.
48-66: LGTM: Well-structured test with parameterization. Consider additional test cases.The test function is well-designed, using parameterization to efficiently test both related and unrelated market scenarios. It effectively utilizes the fixtures and checks the correlation calculation against a threshold.
Consider adding the following test cases to increase coverage:
- Test with edge case correlations (e.g., correlation very close to the threshold).
- Test the behavior when markets have identical questions.
- Test with markets in different languages to ensure language-agnostic correlation.
Example:
@pytest.mark.parametrize( "market1_question, market2_question, expected_correlation", [ ("Will X happen?", "Will X occur?", 0.99), # Nearly identical ("Will X happen?", "Will X happen?", 1.0), # Identical ("Will X happen?", "Wird X passieren?", 0.9), # Different languages ] ) def test_correlation_edge_cases( arbitrage_agent: DeployableOmenArbitrageAgent, market1_question: str, market2_question: str, expected_correlation: float, ) -> None: market1 = Mock(spec=OmenAgentMarket, question=market1_question) market2 = Mock(spec=OmenAgentMarket, question=market2_question) correlation = arbitrage_agent.calculate_correlation_between_markets( market=market1, related_market=market2 ) assert pytest.approx(correlation.correlation, abs=0.01) == expected_correlationThis additional test would cover more scenarios and increase the robustness of your test suite.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
⛔ Files ignored due to path filters (1)
poetry.lockis excluded by!**/*.lock,!**/*.lock
📒 Files selected for processing (3)
- prediction_market_agent/agents/arbitrage_agent/data_models.py (1 hunks)
- prediction_market_agent/agents/arbitrage_agent/deploy.py (1 hunks)
- tests/agents/arbitrage_agent/test_arbitrage_agent.py (1 hunks)
🧰 Additional context used
🔇 Additional comments (5)
tests/agents/arbitrage_agent/test_arbitrage_agent.py (2)
1-10: LGTM: Imports are appropriate and well-organized.The imports section includes all necessary modules and classes for the test file. The organization is logical, separating standard library imports from project-specific ones.
1-66: Overall, excellent test structure with room for minor enhancements.This test file is well-organized and effectively tests the core functionality of the arbitrage agent. The use of fixtures, parameterization, and mocking demonstrates good testing practices.
To further improve the test suite:
- Consider adding type hints to mock objects in fixtures.
- Expand test cases to cover edge scenarios and language-agnostic correlation.
These enhancements will increase the robustness and clarity of your test suite, ensuring better coverage of the arbitrage agent's functionality.
prediction_market_agent/agents/arbitrage_agent/data_models.py (1)
16-18: Verify compatibility with Pydantic version and decorator usageThe use of
@computed_fieldalongside@propertymay not be necessary and could lead to unexpected behavior. Additionally, the@computed_fielddecorator is available in Pydantic v2.Ensure that the project uses Pydantic v2. If not, consider one of the following:
- Option 1: Remove the
@computed_fielddecorator if not required.- Option 2: Adjust the code to match the appropriate Pydantic version's conventions.
Example adjustment:
- @computed_field # type: ignore[prop-decorator] @property def potential_profit_per_bet_unit(self) -> float:Please verify that the computed properties behave as intended after making changes.
Also applies to: 36-38, 45-47
prediction_market_agent/agents/arbitrage_agent/deploy.py (2)
74-74: Verify the model name 'gpt-4o-mini'In the
_build_chainmethod, the model parameter is set to'gpt-4o-mini'. Please ensure that this is the correct model name and that it is accessible via the OpenAI API.
138-138: Ensure unique market identificationThe condition
related_market.id != market.idchecks if the markets are different. Verify thatidattributes are correctly populated and comparable betweenAgentMarketand the market objects retrieved from the subgraph handler.
| "main_market_question": market, | ||
| "related_market_question": related_market, |
There was a problem hiding this comment.
Incorrect arguments passed to self.chain.invoke
In the get_correlated_markets method, you're passing market and related_market objects to self.chain.invoke, whereas the chain expects strings containing the market questions.
Apply this diff to fix the arguments:
- "main_market_question": market,
- "related_market_question": related_market,
+ "main_market_question": market.question,
+ "related_market_question": related_market.question,📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "main_market_question": market, | |
| "related_market_question": related_market, | |
| "main_market_question": market.question, | |
| "related_market_question": related_market.question, |
| self, pair: CorrelatedMarketPair | ||
| ) -> list[Trade]: | ||
| market_to_bet_yes, market_to_bet_no = pair.main_market, pair.related_market | ||
| total_amount: BetAmount = pair.main_market.get_tiny_bet_amount() |
There was a problem hiding this comment.
This seems like something that should be configurable at the top level
| Trade( | ||
| trade_type=TradeType.BUY, | ||
| outcome=True, | ||
| amount=TokenAmount( | ||
| amount=amount_yes, currency=market_to_bet_yes.currency | ||
| ), | ||
| ), | ||
| Trade( | ||
| trade_type=TradeType.BUY, | ||
| outcome=False, | ||
| amount=TokenAmount( | ||
| amount=amount_no, currency=market_to_bet_no.currency | ||
| ), | ||
| ), |
There was a problem hiding this comment.
Buying [market1: yes, market2: no] assumes positive correlation between markets. I think you need to have some logic that chooses the directions based on correlation
There was a problem hiding this comment.
Positive correlation is assumed here -> https://github.com/gnosis/prediction-market-agent/pull/511/files/433656d955694050935c662e1574f80c552d3cc7#diff-1b9d8df9eb7cb55918e642ebc0f26902d5211d4946a8c4d0454e9889dd05c371R138-R140
I didn't implement logic for the negatively correlated case (it was on an earlier version), I think positive correlation is a good start. Also, negative correlation is harder to determine, e.g. Trump president vs Kamala president is 100% negatively correlated but not trivial for LLM to tell, hence preferred to stick to 100% correlation case.
Happy with suggestions though.
There was a problem hiding this comment.
Oh okay. Fwiw I think it's okay if the LLMs find it hard to determine that, as long as there aren't false positives.
There was a problem hiding this comment.
Going with your bool suggestion from the other thread.
There was a problem hiding this comment.
Also, negative correlation is harder to determine, e.g. Trump president vs Kamala president is 100% negatively correlated but not trivial for LLM to tell, hence preferred to stick to 100% correlation case.
What did you try? It seems to work fine for me, see the comment above.
There was a problem hiding this comment.
I believe I addressed this case (negative correlation) with my latest changes, please see 7afbb41#diff-7b9692129117b2cc9d653a28cdd7bb8f5fbaf3cc6020d6ed791b8642efc63467R62-R65)
| correlation_threshold: float = 0.8 | ||
|
|
||
| def load(self) -> None: | ||
| self.subgraph_handler = OmenSubgraphHandler() |
There was a problem hiding this comment.
Any reason you've made this agent omen-specific? Looks like the only thing you've used the sgh for is to get markets. So it looks like this could easily be changed to become market-platform agnostic
There was a problem hiding this comment.
I made it Omen-specific because the related_markets should be queryable via Pinecone (because of the TTA agent), and only Omen agents are stored there currently.
There was a problem hiding this comment.
Oh I didn't realise that. Where does the indexing of omen markets in the pincone db happen? If it's not part of this agent, it goes against @kongzii's philosophy of "anyone should be able to run the agents (if they have appropriate api keys), and get the same kind of results".
Regardless, there doesn't seem to be a need to have any Omen dependencies here - you could just have a if market_type != MarketType.Omen: raise ... check
There was a problem hiding this comment.
Indexing occurs below
I don't think it prevents someone from running the agent - you just need Pinecone api keys and, on the first run, it will fill out Pinecone and have the same dataset as we do.
Regardless, there doesn't seem to be a need to have any Omen dependencies here - you could just have a if market_type != MarketType.Omen: raise ... check
Makes sense to me, implementing this.
There was a problem hiding this comment.
Well it does if the user wants to run just this agent, and not the DeployableThinkThoroughlyAgent as well! I think we need to do self.pinecone_handler.insert_all_omen_markets_if_not_exists(...) somewhere here
There was a problem hiding this comment.
Sure - added 55aee24
There was a problem hiding this comment.
Just had a look at what pinecone_handler.update_markets is doing. Is this what we want? Two things I'm not sure about:
- It doesn't look like it's removing closed markets from the index
- It only fetches the last 7 days' worth of created markets
created_after = utcnow() - datetime.timedelta(days=7). Don't we want all open markets?
There was a problem hiding this comment.
Implemented changes as suggested by @evangriffiths and discussed with @kongzii :
-> Closing markets are NOT removed, they might be useful for TTA - only issue here is using closed markets inside ArbitrageAgent, we filter those out now (see DeployableArbitrageAgent::get_correlated_markets).
-> Fetching all open markets instead of markets created in the last 7 days (pinecone_handler::insert_all_omen_markets_if_not_exists) and storing them in Pinecone.
There was a problem hiding this comment.
Actionable comments posted: 4
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (2)
- prediction_market_agent/agents/arbitrage_agent/deploy.py (1 hunks)
- prediction_market_agent/agents/arbitrage_agent/prompt.py (1 hunks)
🧰 Additional context used
There was a problem hiding this comment.
Actionable comments posted: 9
🧹 Outside diff range and nitpick comments (1)
prediction_market_agent/agents/arbitrage_agent/data_models.py (1)
12-35: LGTM: CorrelatedMarketPair class and potential_profit_per_bet_unit method are well-implemented.The class structure and method implementation are appropriate. The method handles the case where total probability is zero, avoiding potential division by zero errors.
However, there's a minor improvement we can make to the comment:
Update the comment to remove the reference to LLM prompt:
- # Negative correlations are not yet supported by the current LLM prompt, hence not handling those for now. + # Negative correlations are not yet supported, hence not handling those for now.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
⛔ Files ignored due to path filters (1)
poetry.lockis excluded by!**/*.lock,!**/*.lock
📒 Files selected for processing (4)
- prediction_market_agent/agents/arbitrage_agent/data_models.py (1 hunks)
- prediction_market_agent/agents/arbitrage_agent/deploy.py (1 hunks)
- prediction_market_agent/agents/arbitrage_agent/prompt.py (1 hunks)
- tests/agents/arbitrage_agent/test_arbitrage_agent.py (1 hunks)
🧰 Additional context used
🔇 Additional comments (12)
tests/agents/arbitrage_agent/test_arbitrage_agent.py (5)
1-10: LGTM: Imports are well-organized and follow best practices.The imports are correctly structured, following the PEP 8 style guide. They include all necessary modules and classes for the test file.
13-24: LGTM: Well-implemented arbitrage_agent fixture.The fixture correctly mocks the
loadmethod and builds the agent's chain, ensuring isolated testing of theDeployableOmenArbitrageAgent.
27-31: LGTM: Market fixtures are well-defined for testing correlation scenarios.The main_market, related_market, and unrelated_market fixtures create appropriate mock OmenAgentMarket instances with relevant questions for testing market correlations.
Also applies to: 33-38, 40-45
48-63: LGTM: Well-structured and comprehensive test function.The
test_correlation_for_similar_marketsfunction is well-implemented:
- It uses parameterization to test both correlated and uncorrelated scenarios efficiently.
- The function correctly utilizes the fixtures to set up test cases.
- It tests the
calculate_correlation_between_marketsmethod of the arbitrage agent.- The assertion checks for the expected correlation result.
The test covers both positive (correlated) and negative (uncorrelated) cases, ensuring thorough testing of the correlation calculation functionality.
1-63: Great job on implementing comprehensive tests for the arbitrage agent!This test file is well-structured and follows best practices for Python testing:
- Proper use of fixtures for setting up test scenarios.
- Effective use of parameterization for testing multiple cases.
- Clear and descriptive test function that covers both correlated and uncorrelated markets.
- Good isolation of the arbitrage agent through mocking.
The tests provide solid coverage for the correlation calculation functionality of the arbitrage agent. This will help ensure the reliability and correctness of the agent's behavior in different market scenarios.
prediction_market_agent/agents/arbitrage_agent/data_models.py (3)
1-5: LGTM: Imports are appropriate and concise.The imports are well-chosen for the functionality implemented in this file. They include necessary components from typing, Pydantic, and the custom AgentMarket class.
7-10: LGTM: Correlation class is well-defined and addresses previous feedback.The Correlation class is simple and clear. The addition of the 'reasoning' field addresses the previous review comment, which will aid in debugging from langfuse traces.
55-71:⚠️ Potential issueHandle potential division by zero in
split_bet_amount_between_yes_and_noThe method correctly implements the splitting logic as described. However, there's still a risk of division by zero if the sum of probabilities is zero.
Add a check to ensure the denominator is not zero before performing the division:
def split_bet_amount_between_yes_and_no( self, total_bet_amount: float ) -> t.Tuple[float, float]: """Splits total bet amount following equations below: A1/p1 = A2/p2 (same profit regardless of outcome resolution) A1 + A2 = total bet amount """ + denominator = ( + self.market_to_bet_yes.current_p_yes + + self.market_to_bet_no.current_p_no + ) + if denominator == 0: + raise ValueError("Sum of probabilities is zero, cannot split bet amount") amount_to_bet_yes = ( total_bet_amount * self.market_to_bet_yes.current_p_yes - / ( - self.market_to_bet_yes.current_p_yes - + self.market_to_bet_no.current_p_no - ) + / denominator ) amount_to_bet_no = total_bet_amount - amount_to_bet_yes return amount_to_bet_yes, amount_to_bet_noThis change ensures that we don't attempt to divide by zero, which would cause a runtime error.
Likely invalid or redundant comment.
prediction_market_agent/agents/arbitrage_agent/deploy.py (4)
1-42: LGTM: Import statements are appropriate and well-organized.The import statements are comprehensive and relevant to the functionality of the
DeployableOmenArbitrageAgentclass. They include necessary modules from both external libraries and internal packages.
73-89: LGTM:_build_chainmethod is well-implementedThe
_build_chainmethod effectively constructs a language model chain using OpenAI's ChatGPT, a custom prompt template, and a Pydantic output parser. The use ofAPIKeys().openai_api_key_secretstr_v1for the API key is a good practice for keeping sensitive information secure.
1-179: Overall assessment: Well-structured implementation with room for improvementsThe
DeployableOmenArbitrageAgentclass provides a solid foundation for arbitrage trading on the Omen platform. The code is well-organized and implements complex logic for market correlation analysis and trade building. However, there are several areas where the implementation could be improved:
- Error handling: Add try-except blocks in methods making external API calls to improve robustness.
- Configurability: Make certain parameters (e.g., model, bet amounts) configurable to enhance flexibility.
- Parameter utilization: Incorporate
answerandexisting_positionparameters in the trade building logic for more informed decision-making.- Input validation: Ensure correct data types are passed to methods, especially when invoking the language model chain.
Addressing these points will significantly enhance the agent's reliability, flexibility, and effectiveness in arbitrage trading scenarios.
55-68:⚠️ Potential issueUse the input
limitparameter inget_marketsThe
get_marketsmethod currently ignores the inputlimitparameter and hardcodes it to 100. To improve flexibility, consider using the inputlimitparameter:return super().get_markets( market_type=market_type, - limit=100, + limit=limit, sort_by=SortBy.HIGHEST_LIQUIDITY, filter_by=FilterBy.OPEN, )This change allows the caller to control the number of markets fetched while still focusing on high liquidity markets.
Likely invalid or redundant comment.
| from prediction_market_agent.utils import APIKeys | ||
|
|
||
|
|
||
| class DeployableArbitrageAgent(DeployableTraderAgent): |
There was a problem hiding this comment.
Any reason you're subclassing DeployableTraderAgent instead of DeployableAgent? Not sure which is better, just asking
There was a problem hiding this comment.
I'm doing so because other trading agents (KnownOutcome, TTA and even coinflip) are subclassing the DeployableTraderAgent - as I understand,it does some langfuse magic and checks for some balances, which doesn't hurt.
There was a problem hiding this comment.
Actionable comments posted: 10
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (4)
- prediction_market_agent/agents/arbitrage_agent/data_models.py (1 hunks)
- prediction_market_agent/agents/arbitrage_agent/deploy.py (1 hunks)
- prediction_market_agent/run_agent.py (3 hunks)
- tests/agents/arbitrage_agent/test_arbitrage_agent.py (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- tests/agents/arbitrage_agent/test_arbitrage_agent.py
🧰 Additional context used
🔇 Additional comments (9)
prediction_market_agent/agents/arbitrage_agent/data_models.py (4)
1-14: LGTM: Well-structured data modelsThe imports and class definitions are appropriate and well-structured. The
CorrelationandCorrelatedMarketPairclasses extend Pydantic'sBaseModel, which is a good practice for data validation and serialization.
16-17: LGTM: Clear and concise__repr__methodThe
__repr__method provides a clear and informative string representation of theCorrelatedMarketPairinstance, showing the questions for both the main and related markets.
19-37: Remove LLM prompt reference and consider adding reasoningThe
potential_profit_per_bet_unitmethod is well-implemented and handles edge cases correctly. However, there are two points to address:
As mentioned in a previous review, remove the reference to the LLM prompt in the comment on line 28.
Consider adding a
reasoningfield to this method, as suggested in a past review comment. This would help with debugging from langfuse traces.Update the comment on line 28:
- # Negative correlations are not yet supported by the current LLM prompt, hence not handling those for now. + # Negative correlations are not yet supported, hence not handling those for now.Consider adding a
reasoningfield to provide context for the calculated profit:reasoning = f"Potential profit calculated based on p_yes={p_yes:.2f} and p_no={p_no:.2f}" return (1.0 / total_probability) - 1.0, reasoningThis change would require updating the return type and adjusting any code that calls this method.
39-46: LGTM: Correct implementation ofmarket_to_bet_yesThe
market_to_bet_yesmethod correctly selects the market with the lower "YES" probability, which is the appropriate logic for determining where to place a "YES" bet in an arbitrage scenario.prediction_market_agent/run_agent.py (3)
15-17: LGTM: Import statement for DeployableArbitrageAgent.The import statement for
DeployableArbitrageAgentis correctly added and follows the existing import style in the file.
80-80: LGTM: Addition of 'arbitrage' to RunnableAgent enum.The
arbitragemember is correctly added to theRunnableAgentenum, following the existing pattern and maintaining the current structure.
Line range hint
1-124: Summary: Successful integration of DeployableArbitrageAgentThe changes in this file successfully integrate the new
DeployableArbitrageAgentinto the existing agent framework. The modifications are consistent with the PR objectives and follow the established patterns in the file. The new agent is properly imported, added to theRunnableAgentenum, and included in theRUNNABLE_AGENTSdictionary.These changes enable the arbitrage agent to be recognized and executed within the system, allowing for YES/NO or NO/YES trades between correlated markets as described in the PR objectives.
No existing functionality appears to be modified or removed, minimizing the risk of unintended side effects. However, as with any new feature, thorough testing is recommended to ensure the arbitrage agent functions correctly within the broader system.
prediction_market_agent/agents/arbitrage_agent/deploy.py (2)
1-47: LGTM: Imports and class definition look good.The imports are comprehensive and well-organized. The class definition extends
DeployableTraderAgentand provides a clear docstring explaining its purpose.
107-138:⚠️ Potential issueFix arguments passed to
self.chain.invokeIn the
get_correlated_marketsmethod, you're passingmarketandrelated_marketobjects toself.chain.invoke, whereas the chain expects strings containing the market questions. Update the code as follows:- "main_market_question": market, - "related_market_question": related_market, + "main_market_question": market.question, + "related_market_question": related_market.question,This change ensures that the correct data is passed to the language model for correlation analysis.
Likely invalid or redundant comment.
There was a problem hiding this comment.
Actionable comments posted: 6
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (1)
- prediction_market_agent/agents/arbitrage_agent/deploy.py (1 hunks)
🧰 Additional context used
🔇 Additional comments (1)
prediction_market_agent/agents/arbitrage_agent/deploy.py (1)
125-131:⚠️ Potential issueFix arguments passed to
self.chain.invoke.The
self.chain.invokecall is passingmarketandrelated_marketobjects directly, but it should be passing theirquestionattributes instead.Apply this diff to fix the issue:
result: Correlation = self.chain.invoke( { - "main_market_question": market, - "related_market_question": related_market, + "main_market_question": market.question, + "related_market_question": related_market.question, }, config=get_langfuse_langchain_config(), )This change ensures that the correct data (the market questions) is passed to the language model for correlation analysis.
Likely invalid or redundant comment.
…trage-agent # Conflicts: # poetry.lock
There was a problem hiding this comment.
Actionable comments posted: 7
🧹 Outside diff range and nitpick comments (1)
prediction_market_agent/agents/arbitrage_agent/deploy.py (1)
80-82: Clarify the purpose of the answer_binary_market method.The
answer_binary_marketmethod currently returns a fixed 50% probability with full confidence for all markets. This doesn't provide any real market analysis. If this is a placeholder, consider adding a TODO comment explaining the intended future implementation.Example:
def answer_binary_market(self, market: AgentMarket) -> ProbabilisticAnswer | None: # TODO: Implement actual market analysis logic # Current implementation returns a placeholder value return ProbabilisticAnswer(p_yes=Probability(0.5), confidence=1.0)This would make it clear that the current implementation is temporary and needs to be replaced with actual analysis logic in the future.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
⛔ Files ignored due to path filters (2)
poetry.lockis excluded by!**/*.lock,!**/*.lockpyproject.tomlis excluded by!**/*.toml
📒 Files selected for processing (2)
- prediction_market_agent/agents/arbitrage_agent/deploy.py (1 hunks)
- prediction_market_agent/agents/think_thoroughly_agent/think_thoroughly_agent.py (0 hunks)
💤 Files with no reviewable changes (1)
- prediction_market_agent/agents/think_thoroughly_agent/think_thoroughly_agent.py
🧰 Additional context used
🔇 Additional comments (2)
prediction_market_agent/agents/arbitrage_agent/deploy.py (2)
1-43: LGTM: Import statements are well-organized and relevant.The import statements are appropriately grouped and include all necessary modules for the implementation of the DeployableArbitrageAgent class. There are no apparent issues or unused imports.
158-188: LGTM: build_trades_for_correlated_markets implementation looks good.The method correctly implements the logic for building trades for a pair of correlated markets. It appropriately splits the bet amount and creates Trade objects for both the main and related markets. The implementation is clear and follows the expected behavior for arbitrage trading.
| @@ -30,7 +30,7 @@ def answer_binary_market(self, market: AgentMarket) -> ProbabilisticAnswer | Non | |||
| ) | |||
|
|
|||
| def before_process_markets(self, market_type: MarketType) -> None: | |||
| self.agent.update_markets() | |||
| self.agent.pinecone_handler.update_markets() | |||
There was a problem hiding this comment.
according to https://github.com/gnosis/prediction-market-agent/pull/511/files#r1800902856 you're not intending to remove the closed markets from the pinecone db for the TTA. But you are changing the behaviour to only add open markets to the db (because PineconeHandler.insert_open_omen_markets_if_not_exists is called for both. Is this intended? how come?
There was a problem hiding this comment.
My understanding is this
-> We have closed (old) markets in the DB, since PineconeHandler has been running for a while - hence we should have all markets (up to the very fresh ones) in the DB
-> Now the current implementation inserts only open markets (if ID not in DB) - this makes sense because the closed markets are already in the DB, only fresh (and new ones) will be missing.
-> In the logic above, we could miss some short-lived markets, that are open-closed in the timespan of 2h (between agent runs). But I don't think these markets are very relevant.
-> Important thing here is to have the market ID (and its question title) on Pinecone for similarity_search - market liquidity and outcome token prices come from the subgraph (for a given market ID).
@evangriffiths does the above clear your concerns, or do you suggest a different implementation?
There was a problem hiding this comment.
-> In the logic above, we could miss some short-lived markets, that are open-closed in the timespan of 2h (between agent runs). But I don't think these markets are very relevant.
It's a strong and already a few times disproven assumption that our agent will run every 2 hours or that we won't need to turn them off even for the whole weekend or more. 🙈
Previous behavior (fetching all of them but having a reasonable limit for created_after to not fetch thousands of markets every time) seemed fine to me.
I see Evan was worried about
It only fetches the last 7 days' worth of created markets created_after = utcnow() - datetime.timedelta(days=7). Don't we want all open markets?
But there should be older markets in the DB as well right? Updated from previous runs. It's cumulating in DB.
On the other hand... both implementation are wrong only in edge-cases so I guess I don't care in the end... okay my 2 cents are finished 😄
There was a problem hiding this comment.
What if we have to nuke our pinecone db for some reason, so we have to start from scratch - we'd want to pull in all old markets right?
There was a problem hiding this comment.
Old markets are now being retrieved
| from prediction_market_agent_tooling.markets.omen.omen import OmenAgentMarket | ||
| from prediction_market_agent_tooling.markets.omen.omen_subgraph_handler import ( | ||
| OmenSubgraphHandler, | ||
| ) |
There was a problem hiding this comment.
As mentioned before, would be preferable IMO to not have any omen-specific dependencies. Should be easy enough to remove now!
There was a problem hiding this comment.
Actionable comments posted: 9
🧹 Outside diff range and nitpick comments (1)
prediction_market_agent/agents/think_thoroughly_agent/models.py (1)
15-17: LGTM with a suggestion: Consider adding a safety check.The update to
from_omen_marketmethod correctly includes the newclose_time_timestampfield. However, to improve robustness, consider adding a safety check for theclose_timeattribute.Here's a suggested improvement:
@staticmethod def from_omen_market(market: OmenMarket) -> "PineconeMetadata": return PineconeMetadata( question_title=market.question_title, market_address=market.id, close_time_timestamp=int(market.close_time.timestamp()) if market.close_time else None, )This change ensures that the method won't raise an AttributeError if
market.close_timeis None.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
⛔ Files ignored due to path filters (2)
poetry.lockis excluded by!**/*.lock,!**/*.lockpyproject.tomlis excluded by!**/*.toml
📒 Files selected for processing (4)
- prediction_market_agent/agents/arbitrage_agent/deploy.py (1 hunks)
- prediction_market_agent/agents/think_thoroughly_agent/models.py (1 hunks)
- prediction_market_agent/agents/think_thoroughly_agent/think_thoroughly_agent.py (0 hunks)
- prediction_market_agent/db/pinecone_handler.py (3 hunks)
💤 Files with no reviewable changes (1)
- prediction_market_agent/agents/think_thoroughly_agent/think_thoroughly_agent.py
🧰 Additional context used
🔇 Additional comments (8)
prediction_market_agent/agents/think_thoroughly_agent/models.py (2)
10-10: LGTM: New field added to enhance market data.The addition of
close_time_timestamp: intto thePineconeMetadataclass is a good improvement. It enhances the market data model by including the closing time, which can be crucial for various market analyses and operations.
Line range hint
1-33: Overall assessment: Good improvements to the market data model.The changes in this file enhance the
PineconeMetadataclass by adding aclose_time_timestampfield and updating thefrom_omen_marketmethod accordingly. These improvements align well with the PR objectives of enhancing market data handling and correlation functionalities. The changes are well-implemented, with only a minor suggestion for adding a safety check in thefrom_omen_marketmethod.prediction_market_agent/agents/arbitrage_agent/deploy.py (2)
143-143:⚠️ Potential issueReplace print statement with logger
Using
Apply this diff to fix the issue:
- print(f"Fetched {len(omen_markets)} related markets for market {market.id}") + logger.info(f"Fetched {len(omen_markets)} related markets for market {market.id}")Likely invalid or redundant comment.
146-152:⚠️ Potential issueFix arguments passed to self.chain.invoke
In the
get_correlated_marketsmethod, you're passingmarketandrelated_marketobjects toself.chain.invoke, whereas the chain expects strings containing the market questions.Apply this diff to fix the arguments:
- "main_market_question": market, - "related_market_question": related_market, + "main_market_question": market.question, + "related_market_question": related_market.question,This change ensures that the correct data is passed to the language model for correlation analysis.
Likely invalid or redundant comment.
prediction_market_agent/db/pinecone_handler.py (4)
16-16: Import Statement Addition ApprovedThe addition of the
DatetimeUTCimport is appropriate and necessary for handling timestamps in the updated methods.
74-74: Use ofget_existing_ids_in_indexThe refactoring to use
get_existing_ids_in_indeximproves code readability by encapsulating the ID retrieval logic.
79-82: Consider Usingdescribe_index_stats()for Accurate ID RetrievalThe current implementation of
get_existing_ids_in_indexusesself.index.list(), which may not reliably retrieve all existing IDs from the index. A previous review suggested usingself.index.describe_index_stats()for a more accurate approach.
115-119: Introduction ofupdate_marketsMethod ApprovedThe new
update_marketsmethod enhances code organization by consolidating the market update process. This improves maintainability and clarity.
| class DeployableArbitrageAgent(DeployableTraderAgent): | ||
| """Agent that places mirror bets on Omen for (quasi) risk-neutral profit.""" | ||
|
|
||
| model = "gpt-4o" | ||
| # trade amount will be divided between correlated markets. | ||
| total_trade_amount = BetAmount(amount=0.1, currency=OmenAgentMarket.currency) | ||
| bet_on_n_markets_per_run = 5 | ||
| max_related_markets_per_market = 10 |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Consider making class attributes configurable
The model, total_trade_amount, bet_on_n_markets_per_run, and max_related_markets_per_market attributes are hardcoded. To improve flexibility and allow for easier testing or adjustments, consider making these configurable through constructor parameters or a configuration file.
Example:
def __init__(self, model: str = "gpt-4o", total_trade_amount: float = 0.1,
bet_on_n_markets_per_run: int = 5, max_related_markets_per_market: int = 10):
self.model = model
self.total_trade_amount = BetAmount(amount=total_trade_amount, currency=OmenAgentMarket.currency)
self.bet_on_n_markets_per_run = bet_on_n_markets_per_run
self.max_related_markets_per_market = max_related_markets_per_marketThis change would allow users to easily adjust these parameters when instantiating the agent.
| def run(self, market_type: MarketType) -> None: | ||
| if market_type != MarketType.OMEN: | ||
| raise RuntimeError( | ||
| "Can arbitrage only on Omen since related markets embeddings available only for Omen markets." | ||
| ) | ||
| self.subgraph_handler = OmenSubgraphHandler() | ||
| self.pinecone_handler = PineconeHandler() | ||
| self.pinecone_handler.update_markets() | ||
| self.chain = self._build_chain() | ||
| super().run(market_type=market_type) |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Add error handling for initialization steps
The run method initializes important components (subgraph_handler, pinecone_handler, and chain) without any error handling. Consider wrapping these initializations in a try-except block to gracefully handle any potential errors during setup.
Example implementation:
def run(self, market_type: MarketType) -> None:
if market_type != MarketType.OMEN:
raise RuntimeError(
"Can arbitrage only on Omen since related markets embeddings available only for Omen markets."
)
try:
self.subgraph_handler = OmenSubgraphHandler()
self.pinecone_handler = PineconeHandler()
self.pinecone_handler.update_markets()
self.chain = self._build_chain()
except Exception as e:
logger.error(f"Error initializing arbitrage agent: {str(e)}")
raise
super().run(market_type=market_type)This change would improve the robustness of the initialization process and provide more informative error messages if something goes wrong.
| created_after=DatetimeUTC.fromtimestamp(start_timestamp) | ||
| if start_timestamp | ||
| else None, |
There was a problem hiding this comment.
Ensure Correct Handling of start_timestamp When Zero
The condition if start_timestamp may fail when start_timestamp is 0, a valid UNIX timestamp. This could result in created_after being set to None unintentionally. Consider checking if start_timestamp is not None to handle zero correctly.
Apply this diff to fix the condition:
created_after=DatetimeUTC.fromtimestamp(start_timestamp)
- if start_timestamp
+ if start_timestamp is not None
else None,📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| created_after=DatetimeUTC.fromtimestamp(start_timestamp) | |
| if start_timestamp | |
| else None, | |
| created_after=DatetimeUTC.fromtimestamp(start_timestamp) | |
| if start_timestamp is not None | |
| else None, |
| def insert_open_omen_markets_if_not_exists( | ||
| self, start_timestamp: int | None = None | ||
| ) -> None: | ||
| subgraph_handler = OmenSubgraphHandler() | ||
| markets = subgraph_handler.get_omen_binary_markets_simple( | ||
| limit=sys.maxsize, | ||
| filter_by=FilterBy.NONE, | ||
| created_after=DatetimeUTC.fromtimestamp(start_timestamp) | ||
| if start_timestamp | ||
| else None, |
There was a problem hiding this comment.
Potential Performance Issue with limit=sys.maxsize
Using limit=sys.maxsize in get_omen_binary_markets_simple may lead to performance issues if there is a large number of markets. Consider implementing pagination or processing the markets in batches to handle large datasets efficiently.
Consider applying this refactor to implement pagination:
def insert_open_omen_markets_if_not_exists(
self, start_timestamp: int | None = None
) -> None:
subgraph_handler = OmenSubgraphHandler()
- markets = subgraph_handler.get_omen_binary_markets_simple(
- limit=sys.maxsize,
- filter_by=FilterBy.NONE,
- created_after=DatetimeUTC.fromtimestamp(start_timestamp)
- if start_timestamp
- else None,
- sort_by=SortBy.NEWEST,
- )
+ markets = []
+ batch_size = 1000
+ offset = 0
+ while True:
+ batch = subgraph_handler.get_omen_binary_markets_simple(
+ limit=batch_size,
+ offset=offset,
+ filter_by=FilterBy.NONE,
+ created_after=DatetimeUTC.fromtimestamp(start_timestamp)
+ if start_timestamp is not None
+ else None,
+ sort_by=SortBy.NEWEST,
+ )
+ if not batch:
+ break
+ markets.extend(batch)
+ offset += batch_size📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| def insert_open_omen_markets_if_not_exists( | |
| self, start_timestamp: int | None = None | |
| ) -> None: | |
| subgraph_handler = OmenSubgraphHandler() | |
| markets = subgraph_handler.get_omen_binary_markets_simple( | |
| limit=sys.maxsize, | |
| filter_by=FilterBy.NONE, | |
| created_after=DatetimeUTC.fromtimestamp(start_timestamp) | |
| if start_timestamp | |
| else None, | |
| def insert_open_omen_markets_if_not_exists( | |
| self, start_timestamp: int | None = None | |
| ) -> None: | |
| subgraph_handler = OmenSubgraphHandler() | |
| markets = [] | |
| batch_size = 1000 | |
| offset = 0 | |
| while True: | |
| batch = subgraph_handler.get_omen_binary_markets_simple( | |
| limit=batch_size, | |
| offset=offset, | |
| filter_by=FilterBy.NONE, | |
| created_after=DatetimeUTC.fromtimestamp(start_timestamp) | |
| if start_timestamp is not None | |
| else None, | |
| sort_by=SortBy.NEWEST, | |
| ) | |
| if not batch: | |
| break | |
| markets.extend(batch) | |
| offset += batch_size |
| @@ -109,15 +112,22 @@ def deduplicate_markets(markets: list[OmenMarket]) -> list[OmenMarket]: | |||
|
|
|||
| return list(unique_market_titles.values()) | |||
|
|
|||
| def insert_all_omen_markets_if_not_exists( | |||
| self, created_after: DatetimeUTC | None = None | |||
| def update_markets(self) -> None: | |||
There was a problem hiding this comment.
- Can remove this function as it just wraps
insert_open_omen_markets_if_not_exists insert_all_omen_markets_if_not_existshas been changed toinsert_open_omen_markets_if_not_exists, but it's (nearly) the same as the old function (i.e. not getting open markets only). So no need for rename.created_after: DatetimeUTChas been changed tostart_timestamp: int. ButOmenSubgraphHandler.get_omen_binary_markets_simple(created_after=...)takesDatetimeUTC. So should change back
There was a problem hiding this comment.
3 points implemented here 9889efc
There was a problem hiding this comment.
We are the builders of PMAT, if you see something doesn't add up (like the need for returning the dummy prediction answer, or the need for over-ruling the arguments passed into get_markets function, let's think about how to design PMAT better, not hacking it around 🙏
| ) -> t.Sequence[AgentMarket]: | ||
| return super().get_markets( | ||
| market_type=market_type, | ||
| limit=50, |
There was a problem hiding this comment.
get_markets accepts the limit argument, but it's overruled by this. Not a nice design.
| limit=50, | |
| limit=limit, |
There was a problem hiding this comment.
I understand your concern (expressed in 4 separate comments) - 50 is not limit, we should use limit.
So what's the recommended way to provide a custom limit apart from overriding the method process_markets?
There was a problem hiding this comment.
For example, you could change the definition of get_markets (in PMAT) from
def get_markets(
self,
market_type: MarketType,
limit: int = MAX_AVAILABLE_MARKETS,
sort_by: SortBy = SortBy.CLOSING_SOONEST,
filter_by: FilterBy = FilterBy.OPEN,
) -> t.Sequence[AgentMarket]:
to just def get_markets(self) -> t.Sequence[AgentMarket], and then it's fine to overload the method like this.
Or you could introduce a new configuration variable, n_markets_to_fetch, similarly as we have bet_on_n_markets_per_run.
Or, like we have BettingStrategy, we could have GetMarketsStrategy.
No strong preferences from my side here.
There was a problem hiding this comment.
Or you could introduce a new configuration variable, n_markets_to_fetch, similarly as we have bet_on_n_markets_per_run.
Me and @evangriffiths also like this. Implementing this on PMAT and referencing the new release on this PR afterwards.
There was a problem hiding this comment.
Implementing this here on PMAT -> https://github.com/gnosis/prediction-market-agent-tooling/pull/532/files
There was a problem hiding this comment.
Implemented, removing get_markets. This thread can be closed IMO.
| return super().get_markets( | ||
| market_type=market_type, | ||
| limit=50, | ||
| sort_by=SortBy.HIGHEST_LIQUIDITY, |
| limit=50, | ||
| sort_by=SortBy.HIGHEST_LIQUIDITY, | ||
| # Fetching most liquid markets since more likely they will have related markets | ||
| filter_by=FilterBy.OPEN, |
Co-authored-by: Peter Jung <peter@jung.ninja>
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Outside diff range and nitpick comments (2)
prediction_market_agent/db/pinecone_handler.py (2)
115-119: Consider removing redundant methodAs pointed out in a previous review comment, this
update_marketsmethod appears to be just a wrapper forinsert_open_omen_markets_if_not_exists. If there's no additional functionality or future plans for this method, consider removing it to simplify the class structure.
120-129: LGTM with a minor suggestionThe changes to
insert_open_omen_markets_if_not_existsimprove flexibility by allowing filtering based on a start timestamp. The use ofDatetimeUTC.to_datetime_utc(start_timestamp)is correct.However, there's a minor issue with the condition:
if start_timestampThis condition will fail when
start_timestampis0, which is a valid UNIX timestamp. Consider changing it to:if start_timestamp is not NoneThis will correctly handle the case where
start_timestampis 0.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (1)
- prediction_market_agent/db/pinecone_handler.py (3 hunks)
🧰 Additional context used
🔇 Additional comments (2)
prediction_market_agent/db/pinecone_handler.py (2)
16-16: LGTM: Import path updateThe import statement for
DatetimeUTChas been updated, which is likely due to project restructuring. This change looks good and should maintain the correct import path.
74-77: LGTM: Improved code structureThe use of the new helper method
get_existing_ids_in_indexenhances code readability and maintainability. This refactoring maintains the same functionality while making the code more modular.
…bitrage-agent # Conflicts: # prediction_market_agent/db/pinecone_handler.py
There was a problem hiding this comment.
Actionable comments posted: 11
🧹 Outside diff range and nitpick comments (1)
prediction_market_agent/agents/arbitrage_agent/deploy.py (1)
55-59: Consider making the agent more market-agnostic.The current implementation is tightly coupled with Omen markets. To improve flexibility and potential reusability, consider designing the agent to be more market-agnostic. This could involve creating an interface for market handlers and allowing different market types to be plugged in.
For example, you could create a
MarketHandlerinterface with methods likeget_markets,insert_markets, etc., and then implement this interface for different market types (Omen, others). This would allow the agent to work with different market types in the future without significant changes to its core logic.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (3)
- prediction_market_agent/agents/arbitrage_agent/deploy.py (1 hunks)
- prediction_market_agent/agents/think_thoroughly_agent/deploy.py (1 hunks)
- prediction_market_agent/db/pinecone_handler.py (3 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- prediction_market_agent/db/pinecone_handler.py
🧰 Additional context used
🔇 Additional comments (1)
prediction_market_agent/agents/arbitrage_agent/deploy.py (1)
146-150:⚠️ Potential issueFix arguments passed to self.chain.invoke.
In the
get_correlated_marketsmethod, you're passingmarketandrelated_marketobjects toself.chain.invoke, whereas the chain expects strings containing the market questions.Update the code as follows:
result: Correlation = self.chain.invoke( { - "main_market_question": market, - "related_market_question": related_market, + "main_market_question": market.question, + "related_market_question": related_market.question, }, config=get_langfuse_langchain_config(), )This change ensures that the correct data is passed to the language model for correlation analysis.
Likely invalid or redundant comment.
| def answer_binary_market(self, market: AgentMarket) -> ProbabilisticAnswer | None: | ||
| return ProbabilisticAnswer(p_yes=Probability(0.5), confidence=1.0) |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Implement actual market analysis in answer_binary_market.
The answer_binary_market method currently returns a fixed 50% probability with full confidence for all markets. This doesn't provide any real market analysis. If this is a placeholder, consider adding a TODO comment explaining the intended future implementation.
Example:
def answer_binary_market(self, market: AgentMarket) -> ProbabilisticAnswer | None:
# TODO: Implement actual market analysis logic
# Current implementation returns a placeholder value
return ProbabilisticAnswer(p_yes=Probability(0.5), confidence=1.0)This would make it clear that the current implementation is temporary and needs to be replaced with actual analysis logic in the future.
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Outside diff range and nitpick comments (7)
tests/db/test_pinecone_handler.py (2)
40-42: Consider using realistic timestamp values in tests
The hardcoded close_time_timestamp=0 might not effectively test real-world scenarios. Consider using realistic future timestamps that match the question dates (e.g., July 2024) mentioned in the test data.
Here's a suggested improvement:
- close_time_timestamp=0,
+ # July 8, 2024 00:00:00 UTC
+ close_time_timestamp=1717977600,Line range hint 52-60: Add test cases for timestamp-related functionality
The current tests don't verify how close_time_timestamp affects the similarity search results. Consider adding test cases for:
- Markets with different closing times
- Expired markets (past timestamps)
- Invalid timestamp values
Would you like me to help generate these additional test cases?
prediction_market_agent/agents/metaculus_agent/deploy.py (1)
85-87: Consider documenting the standardized prediction storage pattern.
The switch to ProcessedMarket and APIKeys appears to be part of a broader standardization effort. This pattern should be consistently applied across all agents, including the new arbitrage agent.
Consider:
- Adding documentation about this standard pattern
- Creating a base class that enforces this pattern
- Updating the README to reflect these architectural changes
scripts/agent_app.py (1)
Line range hint 1-184: Consider enhancing UI for arbitrage agent scenarios.
Since this PR introduces an arbitrage agent for correlated markets, the Streamlit UI could be enhanced to better support arbitrage scenarios. Consider:
- Adding UI elements to display correlated market pairs
- Showing potential arbitrage opportunities
- Visualizing the expected profit calculations
Example enhancement for the agent_app function:
def agent_app() -> None:
# ... existing code ...
# Add UI elements for arbitrage scenarios when arbitrage agent is selected
if any("ArbitrageAgent" in name for name in agent_class_names):
st.subheader("Arbitrage Settings")
show_correlated_markets = st.checkbox("Show correlated markets", value=False)
if show_correlated_markets:
st.info("Displaying markets with correlation > 0.8")
# Add logic to display correlated market pairs
# Add expected profit calculationsprediction_market_agent/agents/known_outcome_agent/benchmark.py (1)
Line range hint 91-130: Consider parameterizing test data for better maintainability.
The test data includes hardcoded dates and real-world events from March 2024. These will become outdated quickly. Consider:
- Moving test data to a separate fixture file
- Using relative dates (e.g., "tomorrow", "next year") instead of specific dates
- Using more generic, timeless examples that don't reference specific events
Example implementation:
# test_data.py
from datetime import timedelta
from prediction_market_agent_tooling.tools.utils import utcnow
def get_test_questions():
tomorrow = utcnow() + timedelta(days=1)
next_year = utcnow() + timedelta(days=365)
return [
QuestionWithKnownOutcome(
question=f"Will Event A occur by {tomorrow.strftime('%d %B %Y')}?",
result=Result.YES,
notes="Generic example for known positive outcome",
),
# ... more generic test cases
]prediction_market_agent/agents/arbitrage_agent/deploy.py (2)
39-41: Enhance the class docstring.
The current docstring is minimal. Consider expanding it to include:
- Detailed description of the agent's purpose
- Key features and capabilities
- Requirements and limitations
- Example usage
- """Agent that places mirror bets on Omen for (quasi) risk-neutral profit."""
+ """An agent that identifies and exploits arbitrage opportunities in correlated prediction markets.
+
+ This agent:
+ - Identifies correlated markets using semantic similarity and LLM-based correlation analysis
+ - Places mirror bets (YES/NO or NO/YES) on correlated markets for risk-neutral profit
+ - Operates only on Omen markets due to the requirement of market embeddings
+
+ Requirements:
+ - Pinecone API access for market similarity search
+ - OpenAI API access for correlation analysis
+
+ Example:
+ agent = DeployableArbitrageAgent()
+ agent.run(market_type=MarketType.OMEN)
+ """60-61: Document placeholder implementation.
The method returns a fixed probability without actual market analysis.
def answer_binary_market(self, market: AgentMarket) -> ProbabilisticAnswer | None:
+ # TODO: This is a placeholder implementation.
+ # The arbitrage agent doesn't use market probabilities directly,
+ # as it relies on correlation analysis and price differences between markets.
return ProbabilisticAnswer(p_yes=Probability(0.5), confidence=1.0)📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
⛔ Files ignored due to path filters (2)
poetry.lockis excluded by!**/*.lock,!**/*.lockpyproject.tomlis excluded by!**/*.toml
📒 Files selected for processing (10)
- prediction_market_agent/agents/arbitrage_agent/deploy.py (1 hunks)
- prediction_market_agent/agents/known_outcome_agent/benchmark.py (2 hunks)
- prediction_market_agent/agents/metaculus_agent/deploy.py (2 hunks)
- prediction_market_agent/agents/think_thoroughly_agent/benchmark.py (2 hunks)
- prediction_market_agent/agents/think_thoroughly_agent/deploy.py (2 hunks)
- prediction_market_agent/agents/think_thoroughly_agent/models.py (1 hunks)
- prediction_market_agent/db/pinecone_handler.py (4 hunks)
- scripts/agent_app.py (1 hunks)
- tests/agents/arbitrage_agent/test_arbitrage_agent.py (1 hunks)
- tests/db/test_pinecone_handler.py (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (4)
- prediction_market_agent/agents/think_thoroughly_agent/deploy.py
- prediction_market_agent/agents/think_thoroughly_agent/models.py
- prediction_market_agent/db/pinecone_handler.py
- tests/agents/arbitrage_agent/test_arbitrage_agent.py
🧰 Additional context used
📓 Learnings (1)
prediction_market_agent/agents/metaculus_agent/deploy.py (1)
Learnt from: kongzii
PR: gnosis/prediction-market-agent#529
File: prediction_market_agent/agents/metaculus_agent/deploy.py:71-75
Timestamp: 2024-10-24T08:53:49.415Z
Learning: Ensure that code suggestions result in meaningful changes before proposing them.
🔇 Additional comments (12)
tests/db/test_pinecone_handler.py (1)
40-42: Verify timestamp handling across different timezones
Since the questions reference specific dates (e.g., "by 5 July 2024", "on 8 July 2024"), it's important to ensure the timestamp handling works correctly across different timezones.
✅ Verification successful
Based on the search results, I can now generate a final response regarding the timestamp handling verification:
Timestamp handling is correctly implemented with UTC standardization
The codebase consistently uses UTC for timestamp handling through:
- Standardized
DatetimeUTCtype andutcnow()utility function - Explicit UTC datetime creation via
utc_datetime()function - Proper timestamp storage and conversion in database models
The test file's use of timestamp=0 is just a test fixture and doesn't reflect the actual timestamp handling in production code.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Search for timezone-related handling in the codebase
rg -A 5 "timestamp|timezone|tzinfo|datetime" --type python
Length of output: 91
Script:
#!/bin/bash
# Search for timezone-related handling in the codebase using ripgrep without file type restriction
rg -A 5 "timestamp|timezone|tzinfo|datetime"
Length of output: 51614
prediction_market_agent/agents/metaculus_agent/deploy.py (2)
4-4: LGTM: Import changes are well-organized.
The new imports for APIKeys and ProcessedMarket are properly organized and support the updated prediction storage mechanism.
Also applies to: 8-12
85-87:
Add error handling and secure API key management.
The current implementation has potential security and reliability concerns:
- Creating
APIKeys()without parameters might not be secure or could fail if keys are required - No error handling for potential API authentication failures
Let's verify the APIKeys implementation:
Consider implementing the following improvements:
- market.store_prediction(
- processed_market=processed_market, keys=APIKeys()
- )
+ try:
+ keys = APIKeys.from_environment() # Or another secure method
+ market.store_prediction(
+ processed_market=processed_market,
+ keys=keys
+ )
+ except (ValueError, AuthenticationError) as e:
+ logger.error(f"Failed to store prediction: {e}")
+ continueprediction_market_agent/agents/think_thoroughly_agent/benchmark.py (2)
22-22: LGTM!
The import of MarketFees is correctly placed with other market-related imports and is necessary for setting market fees in the benchmark environment.
48-48: LGTM! Setting zero fees is appropriate for benchmarking.
Setting fees=MarketFees.get_zero_fees() ensures that market fees don't affect the benchmark results, providing cleaner performance metrics for agent evaluation.
scripts/agent_app.py (1)
81-81: Parameter rename aligns with trading terminology.
The rename from place_bet to place_trades better reflects the nature of the operations, especially with the introduction of the arbitrage agent that performs correlated market trades.
Let's verify the consistency of this parameter name change across the codebase:
✅ Verification successful
Parameter rename is consistent with trading terminology
The change from place_bet to place_trades is verified to be consistent across the codebase:
- No remaining instances of
place_betwere found place_tradesparameter is used consistently in agent instantiations inscripts/agent_app.pyandprediction_market_agent/agents/think_thoroughly_agent/deploy.py- The parameter aligns with the trading-focused nature of the agents, especially with the new arbitrage agent functionality
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Check for any remaining instances of 'place_bet' that might need updating
# and verify the consistent usage of 'place_trades'
echo "Checking for any remaining 'place_bet' instances:"
rg "place_bet" -l
echo -e "\nVerifying 'place_trades' usage in agent instantiations:"
rg "place_trades\s*=" -l
Length of output: 377
Script:
#!/bin/bash
# Let's examine the actual usage context of place_trades in both files
echo "Content of place_trades in agent_app.py:"
rg "place_trades" -B 2 -A 2 scripts/agent_app.py
echo -e "\nContent of place_trades in deploy.py:"
rg "place_trades" -B 2 -A 2 prediction_market_agent/agents/think_thoroughly_agent/deploy.py
# Let's also check for any related test files that might need updates
echo -e "\nChecking for related test files:"
fd "test.*\.py$" --exec rg "place_bet|place_trades" -l {}
Length of output: 727
Script:
#!/bin/bash
# Let's check the agent class definitions to understand the parameter usage
echo "Checking AgentClass definition and inheritance:"
ast-grep --pattern 'class $_ {
$$$
def __init__($$$) {
$$$
}
$$$
}'
echo -e "\nChecking for any agent base classes:"
rg "class.*Agent.*\(" -A 5
Length of output: 19004
Script:
#!/bin/bash
# Let's check the base trader agent class to understand the parameter inheritance
echo "Checking DeployableTraderAgent class definition:"
rg "class DeployableTraderAgent" -A 10 -B 2
echo -e "\nChecking for place_trades parameter in __init__ methods:"
ast-grep --pattern 'def __init__($$$place_trades$$$) {
$$$
}'
Length of output: 1474
prediction_market_agent/agents/known_outcome_agent/benchmark.py (2)
14-14: LGTM: Clean import addition.
The import follows the existing style and is appropriately placed with other market-related imports.
48-48: Verify consistent zero fees usage across agent markets.
The addition of zero fees is logical for benchmarking purposes. However, let's verify that this approach is consistently applied across other agent market implementations.
✅ Verification successful
Zero fees consistently used in agent market benchmarks
Verification confirms that MarketFees.get_zero_fees() is consistently used in both agent market implementations:
prediction_market_agent/agents/known_outcome_agent/benchmark.pyprediction_market_agent/agents/think_thoroughly_agent/benchmark.py
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Check for consistent usage of MarketFees.get_zero_fees() across agent market implementations
# Test: Search for AgentMarket instantiations and verify fees parameter usage
# Look for AgentMarket instantiations
ast-grep --pattern 'AgentMarket($$$)'
# Look for MarketFees usage
rg -A 2 'MarketFees.get_zero_fees'
Length of output: 3786
prediction_market_agent/agents/arbitrage_agent/deploy.py (4)
143-174: LGTM! Well-structured implementation.
The method correctly builds trades for correlated markets with appropriate logging.
80-90:
Add error handling for chain invocation.
The method should handle potential chain invocation failures gracefully.
@observe()
def calculate_correlation_between_markets(
self, market: AgentMarket, related_market: AgentMarket
- ) -> Correlation:
+ ) -> Correlation | None:
+ """Calculate correlation between two markets using LLM analysis.
+
+ Args:
+ market: The main market
+ related_market: The market to compare with
+
+ Returns:
+ Correlation object if successful, None if analysis fails
+ """
+ try:
correlation: Correlation = self.chain.invoke(
{
"main_market_question": market.question,
"related_market_question": related_market.question,
}
)
return correlation
+ except Exception as e:
+ logger.error(
+ f"Failed to calculate correlation between markets "
+ f"{market.id} and {related_market.id}: {str(e)}"
+ )
+ return NoneLikely invalid or redundant comment.
63-78:
Add error handling and make LLM parameters configurable.
The method lacks error handling and has hardcoded LLM parameters.
- def _build_chain(self) -> RunnableSerializable[t.Any, t.Any]:
+ def _build_chain(self, temperature: float = 0) -> RunnableSerializable[t.Any, t.Any]:
+ """Build the LangChain chain for correlation analysis.
+
+ Args:
+ temperature: Controls randomness in the model's output (0 to 1)
+
+ Returns:
+ A chain that processes market questions and returns correlation analysis
+
+ Raises:
+ RuntimeError: If chain initialization fails
+ """
+ try:
llm = ChatOpenAI(
- temperature=0,
+ temperature=temperature,
model=self.model,
api_key=APIKeys().openai_api_key_secretstr_v1,
)
parser = PydanticOutputParser(pydantic_object=Correlation)
prompt = PromptTemplate(
template=PROMPT_TEMPLATE,
input_variables=["main_market_question", "related_market_question"],
partial_variables={"format_instructions": parser.get_format_instructions()},
)
chain = prompt | llm | parser
return chain
+ except Exception as e:
+ logger.error(f"Failed to build correlation chain: {str(e)}")
+ raise RuntimeError(f"Chain initialization failed: {str(e)}") from eLikely invalid or redundant comment.
42-47: 🛠️ Refactor suggestion
Make configuration parameters customizable.
The class has several hardcoded values that should be configurable through constructor parameters to improve flexibility and testability.
- model = "gpt-4o"
- total_trade_amount = BetAmount(amount=0.1, currency=OmenAgentMarket.currency)
- bet_on_n_markets_per_run = 5
- max_related_markets_per_market = 10
- n_markets_to_fetch = 50
+ def __init__(
+ self,
+ model: str = "gpt-4o",
+ total_trade_amount: float = 0.1,
+ bet_on_n_markets_per_run: int = 5,
+ max_related_markets_per_market: int = 10,
+ n_markets_to_fetch: int = 50,
+ ):
+ """Initialize the arbitrage agent with custom configuration.
+
+ Args:
+ model: The OpenAI model to use for correlation analysis
+ total_trade_amount: Total amount to be divided between correlated markets
+ bet_on_n_markets_per_run: Maximum number of markets to bet on per run
+ max_related_markets_per_market: Maximum number of related markets to fetch per market
+ n_markets_to_fetch: Total number of markets to fetch from the chain
+ """
+ super().__init__()
+ self.model = model
+ self.total_trade_amount = BetAmount(amount=total_trade_amount, currency=OmenAgentMarket.currency)
+ self.bet_on_n_markets_per_run = bet_on_n_markets_per_run
+ self.max_related_markets_per_market = max_related_markets_per_market
+ self.n_markets_to_fetch = n_markets_to_fetchLikely invalid or redundant comment.
| @observe() | ||
| def build_trades( | ||
| self, | ||
| market: AgentMarket, | ||
| answer: ProbabilisticAnswer, | ||
| existing_position: Position | None, | ||
| ) -> list[Trade]: | ||
| trades = [] | ||
| correlated_markets = self.get_correlated_markets(market=market) | ||
| for pair in correlated_markets: | ||
| # We want to profit at least 0.5% per market (value chosen as initial baseline). | ||
| if pair.potential_profit_per_bet_unit() > 0.005: | ||
| trades_for_pair = self.build_trades_for_correlated_markets(pair) | ||
| trades.extend(trades_for_pair) | ||
|
|
||
| return trades |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Make profit threshold configurable and address unused parameters.
The method has a hardcoded profit threshold and unused parameters.
@observe()
def build_trades(
self,
market: AgentMarket,
- answer: ProbabilisticAnswer,
- existing_position: Position | None,
+ min_profit_threshold: float = 0.005,
) -> list[Trade]:
+ """Build trades for correlated markets with sufficient profit potential.
+
+ Args:
+ market: The market to build trades for
+ min_profit_threshold: Minimum profit threshold (default: 0.5%)
+
+ Returns:
+ List of trades to execute
+ """
trades = []
correlated_markets = self.get_correlated_markets(market=market)
for pair in correlated_markets:
- # We want to profit at least 0.5% per market (value chosen as initial baseline).
- if pair.potential_profit_per_bet_unit() > 0.005:
+ if pair.potential_profit_per_bet_unit() > min_profit_threshold:
trades_for_pair = self.build_trades_for_correlated_markets(pair)
trades.extend(trades_for_pair)
return trades📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| @observe() | |
| def build_trades( | |
| self, | |
| market: AgentMarket, | |
| answer: ProbabilisticAnswer, | |
| existing_position: Position | None, | |
| ) -> list[Trade]: | |
| trades = [] | |
| correlated_markets = self.get_correlated_markets(market=market) | |
| for pair in correlated_markets: | |
| # We want to profit at least 0.5% per market (value chosen as initial baseline). | |
| if pair.potential_profit_per_bet_unit() > 0.005: | |
| trades_for_pair = self.build_trades_for_correlated_markets(pair) | |
| trades.extend(trades_for_pair) | |
| return trades | |
| @observe() | |
| def build_trades( | |
| self, | |
| market: AgentMarket, | |
| min_profit_threshold: float = 0.005, | |
| ) -> list[Trade]: | |
| """Build trades for correlated markets with sufficient profit potential. | |
| Args: | |
| market: The market to build trades for | |
| min_profit_threshold: Minimum profit threshold (default: 0.5%) | |
| Returns: | |
| List of trades to execute | |
| """ | |
| trades = [] | |
| correlated_markets = self.get_correlated_markets(market=market) | |
| for pair in correlated_markets: | |
| if pair.potential_profit_per_bet_unit() > min_profit_threshold: | |
| trades_for_pair = self.build_trades_for_correlated_markets(pair) | |
| trades.extend(trades_for_pair) | |
| return trades |
| @observe() | ||
| def get_correlated_markets(self, market: AgentMarket) -> list[CorrelatedMarketPair]: | ||
| # We try to find similar, open markets which point to the same outcome. | ||
| correlated_markets = [] | ||
| # We only wanted to find related markets that are open. | ||
| # We intentionally query more markets in the hope it yields open markets. | ||
| # We could store market_status (open, closed) in Pinecone, but we refrain from it | ||
| # to keep the chain data (or graph) as the source-of-truth, instead of managing the | ||
| # update process of the vectorDB. | ||
|
|
||
| related = self.pinecone_handler.find_nearest_questions_with_threshold( | ||
| limit=self.max_related_markets_per_market, | ||
| text=market.question, | ||
| filter_on_metadata={ | ||
| "close_time_timestamp": {"gte": utcnow().timestamp() + 3600} | ||
| }, # closing 1h from now | ||
| ) | ||
|
|
||
| omen_markets = self.subgraph_handler.get_omen_binary_markets( | ||
| limit=len(related), | ||
| id_in=[i.market_address for i in related if i.market_address != market.id], | ||
| resolved=False, | ||
| ) | ||
|
|
||
| # Order omen_markets in the same order as related | ||
| related_market_addresses = [i.market_address for i in related] | ||
| omen_markets = sorted( | ||
| omen_markets, key=lambda m: related_market_addresses.index(m.id) | ||
| ) | ||
|
|
||
| print(f"Fetched {len(omen_markets)} related markets for market {market.id}") | ||
|
|
||
| for related_market in omen_markets: | ||
| result: Correlation = self.chain.invoke( | ||
| { | ||
| "main_market_question": market, | ||
| "related_market_question": related_market, | ||
| }, | ||
| config=get_langfuse_langchain_config(), | ||
| ) | ||
| if result.near_perfect_correlation is not None: | ||
| related_agent_market = OmenAgentMarket.from_data_model(related_market) | ||
| correlated_markets.append( | ||
| CorrelatedMarketPair( | ||
| main_market=market, | ||
| related_market=related_agent_market, | ||
| correlation=result, | ||
| ) | ||
| ) | ||
| return correlated_markets | ||
|
|
There was a problem hiding this comment.
Fix multiple issues in get_correlated_markets method.
The method has several issues that need to be addressed:
- Using print instead of logger
- Incorrect arguments to chain.invoke
- Missing error handling for API calls
@observe()
def get_correlated_markets(self, market: AgentMarket) -> list[CorrelatedMarketPair]:
+ """Find markets that are correlated with the given market.
+
+ Args:
+ market: The market to find correlations for
+
+ Returns:
+ List of correlated market pairs
+ """
correlated_markets = []
+ try:
related = self.pinecone_handler.find_nearest_questions_with_threshold(
limit=self.max_related_markets_per_market,
text=market.question,
filter_on_metadata={
"close_time_timestamp": {"gte": utcnow().timestamp() + 3600}
},
)
+ except Exception as e:
+ logger.error(f"Failed to fetch related markets from Pinecone: {str(e)}")
+ return []
+ try:
omen_markets = self.subgraph_handler.get_omen_binary_markets(
limit=len(related),
id_in=[i.market_address for i in related if i.market_address != market.id],
resolved=False,
)
+ except Exception as e:
+ logger.error(f"Failed to fetch markets from Omen subgraph: {str(e)}")
+ return []
related_market_addresses = [i.market_address for i in related]
omen_markets = sorted(
omen_markets, key=lambda m: related_market_addresses.index(m.id)
)
- print(f"Fetched {len(omen_markets)} related markets for market {market.id}")
+ logger.info(f"Fetched {len(omen_markets)} related markets for market {market.id}")
for related_market in omen_markets:
+ try:
result: Correlation = self.chain.invoke(
{
- "main_market_question": market,
- "related_market_question": related_market,
+ "main_market_question": market.question,
+ "related_market_question": related_market.question,
},
config=get_langfuse_langchain_config(),
)
+ except Exception as e:
+ logger.error(f"Failed to calculate correlation: {str(e)}")
+ continue
if result.near_perfect_correlation is not None:
related_agent_market = OmenAgentMarket.from_data_model(related_market)
correlated_markets.append(
CorrelatedMarketPair(
main_market=market,
related_market=related_agent_market,
correlation=result,
)
)
return correlated_markets📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| @observe() | |
| def get_correlated_markets(self, market: AgentMarket) -> list[CorrelatedMarketPair]: | |
| # We try to find similar, open markets which point to the same outcome. | |
| correlated_markets = [] | |
| # We only wanted to find related markets that are open. | |
| # We intentionally query more markets in the hope it yields open markets. | |
| # We could store market_status (open, closed) in Pinecone, but we refrain from it | |
| # to keep the chain data (or graph) as the source-of-truth, instead of managing the | |
| # update process of the vectorDB. | |
| related = self.pinecone_handler.find_nearest_questions_with_threshold( | |
| limit=self.max_related_markets_per_market, | |
| text=market.question, | |
| filter_on_metadata={ | |
| "close_time_timestamp": {"gte": utcnow().timestamp() + 3600} | |
| }, # closing 1h from now | |
| ) | |
| omen_markets = self.subgraph_handler.get_omen_binary_markets( | |
| limit=len(related), | |
| id_in=[i.market_address for i in related if i.market_address != market.id], | |
| resolved=False, | |
| ) | |
| # Order omen_markets in the same order as related | |
| related_market_addresses = [i.market_address for i in related] | |
| omen_markets = sorted( | |
| omen_markets, key=lambda m: related_market_addresses.index(m.id) | |
| ) | |
| print(f"Fetched {len(omen_markets)} related markets for market {market.id}") | |
| for related_market in omen_markets: | |
| result: Correlation = self.chain.invoke( | |
| { | |
| "main_market_question": market, | |
| "related_market_question": related_market, | |
| }, | |
| config=get_langfuse_langchain_config(), | |
| ) | |
| if result.near_perfect_correlation is not None: | |
| related_agent_market = OmenAgentMarket.from_data_model(related_market) | |
| correlated_markets.append( | |
| CorrelatedMarketPair( | |
| main_market=market, | |
| related_market=related_agent_market, | |
| correlation=result, | |
| ) | |
| ) | |
| return correlated_markets | |
| @observe() | |
| def get_correlated_markets(self, market: AgentMarket) -> list[CorrelatedMarketPair]: | |
| """Find markets that are correlated with the given market. | |
| Args: | |
| market: The market to find correlations for | |
| Returns: | |
| List of correlated market pairs | |
| """ | |
| correlated_markets = [] | |
| try: | |
| related = self.pinecone_handler.find_nearest_questions_with_threshold( | |
| limit=self.max_related_markets_per_market, | |
| text=market.question, | |
| filter_on_metadata={ | |
| "close_time_timestamp": {"gte": utcnow().timestamp() + 3600} | |
| }, # closing 1h from now | |
| ) | |
| except Exception as e: | |
| logger.error(f"Failed to fetch related markets from Pinecone: {str(e)}") | |
| return [] | |
| try: | |
| omen_markets = self.subgraph_handler.get_omen_binary_markets( | |
| limit=len(related), | |
| id_in=[i.market_address for i in related if i.market_address != market.id], | |
| resolved=False, | |
| ) | |
| except Exception as e: | |
| logger.error(f"Failed to fetch markets from Omen subgraph: {str(e)}") | |
| return [] | |
| # Order omen_markets in the same order as related | |
| related_market_addresses = [i.market_address for i in related] | |
| omen_markets = sorted( | |
| omen_markets, key=lambda m: related_market_addresses.index(m.id) | |
| ) | |
| logger.info(f"Fetched {len(omen_markets)} related markets for market {market.id}") | |
| for related_market in omen_markets: | |
| try: | |
| result: Correlation = self.chain.invoke( | |
| { | |
| "main_market_question": market.question, | |
| "related_market_question": related_market.question, | |
| }, | |
| config=get_langfuse_langchain_config(), | |
| ) | |
| except Exception as e: | |
| logger.error(f"Failed to calculate correlation: {str(e)}") | |
| continue | |
| if result.near_perfect_correlation is not None: | |
| related_agent_market = OmenAgentMarket.from_data_model(related_market) | |
| correlated_markets.append( | |
| CorrelatedMarketPair( | |
| main_market=market, | |
| related_market=related_agent_market, | |
| correlation=result, | |
| ) | |
| ) | |
| return correlated_markets |
…trage-agent # Conflicts: # poetry.lock # prediction_market_agent/agents/metaculus_agent/deploy.py # prediction_market_agent/agents/think_thoroughly_agent/deploy.py # prediction_market_agent/run_agent.py # pyproject.toml # scripts/agent_app.py
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Outside diff range and nitpick comments (1)
prediction_market_agent/agents/think_thoroughly_agent/benchmark.py (1)
Line range hint 34-47: Consider making market fees configurable for realistic benchmarking.
While using zero fees (MarketFees.get_zero_fees()) is suitable for basic testing, it might not accurately represent real-world scenarios where fees impact arbitrage opportunities. Consider making the fees configurable through the function parameters to enable testing with different fee structures.
Here's a suggested implementation:
-def build_binary_agent_market_from_question(question: str) -> AgentMarket:
+def build_binary_agent_market_from_question(
+ question: str,
+ fees: t.Optional[MarketFees] = None
+) -> AgentMarket:
return AgentMarket(
id=question,
description=None,
url=question,
close_time=utcnow() + timedelta(days=1),
volume=None,
question=question,
current_p_yes=Probability(0.5),
created_time=utc_datetime(2024, 1, 1),
resolution=None,
outcomes=["YES", "NO"],
outcome_token_pool=None,
- fees=MarketFees.get_zero_fees(),
+ fees=fees if fees is not None else MarketFees.get_zero_fees(),
)🧰 Tools
🪛 Ruff
23-23: Redefinition of unused MarketFees from line 20
(F811)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (4)
- prediction_market_agent/agents/think_thoroughly_agent/benchmark.py (1 hunks)
- prediction_market_agent/agents/think_thoroughly_agent/deploy.py (1 hunks)
- prediction_market_agent/agents/think_thoroughly_agent/think_thoroughly_agent.py (0 hunks)
- prediction_market_agent/run_agent.py (3 hunks)
💤 Files with no reviewable changes (1)
- prediction_market_agent/agents/think_thoroughly_agent/think_thoroughly_agent.py
🚧 Files skipped from review as they are similar to previous changes (1)
- prediction_market_agent/agents/think_thoroughly_agent/deploy.py
🧰 Additional context used
🪛 Ruff
prediction_market_agent/agents/think_thoroughly_agent/benchmark.py
23-23: Redefinition of unused MarketFees from line 20
(F811)
🔇 Additional comments (3)
prediction_market_agent/run_agent.py (3)
15-17: LGTM: Import statement properly added.
The import statement for DeployableArbitrageAgent has been correctly added, resolving the previously identified issue.
84-84: LGTM: RunnableAgent enum properly updated.
The addition of the arbitrage enum value follows the established naming conventions and structure.
112-112: LGTM: Agent mapping correctly configured.
The DeployableArbitrageAgent is properly mapped to RunnableAgent.arbitrage in the RUNNABLE_AGENTS dictionary.
Let's verify the integration:
✅ Verification successful
Mapping and implementation verified successfully
The verification confirms that:
DeployableArbitrageAgentis properly defined inprediction_market_agent/agents/arbitrage_agent/deploy.py- It correctly inherits from
DeployableTraderAgent - The mapping in
run_agent.pyis accurate:RunnableAgent.arbitrage: DeployableArbitrageAgent - The agent has a complete implementation with required methods like
run()and proper test coverage
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Verify the complete integration of DeployableArbitrageAgent
# Test 1: Verify the agent class exists and inherits from DeployableAgent
echo "Checking agent class definition:"
ast-grep --pattern 'class DeployableArbitrageAgent($$$): $$$'
# Test 2: Verify the agent can be instantiated and run
echo "Checking run method implementation:"
ast-grep --pattern 'def run(self, market_type: MarketType) -> None: $$$'
# Test 3: Check for any direct usage of the agent
echo "Checking agent usage:"
rg -A 2 'DeployableArbitrageAgent' --type python
Length of output: 20990
Script:
#!/bin/bash
# Let's verify the agent usage and mapping
echo "Checking agent mapping and usage:"
rg -A 2 'DeployableArbitrageAgent'
Length of output: 2089
| @@ -20,6 +20,7 @@ | |||
| MarketFees, | |||
| SortBy, | |||
| ) | |||
| from prediction_market_agent_tooling.markets.market_fees import MarketFees | |||
There was a problem hiding this comment.
Remove duplicate import of MarketFees.
The MarketFees class is already imported through the agent_market module on line 20.
Apply this diff to fix the duplicate import:
-from prediction_market_agent_tooling.markets.market_fees import MarketFees📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| from prediction_market_agent_tooling.markets.market_fees import MarketFees |
🧰 Tools
🪛 Ruff
23-23: Redefinition of unused MarketFees from line 20
(F811)
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Outside diff range and nitpick comments (2)
prediction_market_agent/agents/specialized_agent/deploy.py (1)
Line range hint 31-49: Consider enhancing market fetching for arbitrage scenarios.
Since this PR introduces arbitrage functionality, consider adding support for fetching correlated markets. This could involve:
- Adding methods to fetch markets with potential correlations
- Supporting filtering by market characteristics that are relevant for arbitrage
Would you like assistance in designing these enhancements? This could help integrate the arbitrage agent's requirements with the existing market fetching infrastructure.
prediction_market_agent/agents/invalid_agent/deploy.py (1)
Line range hint 1-50: Consider documenting agent interaction strategy.
Given that this PR introduces a new arbitrage agent that will coexist with the InvalidAgent, it would be valuable to document how these agents coordinate their activities, particularly in scenarios where a market might be both invalid and part of a correlated pair.
Consider addressing these points:
- How do we prevent conflicting bets between invalid and arbitrage agents?
- Should the arbitrage agent skip markets that are likely to be invalid?
- How does the trade interval coordination work between different agent types?
Would you like me to help draft documentation covering these interaction patterns?
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (2)
- prediction_market_agent/agents/invalid_agent/deploy.py (1 hunks)
- prediction_market_agent/agents/specialized_agent/deploy.py (1 hunks)
🔇 Additional comments (3)
prediction_market_agent/agents/specialized_agent/deploy.py (2)
31-31: LGTM! Good refactoring of the market fetch limit.
Moving the limit to a class attribute improves configurability and makes the market fetching behavior more consistent across agent instances.
45-45: Verify impact on derived classes.
The change from parameter to class attribute looks good. However, let's verify that no derived classes are overriding get_markets() with the old signature.
✅ Verification successful
No impact on derived classes found
The verification shows that there are no derived classes overriding get_markets() with the old signature containing the limit parameter. The only implementations found are:
- The base implementation in
prediction_market_agent/agents/specialized_agent/deploy.pywhich is the one being modified - An unrelated implementation in
prediction_market_agent/agents/metaculus_agent/deploy.pywhich uses a different parameter structure
The change from parameter to class attribute is safe to proceed with.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Check for any classes that override get_markets with the old signature
# containing the 'limit' parameter
# Search for class definitions that inherit from GetMarketCreatorsStalkerMarkets
ast-grep --pattern 'class $_ ($$$GetMarketCreatorsStalkerMarkets$$$) {
$$$
get_markets($$$limit$$$) {
$$$
}
$$$
}'
Length of output: 128
Script:
#!/bin/bash
# Let's try a different approach to find derived classes and their get_markets implementations
# First, find all Python files that might contain derived classes
rg -l "GetMarketCreatorsStalkerMarkets" --type py
# Then look for any get_markets method implementations with context
rg "def get_markets" -A 5 --type py
Length of output: 1493
prediction_market_agent/agents/invalid_agent/deploy.py (1)
25-27: Verify dynamic trade interval behavior.
The change to use a dynamic trade interval per market (same_market_trade_interval.get(market=market)) provides better flexibility, but we should ensure it works correctly with all market types.
Let's verify the implementation of the trade interval getter:
Consider adding a docstring or comment explaining how the trade interval is calculated for different market types, as this knowledge would be valuable for maintainers working with both invalid and arbitrage agents.
-> Added Arbitrage agent for placing YES/NO or NO/YES trades between correlated markets
-> Added test
Some examples of correlated markets + expected profit
Edit - now that correlation < 0 (
near_perfect_correlation = False) can also be handled, found some inversely correlated pairs