Server refactor

[jpodivin]

General code refactoring in anticipation of deployment as a server. The server option is only intended for experimental use, and will not be documented at this time.

Among other changes:

• prompt has been simplified
• CLI args now have proper help
• new optional dependencies for server were specified
• log retrieval function now has type annotations
• constants, extractors and utilities were moved into separate submodules

[jkonecny12]

Last commit has a typo impor.

[jkonecny12]

Looks good to me. Just a few nitpicks and questions.

[jkonecny12]

It's also taking local paths now. I think it should be mentioned in help.

[jkonecny12]

I wonder what local means here.

[jpodivin]

in the file system.

[jkonecny12]

I think it needs more explanation. Does it mean path to local model or name of the model in cache directory? If the second how it should be specified.

Could you please improve the help a bit?

[jpodivin]

It's path, not a name, the help says that. At no point is any cache mentioned. User shouldn't know about it and he shouldn't even know about it. It's an implementation detail.

[TomasTomecek]

requests.exceptions.ReadTimeout: HTTPConnectionPool(host='localhost', port=8080): Read timed out. (read timeout=200)

I can see the timeout is configurable but we'll need to increase the default to at least 10 minutes; it's really frustrating to wait just to get timeouted

[TomasTomecek]

I finally have a successful run \o/
after this is merged, I'll provide docs for readme how to run the server :)

{'content': "[1]:\nThe output of the task shows that the SRPM file was downloaded and extracted successfully.\n\n[2]:\nThe output of the SRPMResults tool confirms the package name, version, and architecture.\n\n[3]:The build log indicates that the build process started and the initial checks were successful.\n\n[4]:\nThe build log shows the first failure during the installation of dependencies. This failure is due to the missing 'librocm-hip-dev' package.\n\n[5]:\nThe second failure occurs during the RPM build process. The error message indicates that the 'gcc-11' compiler is not installed.\n\n[6]:\nThe error message also suggests that the 'gcc-rocm' compiler should be used instead. The 'gcc-rocm' compiler is a part of the 'rocm-devel' package, which is not installed as well.\n\n[7]:\nThe 'rocm-devel' package is not included in the list of dependencies, which wasgenerated from the SRPM file.\n\n[8]:\nThe 'rocm-dev' package, which is required for the 'librocm-hip-dev' package, is not installed either. This package is not included in the list of dependencies as well.\n\nExplanation:\n\nThe issue arises from the missing dependencies in the SRPM file, specifically 'rocm-devel' and 'rocm-dev'. The absence of these packages leads to the installation failures of 'librocm-hip-dev' and 'gcc-rocm'.\n\nThe root cause of this issue lies in the misconfiguration or incorrect generation of the SRPM file. The missing dependencies are likely to be overlooked during the SRPM creation process.\n\nTo resolve the issue, the dependencies should be added to the SRPM file, and the task should be re-run.\n\nTo prevent similar issues from occurring in the future, make sure to double-check the SRPM file for missing dependencies before distributing it. Also, consider installing the required packages during the task setup process, if they are not typically included in the list of dependencies.",
 'id_slot': 0,
 'stop': True,
 'model': '/home/tt/.cache/logdetective/mistral-7b-instruct-v0.2.Q4_K_S.gguf',
 'tokens_predicted': 467,
 'tokens_evaluated': 270,
 'generation_settings': {'n_ctx': 512,
  'n_predict': -1,
  'model': '/home/tt/.cache/logdetective/mistral-7b-instruct-v0.2.Q4_K_S.gguf',
  'seed': 4294967295,
  'temperature': 0.800000011920929,
  'dynatemp_range': 0.0,
  'dynatemp_exponent': 1.0,
  'top_k': 40,
  'top_p': 0.949999988079071,
  'min_p': 0.05000000074505806,
  'tfs_z': 1.0,
  'typical_p': 1.0,
  'repeat_last_n': 64,
  'repeat_penalty': 1.0,
  'presence_penalty': 0.0,
  'frequency_penalty': 0.0,
  'penalty_prompt_tokens': [],
  'use_penalty_prompt_tokens': False,
  'mirostat': 0,
  'mirostat_tau': 5.0,
  'mirostat_eta': 0.10000000149011612,
  'penalize_nl': False,
  'stop': [],
  'n_keep': 0,
  'n_discard': 0,
  'ignore_eos': False,
  'stream': False,
  'logit_bias': [],
  'n_probs': 0,
  'min_keep': 0,
  'grammar': '',
  'samplers': ['top_k',
   'tfs_z',
   'typical_p',
   'top_p',
   'min_p',
   'temperature']},
 'prompt': '\nGiven following log snippets, and nothing else, explain what failure, if any, occured during build of this package.\n\nLogging PID: 6270\n\nTask:\n\n{\'appstream\': False,\n \'background\': False,\n \'build_id\': 7495493,\n \'chroot\': None,\n \'package_name\': None,\n \'project_dirname\': \'review-2279158-python-triton-rocm\',\n \'project_name\': \'review-2279158-python-triton-rocm\',\n \'project_owner\': \'ttomecek\',\n \'repos\': [],\n \'sandbox\': \'ttomecek/review-2279158-python-triton-rocm--ttomecek\',\n \'source_json\': {\'pkg\': \'python-triton-rocm-2.3.0-1.fc41.src.rpm\',\n                 \'tmp\': \'tmpueeppiy3\',\n                 \'url\': \'https://copr.fedorainfracloud.org/tmp/tmpueeppiy3/python-triton-rocm-2.3.0-1.fc41.src.rpm\'},\n \'source_type\': 2,\n \'submitter\': \'ttomecek\',\n \'task_id\': \'7495493\'}\n\n\nOutput: [\'python-triton-rocm-2.3.0-1.fc41.src.rpm\']\n\nRunning SRPMResults tool\n\nPackage info: {\n    "name": "python-triton-rocm",\n    "epoch": null,\n    "version": "2.3.0",\n    "release": "1",\n    "exclusivearch": [\n        "x86_64"\n    ],\n    "excludearch": []\n\n}\n\nSRPMResults finished\n\n\n\nAnalysis of the failure must be in a format of [X] : [Y], where [X] is a log snippet, and [Y] is the explanation.\n\nFinally, drawing on information from all snippets, provide complete explanation of the issue.\n\nAnalysis:\n\n',
 'truncated': True,
 'stopped_eos': True,
 'stopped_word': False,
 'stopped_limit': False,
 'stopping_word': '',
 'tokens_cached': 481,
 'timings': {'prompt_n': 270,
  'prompt_ms': 191864.2,
  'prompt_per_token_ms': 710.6081481481482,
  'prompt_per_second': 1.4072453328969137,
  'predicted_n': 467,
  'predicted_ms': 631375.292,
  'predicted_per_token_ms': 1351.981353319058,
  'predicted_per_second': 0.7396551716819479}}

[TomasTomecek]

LGTM, none of the comments/suggestions are blocking

[TomasTomecek]

I'm not a fan of callable instances of classes. It looks pretty confusing in the code.

[TomasTomecek]

we need to get logging setup to the server as well

[jpodivin]

You mean the silent option? I was thinking about that and decided against it. The major reason for quiet mode is that all the output pollutes the terminal and makes things difficult to read. But if it's going to run as a server, we don't need to worry about that.

Especially if it's going to be containerized, because in that case the only logs stored for the pod will all belong to our service, and will be easy to filter.

[TomasTomecek]

sorry, I commented on the wrong line

my point is that there is no logger handler for logdetective module when running directly from CLI using fastapi so there are no logs printed in the console; I manually added one while playing with this PR so that I could see what's happening

[TomasTomecek]

Thanks for the updates, still LGTM!