Added dpkg configuration files #537

joachimmetz · 2020-06-02T16:36:59Z

@aarontp I think this is ready now, PTAL

joachimmetz · 2020-06-03T07:41:56Z

Package now installs but getting error "Unknown database: user"

Troubleshooting with turbiniactl -S server

Missing python3-docker dependency Make docker requirement optional #542
python3-openssl and python3-ipaddress dependency for Ubuntu provided python3-urllib3, using GIFT PPA provided one instead
protobuf unexpected keyword argument 'serialized_options'

need to regenerate the protoc generated python files when packaging?

Traceback (most recent call last):
  File "/usr/bin/turbiniactl", line 11, in <module>
    load_entry_point('turbinia==20190819', 'console_scripts', 'turbiniactl')()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 480, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2693, in load_entry_point
    return ep.load()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2324, in load
    return self.resolve()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2330, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python3.6/dist-packages/turbinia/turbiniactl.py", line 32, in <module>
    from turbinia.lib import google_cloud
  File "/usr/lib/python3.6/dist-packages/turbinia/lib/google_cloud.py", line 25, in <module>
    from google.cloud import logging as cloud_logging
  File "/usr/lib/python3/dist-packages/google/cloud/logging/__init__.py", line 22, in <module>
    from google.cloud.logging.client import Client
  File "/usr/lib/python3/dist-packages/google/cloud/logging/client.py", line 21, in <module>
    from google.cloud.logging import _gapic
  File "/usr/lib/python3/dist-packages/google/cloud/logging/_gapic.py", line 20, in <module>
    from google.cloud.logging_v2.gapic.config_service_v2_client import ConfigServiceV2Client
  File "/usr/lib/python3/dist-packages/google/cloud/logging_v2/__init__.py", line 17, in <module>
    from google.cloud.logging_v2 import types
  File "/usr/lib/python3/dist-packages/google/cloud/logging_v2/types.py", line 18, in <module>
    from google.api import distribution_pb2
  File "/usr/lib/python3/dist-packages/google/api/distribution_pb2.py", line 26, in <module>
    dependencies=[google_dot_protobuf_dot_any__pb2.DESCRIPTOR,google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,])
TypeError: __new__() got an unexpected keyword argument 'serialized_options'

Looks like libprotobuf is still the old version

joachimmetz · 2020-06-15T20:50:40Z

after manually updating ubutnu 18.04 to libprotobuf17 and corresponding python3-protobuf

turbiniactl -S server
[WARNING] Using fallback source config. Copy turbinia/config/turbinia_config_tmpl.py to ~/.turbiniarc or /etc/turbinia/turbinia.conf, edit, and re-run.
[INFO] Turbinia version: 20200615
Traceback (most recent call last):
  File "/usr/bin/turbiniactl", line 11, in <module>
    load_entry_point('turbinia==20200615', 'console_scripts', 'turbiniactl')()
  File "/usr/lib/python3.6/dist-packages/turbinia/turbiniactl.py", line 414, in main
    from turbinia import client as TurbiniaClientProvider
  File "/usr/lib/python3.6/dist-packages/turbinia/client.py", line 36, in <module>
    from turbinia import task_manager
  File "/usr/lib/python3.6/dist-packages/turbinia/task_manager.py", line 36, in <module>
    from google.cloud import pubsub
  File "/usr/lib/python3/dist-packages/google/cloud/pubsub.py", line 20, in <module>
    from google.cloud.pubsub_v1 import PublisherClient
  File "/usr/lib/python3/dist-packages/google/cloud/pubsub_v1/__init__.py", line 17, in <module>
    from google.cloud.pubsub_v1 import types
  File "/usr/lib/python3/dist-packages/google/cloud/pubsub_v1/types.py", line 32, in <module>
    from google.cloud.pubsub_v1.proto import pubsub_pb2
  File "/usr/lib/python3/dist-packages/google/cloud/pubsub_v1/proto/pubsub_pb2.py", line 30, in <module>
    create_key=_descriptor._internal_create_key,
AttributeError: module 'google.protobuf.descriptor' has no attribute '_internal_create_key'

joachimmetz · 2020-06-16T04:13:00Z

To generate new "/usr/lib/python3/dist-packages/google/cloud/pubsub_v1/proto/pubsub_pb2.py"

git clone https://pypi.org/project/googleapis-common-protos/ the sdist package on pypi does not include proto files
install protobuf-compiler
protoc --proto_path=<location of googleapis-common-protos> --proto_path=/usr/lib/python3/dist-packages/google/cloud/pubsub_v1/proto/ --python_out=/usr/lib/python3/dist-packages/google/cloud/pubsub_v1/proto/ pubsub.proto

Additional packaging changes in log2timeline/l2tdevtools#876

joachimmetz · 2020-06-16T04:15:16Z

Next warning likely due to incomplete configuration

turbiniactl -S server
[WARNING] Using fallback source config. Copy turbinia/config/turbinia_config_tmpl.py to ~/.turbiniarc or /etc/turbinia/turbinia.conf, edit, and re-run.
[INFO] Turbinia version: 20200615
Traceback (most recent call last):
  File "/usr/bin/turbiniactl", line 11, in <module>
    load_entry_point('turbinia==20200615', 'console_scripts', 'turbiniactl')()
  File "/usr/lib/python3.6/dist-packages/turbinia/turbiniactl.py", line 603, in main
    jobs_blacklist=args.jobs_blacklist, jobs_whitelist=args.jobs_whitelist)
  File "/usr/lib/python3.6/dist-packages/turbinia/client.py", line 876, in __init__
    self.task_manager = task_manager.get_task_manager()
  File "/usr/lib/python3.6/dist-packages/turbinia/task_manager.py", line 62, in get_task_manager
    return PSQTaskManager()
  File "/usr/lib/python3.6/dist-packages/turbinia/task_manager.py", line 599, in __init__
    super(PSQTaskManager, self).__init__()
  File "/usr/lib/python3.6/dist-packages/turbinia/task_manager.py", line 105, in __init__
    self.state_manager = state_manager.get_state_manager()
  File "/usr/lib/python3.6/dist-packages/turbinia/state_manager.py", line 62, in get_state_manager
    return DatastoreStateManager()
  File "/usr/lib/python3.6/dist-packages/turbinia/state_manager.py", line 172, in __init__
    self.client = datastore.Client(project=config.TURBINIA_PROJECT)
  File "/usr/lib/python3/dist-packages/google/cloud/datastore/client.py", line 242, in __init__
    project=project, credentials=credentials, _http=_http
  File "/usr/lib/python3/dist-packages/google/cloud/client.py", line 226, in __init__
    _ClientProjectMixin.__init__(self, project=project)
  File "/usr/lib/python3/dist-packages/google/cloud/client.py", line 178, in __init__
    project = self._determine_default(project)
  File "/usr/lib/python3/dist-packages/google/cloud/datastore/client.py", line 270, in _determine_default
    return _determine_default_project(project)
  File "/usr/lib/python3/dist-packages/google/cloud/datastore/client.py", line 87, in _determine_default_project
    project = _base_default_project(project=project)
  File "/usr/lib/python3/dist-packages/google/cloud/_helpers.py", line 186, in _determine_default_project
    _, project = google.auth.default()
  File "/usr/lib/python3/dist-packages/google/auth/_default.py", line 321, in default
    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started

created /etc/turbinia
copied /etc/default/turbinia to /etc/turbinia/turbinia.conf
sudo -u turbinia turbiniactl -S server

joachimmetz · 2020-06-16T07:22:06Z

Note to self, fix package to include systemd files

systemd files are now being packaged

jleaniz · 2023-07-21T15:11:57Z

@joachimmetz is this still being worked on? Let me know if we should re-assign this work.

joachimmetz · 2023-07-21T20:43:45Z

let's close it for now, if needed reopen

joachimmetz force-pushed the dpkg branch 9 times, most recently from 63cea26 to 51d91c8 Compare June 3, 2020 07:15

joachimmetz mentioned this pull request Jun 3, 2020

Make docker requirement optional #542

Open

joachimmetz force-pushed the dpkg branch from 51d91c8 to f296ef3 Compare June 3, 2020 07:51

joachimmetz force-pushed the dpkg branch 3 times, most recently from 700608c to 702b4ab Compare June 15, 2020 17:41

Added dpkg configuration files

dacef62

joachimmetz force-pushed the dpkg branch from 702b4ab to dacef62 Compare June 16, 2020 06:09

joachimmetz marked this pull request as ready for review June 16, 2020 16:45

joachimmetz closed this Jul 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added dpkg configuration files #537

Added dpkg configuration files #537

joachimmetz commented Jun 2, 2020 •

edited

Loading

joachimmetz commented Jun 3, 2020 •

edited

Loading

joachimmetz commented Jun 15, 2020

joachimmetz commented Jun 16, 2020 •

edited

Loading

joachimmetz commented Jun 16, 2020 •

edited

Loading

joachimmetz commented Jun 16, 2020 •

edited

Loading

jleaniz commented Jul 21, 2023

joachimmetz commented Jul 21, 2023

Added dpkg configuration files #537

Added dpkg configuration files #537

Conversation

joachimmetz commented Jun 2, 2020 • edited Loading

joachimmetz commented Jun 3, 2020 • edited Loading

joachimmetz commented Jun 15, 2020

joachimmetz commented Jun 16, 2020 • edited Loading

joachimmetz commented Jun 16, 2020 • edited Loading

joachimmetz commented Jun 16, 2020 • edited Loading

jleaniz commented Jul 21, 2023

joachimmetz commented Jul 21, 2023

joachimmetz commented Jun 2, 2020 •

edited

Loading

joachimmetz commented Jun 3, 2020 •

edited

Loading

joachimmetz commented Jun 16, 2020 •

edited

Loading

joachimmetz commented Jun 16, 2020 •

edited

Loading

joachimmetz commented Jun 16, 2020 •

edited

Loading