Skip to content

otdm tools

Jump to bottom
Bartlomiej Ceglik edited this page Mar 15, 2023 · 56 revisions

Legend:

. - Why I need it?

. . - Idea of otdmDriverProto.py

. - Examples of using it

. - CHANGELOG

. - NOTES

. - CHANGELOG

Automatic information from otdm-tools .deb/control

Package:otdm-tools

Version: 0.26.1

Section: otdm

Priority: optional

Architecture: all

Depends: jq, curl, python (>=3.0) | python3

#Recommends: jq, python (>=3.0) | python3

Recommends: python3-pip, python3-apt | python-apt

#Pre-Depends: otdm-installer (>=0.1) | otdm-installer-dummy (>=0.1)

Maintainer: Bartlomiej Ceglik yoyoek@wp.pl

Tag: oiyshTerminal, otdm

Homepage: https://github.com/yOyOeK1/oiyshTerminal

Description: It's a set of tool to help with the work

of downloding http jsons parsing extracting importing

exporting injecting datasource dashboards items to

grafana. Can do adding and removing operations from

files in Node-red in flows.

Dependencies:

jq curl python (>=3.0) python3 #Pre-Depends: otdm-installer (>=0.1) otdm-installer-dummy (>=0.1)

otdm-tools

With grate power comse grate responsibility. It's a power user tool.

Why I need it?

It's for me all the time arround the similar subject. Reading files, getting data from http. O this data is json, that data is in mysql I need a driver to talk to mysql then make query then I will have a data. New project similar story.

otdm-tools is a My attempt to eliminate a idea of precise work type. My understenging is. You wast to make "work / task". You know what is a medium providing / holding data.

It's becoming your universal worker, qualiffy for the exact work you have.

knowing what to do with:

  • (-ddpkg) dpkg

    • "*" otdm package list

    • "app" get package details

    • install package

    • uninstall package

  • (-dfs) file system

    • read file / directory

    • write to file

    • make directory

  • (-osType) recognize os

    • termux 117

    • ubuntu 22.04 and .

  • Grafana:

    • (-dganById) annotations

    • (-dgdbByUid) dashboards by uid

    • datasources by uid

  • Node-red:

    • (-dnrfByUid) flow(s)

  • Internal cliper

  • (-webCmdSubProcess) Shell command wraper to mqtt / webSocket

    • with live stdio and stdout to mqtt :)

  • Makes backups of (-mkbp):

    • ds|gdsuid|gdsid|gdsname|gdhs|gdhuid|gdhid|gdhname|*

  • Deb files:

    • pre deploy actions

    • build .deb

    • deploy .deb to repository

    • update repository

Home directory: /data/data/com.termux/files/

Yes. It's there to have any chance for me to have it working on android :). If you use otdm-tool on different platform then termux it's Cool !

I'm using it on ubuntu and it's fine. Main part is in python3 so it's makin not so big differenc wher it's running. But it expect from you to have: python3, pip, jq

Config file: /data/data/com.termux/files/home/.otdm/config.json

Some info for drivers and what to use as a defoults. If you use otdm-tools by installing otdm-installer-dummy This is a place where you need to set up ip, ports and other setting to mache your system. Setting for Node-red, grafana, mysql. If you don't have some of those services on your system ma by you have access to it on different machine? Set it up in this file.

Idea of otdmDriverProto.py

It's a definition of a universal driver. Can have:

  • name - keyWord

  • help

  • GET - reading task

  • POST - writing task

  • DELETE - delete task

By using this solution we can ignore what kind of a work we want to do. And a specially how? It's delegated to otdm-tools :)

By expanding otdmDriverProto.py you can have workers knowing what to do with:

  • dpkg

  • file system

  • Grafana:

    • annotations check family of otdm-grafana-an-

    • dashboards by uid check family of otdm-grafana-db-

    • datasources by uid check family of otdm-grafana-ds-

  • Node-red:

    • flow(s) check family of otdm-nrf-

  • Web (.js) <=> mqtt <=> shell (sh) check family of otdm-p-

Examples of using it:

  • getting info from dpkg about all otdm compatible .deb's
otdmTools.py -ddpkg '*' -oFile '/tmp/debs_status.json'

It will ask dpkg for all .deb's in repository / current package data base maching otdm. Result go to oFile path.

  • add note to internal cliper .otdm/cliper.json
otdmTools.py -addNote "This is a example note 1."
  • it will try to help.
otdmTools.py -h
  • make backup of all
otdmTools.py -mkbp "*"
  • wrap bash command to mqtt or webSocket
otdmTools.py \
 -webCmdSubProcess "[/usr/bin/pkexec,--disable-internal-agent,whoami]" \
 -pH "pH93419_" \
 -oFile "--"

It will start system authentication proces then run "whoami". Will start mqtt topic:

  • subP/pH93419_/status - will public starting or done

  • subP/pH93419_/line - will give line by line from command

  • subP/pH93419_/in - it's a stdin of command so if you have prompt or something you can interact with app.

In action: https://www.youtube.com/watch?v=7Kc2dpNmxh4

CHANGELOG

  • migration in ddpkg from dpkg-query to apt-cache

NOTES

  • prompting system for $ auth /usr/bin/pkexec --disable-internal-agent /bin/echo 'foo'
  • list all files in .deb dpkg -c otdm-tools*

otdm-tools-packitso

Set of helpers based on idea of otdm-tools proto driver. Building sets of interchangable works. So works how is don is not important. This is delegate to otdm-tools proto driver. Driver know how to make a X work for us.

-packitso [action] - (?|lsWork|yes)

-work [work] - (nrf|fs|yss-site|.debBranch) work type (selection of tools is delegated )

-forceHost [ip] - if needed for data sourcing different then your otdm-tool config file .otdm/config.json

-ident [uid] - identification of work peas to source Node-red - uid of node flow fs - file / directory path yss-site - path to site directory (need to be external?) .debBranch - name of debian package (need to be packitso?)

-oFile [pathToDir] - path to directory to store results of operation.

Example

  • Getting workers list... store result in /tmp/workers.json

$ otdmTools.py -packitso lsWork -oFile '/tmp/workers.json'

It will give you json of in system drivers proto in array. Entrence dict with :name: and :keyWord of driver.

in my case is my selection

{ "name": "ddpkg", "keyWord": "ddpkg" }, {

  • Getting all flows from node-red store result in /tmp/currentFlowsAtDell.json with additional forceHost option.

$ otdmTools.py -forceHost 192.168.43.220 -packitso ls -work -ident '*' -oFile '/tmp/currentFlowsAtDell.json'

It will give you json of all flows in json as you get it from :1880/flows GET only tabs dictionaries.

  • Getting one flow from Node-red store it in /tmp/flow_d43.json

$ ./otdmTools.py -packitso ls -work 3 -ident 'd42e4dfde145ec63' -oFile '/tmp/flow_d43.json'

Json is a api result of Node-red HTTP API as work in keyWork from lsWork in this case is

  • Getting datasource from grafana ...

$ otdmTools.py -forceHost 192.168.43.1 -packitso ls -work 0 -ident 'p8Fm6u87k' -oFile '--'

Directory where backups will appear...

CHANGELOG

dpkg (0.26.1) unstable; urgency=low

[ B.Ceglik ]

  • protoDriver get for listing isPackitso[bool] iKey, iUid to to auto matching

-- B.Ceglik <yoyoek@wp.pl 23-02-21

dpkg (0.25.9) unstable; urgency=low

[ B.Ceglik ]

  • making in more quiet ... driver proto POST
  • fix driver proto -iFile -iStr
  • added otdmSh.sh for posible gate to make pack it so more smooth

-- B.Ceglik <yoyoek@wp.pl 23-02-21

dpkg (0.24.01) unstable; urgency=low

[ B.Ceglik ]

  • now it's checking if .bashrc have path updated
  • otdmTools.py -dnrfByUid [16charUid] -act DELETE - will delete flow
  • tdmTools.py -osType "?" -oFile /tmp/os.t

-- B.Ceglik <yoyoek@wp.pl 02-02-23

dpkg (0.23.01) unstable; urgency=low

[ B.Ceglik ]

  • ./otdmTools.py -webCmdSubProcess "[mplayer,--help]" -pH "pH93419_" -oFile "/tmp/otdmq_1674581985945_0.24472030095379482.json" it's not helthy :P
  • sub process with autentication on system site. $ otdmTools.py
    -webCmdSubProcess "[/usr/bin/pkexec,--disable-internal-agent,whoami]"
    -pH "pH93419_"
    -oFile "--" [ sudo,--stdin,-E,-k,whoami]
  • Bumping version.
  • ddpkg driver proto to talk with dpkg-query ma by do some more thinks but for now it's a bridge for otdm-yss-otdm. So otdm installed yss and in it you have otdm so it will be a part of otdm-yss-* family. So web site app will have knolage about appps from repository :)

-- B.Ceglik <yoyoek@wp.pl 23-01-22

dpkg (0.20) unstable; urgency=low

[ B.Ceglik ]

  • Bumping version. Having some problems winh repo updater now it will be fixt!
  • Making first full auto deploy of path working: [instance of grafana datasource by name is taken] --> [packt] --> [installd]

-- B.Ceglik <yoyoek@wp.pl 22-12-28

dpkg (0.16) unstable; urgency=low

[ B.Ceglik ]

  • otdmSh.sh for some cinda library for installation and uninstall process. for now: exitIf 1 2 3 Exit with message "2" if 1 is != "0" exit "3" as exit 3
  • Unify argument testing mechanisms and error raporting.

-- B.Ceglik <yoyoek@wp.pl 22-12-28

dpkg (0.15 unstable; urgency=low

[ B.Ceglik ] $ otdmTools.py -extract gdsInjRes -if /tmp/injRes.json Will now extract name,id,uid,orgId and put it in files as suffix using name of the value. So file /tmp/injRes.json will generate /tmp/injRes.json_uid with uid from input json. It's like this for rest of variables.

  • printNote() now unify all notes :) $ otdmTools.py -lastNote 2 displays two last notes :)
  • clipIt system implemented. Plan is to have a way to pass / find / store notes / store objects. Even if instance finished. It is readable by next one. More about it. Use case: -addNote "Add a important note to cliper" or: in code clipIt( obj, note )
  • past 1K lines marker in otdmTools.py :)
  • yes / no dylema. Overrite or not overrite. I added a dialog for now. It's in otdmTools.py so it's for automation not for someone sitting in front of running script. I will see how it will work.
  • $ otdmTools.py -tasks ./DEBIAN/otdmTasker.json - now you can set a tasker file for tools to do. It's reading it from file and executing one by one. In same instance so you have option to get some values what normally is difficult.
  • $ otdmMake.sh [DIR] - is a script to start building process of deb file. Dir is a working debian sourc director. It is making pre run if any ./DEBIAN/otdm
  • changing place of python dependencis to be able to install on wanilla dpkg instance. Ok it's settle down on. It will shout on partial dpkg instance. It means than if you have a dpkg not really use in your system there is no way to go around postist execution problem. otdm-tools needs pip that comes with python. So option on system widhout dpkg system. Installing dpkg, then lot of --force-depends
  • U can now do $ otdmTools.py -mkbp "flows" Makes backup of all flows and put it with date to .otdm/Backups $ otdmTools.py -mkbp "gds" or $ otdmTools.py -forceHost "192.168.43.1" -mkbp "gds" Makes backup of all datasources in grafana puts it to .otdm/Backups if -forceHost added files gets prefix from host name.

-- B.Ceglik <yoyoek@wp.pl 22-12-26

[ B.Ceglik ]

  • added dependencies to start installation tool chain if needed. it needs otdm-installer or otdm-installer-dummy

-- B.Ceglik <yoyoek@wp.pl 22-12-26

dpkg (0.11)

[ B.Ceglik ]

  • -testAll 1 - new arg set. It's going by grafana and node-red doing test if function are ok.
  • new release to see if I can deploy node-red flow from $ apt install otdm-nrf-testmqttmysql starting from unknown status of system deb based. To get to point where I will see in front of me browser with that flow.
  • preinst now go checking of pip and try to install deps if needed. checking jq if there is no checking if there is a pkg, then use it to install jq.

-- B.Ceglik <yoyoek@wp.pl 22-12-26

dpkg (0.10) unstable; urgency=low

[ B.Ceglik ]

  • It's a first release with python wersion of tools. Move from otdmTools to otdm-tools.

-- B.Ceglik <yoyoek@wp.pl 22-12-25

dpkg (0.1) unstable; urgency=low

[ B.Ceglik ]

  • Start of a otdm idea.....

-- B.Ceglik <yoyoek@wp.pl 22-12-25

otdm family:

index cmqtt2mysql db-init db-init-dummy grafana-btzoom grafana-ds-MySql installer-dummy intranetrepo-debianish intranetrepo-termux mqtt-installer mqtt-installer-dummy mrepo termux-wake-lock node-red-installer node-red-installer-dummy nrf-yss nrf-ot-test nrf-audio-onhost nrf-hhbell-binary yss yss-project-manager yss-packitso yss-shareitso yss-calibration yss-autopilot-v3-svg yss-remote-camera yss-basic-sail yss-iloo-nav yss-multisvg yss-music-sync yss-iloo-nav-3d yss-3d-compass1 yss-test-functions yss-test-three-js yss-test-maedv yss-test-mdynoform yss-imagehistorye yss-test-functionse yss-otdme otdmp-nrf-http-api-plc tools

Clone this wiki locally