🐎 New level of python develop. Source code from github
More dynamic in design, more configuration driven
[-;-/=_
`-; \=_ ___ _________ __ __ ____ ____
) ,"-...--./===--__ / ____/ | / / / / / __ \/ __ \
__|/ / ]` / / __/ /| | / / / / / / / / /_/ /
/;--> >...-\ <\_ / /_/ / ___ |/ /___/ /___/ /_/ / ____/
`- <<, 7/-.\, \____/_/ |_/_____/_____/\____/_/
`- /( `-
pip install gallop
from gallop.config import BaseConfig
config = BaseConfig(a=1)
config.a = 2
config.b = 3
config.to_json("some/path.json")
config.to_yaml("some/path.yaml")
You can turn any callable execution in to configuration, eg save the following
func_name: use:logging.warning
args:
- "hello world"
to sometask.yaml
and run gallop sometask
Is the same to run the python script
import logging
logging.warning("hello world")
We can change the value on the run, while pointing to the position in the config, using a chain of keys (keys to dict or list).
eg. to change the key args
, of its 1st element, we can run
gallop sometask --param:args.0 changed_world
checkin: somekey
, save the result to a centralized dictionary with keysomekey
checkout: somekey
, use the result from the centralized dictionary with keysomekey
checkout: env:DATA_HOME
, use the result from the environment variableDATA_HOME
use:some.module
, use or import the modulesome.module
as the callable, egfunc_name: use:os.path.join
, use the functionos.path.join
func_name: use:pandas.DataFrame
, use the classpandas.DataFrame
If you have
transformers
installed, you can run the following example directly to featurize a sentence
pred_task:
- func_name: use:transformers.AutoModel.from_pretrained
args:
- bert-base-uncased
checkin: model
description: |
Load pretrained model
- func_name: use:transformers.AutoTokenizer.from_pretrained
args:
- bert-base-uncased
checkin: tokenizer
description: |
Load pretrained tokenizer
- func_name: tokenizer
args:
- - "Hello, the capital of [MASK] is Paris"
kwargs:
return_tensors: pt
max_length: 128
truncation: True
padding: True
checkin: inputs
- func_name: use:logging.warning
args:
- func_name: model
kwargs:
input_ids:
checkout: inputs.input_ids
attention_mask:
checkout: inputs.attention_mask
checkin: features
This equals to the following python
script
import logging
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("bert-base-uncased")
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
inputs = tokenizer(
"Hello, the capital of [MASK] is Paris",
return_tensors="pt",
max_length=128,
truncation=True,
padding=True,
)
logging.warning(model(
input_ids=inputs.input_ids,
attention_mask=inputs.attention_mask,
))
Save the yaml to run_bert.yaml
and you can use gallop run_bert --output features
to run the task.
Run in commandline with changed value, and printout one of the checkout value
gallop run_bert --loglevel debug --output features
And run it with CHANGED VALUE
gallop run_bert 、
--param:pred_task.2.args.0.0 "The [MASK] house is where the POTUS live and work"
From the same lead author