Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inconsistent IDs #1

Open
thammegowda opened this issue Nov 22, 2021 · 2 comments
Open

Inconsistent IDs #1

thammegowda opened this issue Nov 22, 2021 · 2 comments

Comments

@thammegowda
Copy link

Hi,

Thanks for your efforts in creating/curating these datasets! These are priceless and greatly advance NLP for Indian languages.

I tried adding them into mtdata thammegowda/mtdata#81
Since the README says your datasets are still growing, I am wondering whats the best long-term strategy is for keeping in sync.

For now, I can grep -i -o 'http[^ ]*zip' README.md, but the immediate concern is about consistency in determining name, version, and languages of datasets from URL.

The way current files are named (which act as ID for corpus) is a bit inconsistent. For example, consider these:

1) https://anuvaad-parallel-corpus.s3-us-west-2.amazonaws.com/oneindia_20210320_en_ml.zip
2) https://anuvaad-parallel-corpus.s3-us-west-2.amazonaws.com/pibarchives_2014_2016_en_ml.zip
3) https://anuvaad-parallel-corpus.s3-us-west-2.amazonaws.com/wikipedia-en-ml-20210201.zip
  • item (1), we can easily split by _ and get (name, version, lang1, lang2), so this is great. we can see oneindia is the name, 20210320 is the version, and en_ml are langs.
  • item (2), seems okay we can call 2014_2016 as version, though it would have been nice to have 2014to2016v1 as version. so splitting by _ would give exactly (name, version, lang1, lang2) as in item 1.
  • item (3) seems abnormal as it doesn't fit (name, version, lang1, lang2). There are more datasets matching item (1) than item (3) pattern, so I am inclined to call this abnormal.

Could you please consider having a consistent format in dataset IDs? It'd greatly help the automated downloaders such as mtdata.
Otherwise, do you really want your users to manually download 196 zip files via browser, and extract and merge them? :)

Thanks.

P.S https://github.com/thammegowda/mtdata#dataset-id
@thammegowda
Copy link
Author

Also, note that there is a bit of inconsistency inside zipping of item (3) as well.

$ unzip oneindia_20210320_en_ml.zip
Archive:  oneindia_20210320_en_ml.zip
   creating: en-ml/
  inflating: en-ml/oneindia_train.ml
  inflating: en-ml/oneindia_train.en
$ unzip pibarchives_2014_2016_en_ml.zip
Archive:  pibarchives_2014_2016_en_ml.zip
  inflating: en-ml/.DS_Store
  inflating: __MACOSX/en-ml/._.DS_Store
  inflating: en-ml/pib_arch_train.en
  inflating: en-ml/pib_arch_train.ml
$ unzip wikipedia-en-ml-20210201.zip
Archive:  wikipedia-en-ml-20210201.zip
  inflating: en-ml/ml.txt
  inflating: en-ml/en.txt

en-ml/wikipedia_train.{en,ml} could have made scripts/automation tools simple to write.

@thammegowda
Copy link
Author

I added these datasets to v0.3.2

pip install -I mtdata==0.3.2

See mappings 
mtdata list | grep Anuvaad  | sed 's;https://anuvaad-parallel-corpus.s3-us-west-2.amazonaws.com;<BASE>;' 
2021-12-05 13:35:24 __init__.get_instance:48 INFO:: Loading index from cache /Users/tg/.mtdata/mtdata.index.0.3.2.pkl
2021-12-05 13:35:25 main.list_data:19 INFO:: Found 182787
Anuvaad-ik_2021-v1-eng-hin      <BASE>/ik-2021-v1-en-hi.zip     en-hi/*.en,en-hi/*.hi
Anuvaad-internal_judicial_2021-v1-eng-hin       <BASE>/internal-judicial-2021-v1-en-hi.zip      en-hi/*.en,en-hi/*.hi
Anuvaad-legal_terms_2021-v1-eng-hin     <BASE>/legal-terms-2021-v1-en-hi.zip    en-hi/*.en,en-hi/*.hi
Anuvaad-nouns_dict_2021-v1-eng-hin      <BASE>/nouns-dict-2021-v1-en-hi.zip     en-hi/*.en,en-hi/*.hi
Anuvaad-pib_2017-2020-eng-hin   <BASE>/pib_2017_2020_en_hi.zip  en-hi/*.en,en-hi/*.hi
Anuvaad-pibarchives_2009-2016-eng-hin   <BASE>/pibarchives_2009_2016_en_hi.zip  en-hi/*.en,en-hi/*.hi
Anuvaad-newsonair_2019-2020-eng-hin     <BASE>/newsonair_2019_2020_en_hi.zip    en-hi/*.en,en-hi/*.hi
Anuvaad-wikipedia-20210201-eng-hin      <BASE>/wikipedia-en-hi-20210201.zip     en-hi/en.txt,en-hi/hi.txt
Anuvaad-drivespark-20210303-eng-hin     <BASE>/drivespark-en-hi-20210303.zip    en-hi/*.en,en-hi/*.hi
Anuvaad-dd_national-20210320-eng-hin    <BASE>/dd-national-en-hi-20210320.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-dd_sports-20210320-eng-hin      <BASE>/dd-sports-en-hi-20210320.zip     en-hi/*.en,en-hi/*.hi
Anuvaad-nativeplanet-20210315-eng-hin   <BASE>/nativeplanet-en-hi-20210315.zip  en-hi/*.en,en-hi/*.hi
Anuvaad-catchnews-20210320-eng-hin      <BASE>/catchnews-en-hi-20210320.zip     en-hi/*.en,en-hi/*.hi
Anuvaad-dwnews_2008-2020-eng-hin        <BASE>/dwnews_2008_2020_en_hi.zip       en-hi/*.en,en-hi/*.hi
Anuvaad-oneindia-20210320-eng-hin       <BASE>/oneindia_20210320_en_hi.zip      en-hi/*.en,en-hi/*.hi
Anuvaad-mk-20210320-eng-hin     <BASE>/mk_20210320_en_hi.zip    en-hi/*.en,en-hi/*.hi
Anuvaad-goodreturns-20210320-eng-hin    <BASE>/goodreturns_20210320_en_hi.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-ie_sports-20210320-eng-hin      <BASE>/ie_sports_20210320_en_hi.zip     en-hi/*.en,en-hi/*.hi
Anuvaad-ie_tech-20210320-eng-hin        <BASE>/ie_tech_20210320_en_hi.zip       en-hi/*.en,en-hi/*.hi
Anuvaad-ie_news-20210320-eng-hin        <BASE>/ie_news_20210320_en_hi.zip       en-hi/*.en,en-hi/*.hi
Anuvaad-ie_lifestyle-20210320-eng-hin   <BASE>/ie_lifestyle_20210320_en_hi.zip  en-hi/*.en,en-hi/*.hi
Anuvaad-ie_general-20210320-eng-hin     <BASE>/ie_general_20210320_en_hi.zip    en-hi/*.en,en-hi/*.hi
Anuvaad-ie_entertainment-20210320-eng-hin       <BASE>/ie_entertainment_20210320_en_hi.zip      en-hi/*.en,en-hi/*.hi
Anuvaad-ie_education-20210320-eng-hin   <BASE>/ie_education_20210320_en_hi.zip  en-hi/*.en,en-hi/*.hi
Anuvaad-ie_business-20210320-eng-hin    <BASE>/ie_business_20210320_en_hi.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-toi-20210320-eng-hin    <BASE>/toi_20210320_en_hi.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-fin_express-20210320-eng-hin    <BASE>/fin_express_20210320_en_hi.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-thewire-20210320-eng-hin        <BASE>/thewire_20210320_en_hi.zip       en-hi/*.en,en-hi/*.hi
Anuvaad-tribune-20210320-eng-hin        <BASE>/tribune_20210320_en_hi.zip       en-hi/*.en,en-hi/*.hi
Anuvaad-zeebiz-20210320-eng-hin <BASE>/zeebiz_20210320_en_hi.zip        en-hi/*.en,en-hi/*.hi
Anuvaad-pa_govt-20210320-eng-hin        <BASE>/pa_govt_20210320_en_hi.zip       en-hi/*.en,en-hi/*.hi
Anuvaad-betterindia-20210320-eng-hin    <BASE>/betterindia_20210320_en_hi.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-jagran_news-20210320-eng-hin    <BASE>/jagran-news_20210320_en_hi.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-jagran_tech-20210320-eng-hin    <BASE>/jagran-tech_20210320_en_hi.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-jagran_education-20210320-eng-hin       <BASE>/jagran-education_20210320_en_hi.zip      en-hi/*.en,en-hi/*.hi
Anuvaad-jagran_entertainment-20210320-eng-hin   <BASE>/jagran-entertainment_20210320_en_hi.zip  en-hi/*.en,en-hi/*.hi
Anuvaad-jagran_business-20210320-eng-hin        <BASE>/jagran-business_20210320_en_hi.zip       en-hi/*.en,en-hi/*.hi
Anuvaad-jagran_sports-20210320-eng-hin  <BASE>/jagran-sports_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi
Anuvaad-jagran_lifestyle-20210320-eng-hin       <BASE>/jagran-lifestyle_20210320_en_hi.zip      en-hi/*.en,en-hi/*.hi
Anuvaad-asianetnews-20210320-eng-hin    <BASE>/asianetnews_20210320_en_hi.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-business_standard-20210320-eng-hin      <BASE>/business_standard_20210320_en_hi.zip     en-hi/*.en,en-hi/*.hi
Anuvaad-pranabmukherjee-20210320-eng-hin        <BASE>/pranabmukherjee_20210320_en_hi.zip       en-hi/*.en,en-hi/*.hi
Anuvaad-lokmat_entertainment-20210501-eng-hin   <BASE>/lokmat-entertainment_20210501_en_hi.zip  en-hi/*.en,en-hi/*.hi
Anuvaad-lokmat_news-20210501-eng-hin    <BASE>/lokmat-news_20210501_en_hi.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-lokmat_lifestyle-20210501-eng-hin       <BASE>/lokmat-lifestyle_20210501_en_hi.zip      en-hi/*.en,en-hi/*.hi
Anuvaad-lokmat_sports-20210501-eng-hin  <BASE>/lokmat-sports_20210501_en_hi.zip en-hi/*.en,en-hi/*.hi
Anuvaad-lokmat_tech-20210501-eng-hin    <BASE>/lokmat-tech_20210501_en_hi.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-lokmat_financial-20210501-eng-hin       <BASE>/lokmat-financial_20210501_en_hi.zip      en-hi/*.en,en-hi/*.hi
Anuvaad-lokmat_healthcare-20210501-eng-hin      <BASE>/lokmat-healthcare_20210501_en_hi.zip     en-hi/*.en,en-hi/*.hi
Anuvaad-zee-30042021-eng-hin    <BASE>/zee_30042021_en_hi.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-ik_2021-v1-eng-ben      <BASE>/ik-2021-v1-en-bn.zip     en-bn/*.en,en-bn/*.bn
Anuvaad-internal_judicial_2021-v1-eng-ben       <BASE>/internal-judicial-2021-v1-en-bn.zip      en-bn/*.en,en-bn/*.bn
Anuvaad-pib_2017-2020-eng-ben   <BASE>/pib_2017_2020_en_bn.zip  en-bn/*.en,en-bn/*.bn
Anuvaad-pibarchives_2015-2016-eng-ben   <BASE>/pibarchives_2015_2016_en_bn.zip  en-bn/*.en,en-bn/*.bn
Anuvaad-prothomalo_2014-2020-eng-ben    <BASE>/prothomalo_2014_2020_en_bn.zip   en-bn/*.en,en-bn/*.bn
Anuvaad-dwnews_2004-2020-eng-ben        <BASE>/dwnews_2004_2020_en_bn.zip       en-bn/*.en,en-bn/*.bn
Anuvaad-oneindia-20210320-eng-ben       <BASE>/oneindia_20210320_en_bn.zip      en-bn/*.en,en-bn/*.bn
Anuvaad-mk-20210320-eng-ben     <BASE>/mk_20210320_en_bn.zip    en-bn/*.en,en-bn/*.bn
Anuvaad-ie_sports-20210320-eng-ben      <BASE>/ie_sports_20210320_en_bn.zip     en-bn/*.en,en-bn/*.bn
Anuvaad-ie_tech-20210320-eng-ben        <BASE>/ie_tech_20210320_en_bn.zip       en-bn/*.en,en-bn/*.bn
Anuvaad-ie_news-20210320-eng-ben        <BASE>/ie_news_20210320_en_bn.zip       en-bn/*.en,en-bn/*.bn
Anuvaad-ie_general-20210320-eng-ben     <BASE>/ie_general_20210320_en_bn.zip    en-bn/*.en,en-bn/*.bn
Anuvaad-ie_entertainment-20210320-eng-ben       <BASE>/ie_entertainment_20210320_en_bn.zip      en-bn/*.en,en-bn/*.bn
Anuvaad-ie_education-20210320-eng-ben   <BASE>/ie_education_20210320_en_bn.zip  en-bn/*.en,en-bn/*.bn
Anuvaad-ie_business-20210320-eng-ben    <BASE>/ie_business_20210320_en_bn.zip   en-bn/*.en,en-bn/*.bn
Anuvaad-ie_lifestyle-20210320-eng-ben   <BASE>/ie_lifestyle_20210320_en_bn.zip  en-bn/*.en,en-bn/*.bn
Anuvaad-asianetnews-20210320-eng-ben    <BASE>/asianetnews_20210320_en_bn.zip   en-bn/*.en,en-bn/*.bn
Anuvaad-sentinel-20210320-eng-ben       <BASE>/sentinel_20210320_en_bn.zip      en-bn/*.en,en-bn/*.bn
Anuvaad-toi-20210320-eng-ben    <BASE>/toi_20210320_en_bn.zip   en-bn/*.en,en-bn/*.bn
Anuvaad-zee-30042021-eng-ben    <BASE>/zee_30042021_en_bn.zip   en-bn/*.en,en-bn/*.bn
Anuvaad-anuvaad_general-corpus-eng-ben  <BASE>/anuvaad_general_corpus_en_bn.zip en-bn/*.en,en-bn/*.bn
Anuvaad-ik_2021-v1-eng-tam      <BASE>/ik-2021-v1-en-ta.zip     en-ta/*.en,en-ta/*.ta
Anuvaad-internal_judicial_2021-v1-eng-tam       <BASE>/internal-judicial-2021-v1-en-ta.zip      en-ta/*.en,en-ta/*.ta
Anuvaad-legal_terms_2021-v1-eng-tam     <BASE>/legal-terms-2021-v1-en-ta.zip    en-ta/*.en,en-ta/*.ta
Anuvaad-nouns_dict_2021-v1-eng-tam      <BASE>/nouns-dict-2021-v1-en-ta.zip     en-ta/*.en,en-ta/*.ta
Anuvaad-pib_2017-2020-eng-tam   <BASE>/pib_2017_2020_en_ta.zip  en-ta/*.en,en-ta/*.ta
Anuvaad-pibarchives_2014-2016-eng-tam   <BASE>/pibarchives_2014_2016_en_ta.zip  en-ta/*.en,en-ta/*.ta
Anuvaad-wikipedia-20210201-eng-tam      <BASE>/wikipedia-en-ta-20210201.zip     en-ta/en.txt,en-ta/ta.txt
Anuvaad-nativeplanet-20210315-eng-tam   <BASE>/nativeplanet-en-ta-20210315.zip  en-ta/*.en,en-ta/*.ta
Anuvaad-oneindia-20210320-eng-tam       <BASE>/oneindia_20210320_en_ta.zip      en-ta/*.en,en-ta/*.ta
Anuvaad-mk-20210320-eng-tam     <BASE>/mk_20210320_en_ta.zip    en-ta/*.en,en-ta/*.ta
Anuvaad-drivespark-20210303-eng-tam     <BASE>/drivespark-en-ta-20210303.zip    en-ta/*.en,en-ta/*.ta
Anuvaad-goodreturns-20210320-eng-tam    <BASE>/goodreturns_20210320_en_ta.zip   en-ta/*.en,en-ta/*.ta
Anuvaad-ie_sports-20210320-eng-tam      <BASE>/ie_sports_20210320_en_ta.zip     en-ta/*.en,en-ta/*.ta
Anuvaad-ie_tech-20210320-eng-tam        <BASE>/ie_tech_20210320_en_ta.zip       en-ta/*.en,en-ta/*.ta
Anuvaad-ie_news-20210320-eng-tam        <BASE>/ie_news_20210320_en_ta.zip       en-ta/*.en,en-ta/*.ta
Anuvaad-ie_lifestyle-20210320-eng-tam   <BASE>/ie_lifestyle_20210320_en_ta.zip  en-ta/*.en,en-ta/*.ta
Anuvaad-ie_entertainment-20210320-eng-tam       <BASE>/ie_entertainment_20210320_en_ta.zip      en-ta/*.en,en-ta/*.ta
Anuvaad-ie_education-20210320-eng-tam   <BASE>/ie_education_20210320_en_ta.zip  en-ta/*.en,en-ta/*.ta
Anuvaad-ie_business-20210320-eng-tam    <BASE>/ie_business_20210320_en_ta.zip   en-ta/*.en,en-ta/*.ta
Anuvaad-asianetnews-20210320-eng-tam    <BASE>/asianetnews_20210320_en_ta.zip   en-ta/*.en,en-ta/*.ta
Anuvaad-toi-20210320-eng-tam    <BASE>/toi_20210320_en_ta.zip   en-ta/*.en,en-ta/*.ta
Anuvaad-zee-30042021-eng-tam    <BASE>/zee_30042021_en_ta.zip   en-ta/*.en,en-ta/*.ta
Anuvaad-ik_2021-v1-eng-mal      <BASE>/ik-2021-v1-en-ml.zip     en-ml/*.en,en-ml/*.ml
Anuvaad-internal_judicial_2021-v1-eng-mal       <BASE>/internal-judicial-2021-v1-en-ml.zip      en-ml/*.en,en-ml/*.ml
Anuvaad-legal_terms_2021-v1-eng-mal     <BASE>/legal-terms-2021-v1-en-ml.zip    en-ml/*.en,en-ml/*.ml
Anuvaad-nouns_dict_2021-v1-eng-mal      <BASE>/nouns-dict-2021-v1-en-ml.zip     en-ml/*.en,en-ml/*.ml
Anuvaad-pib_2017-2020-eng-mal   <BASE>/pib_2017_2020_en_ml.zip  en-ml/*.en,en-ml/*.ml
Anuvaad-pibarchives_2014-2016-eng-mal   <BASE>/pibarchives_2014_2016_en_ml.zip  en-ml/*.en,en-ml/*.ml
Anuvaad-wikipedia-20210201-eng-mal      <BASE>/wikipedia-en-ml-20210201.zip     en-ml/en.txt,en-ml/ml.txt
Anuvaad-oneindia-20210320-eng-mal       <BASE>/oneindia_20210320_en_ml.zip      en-ml/*.en,en-ml/*.ml
Anuvaad-mk-20210320-eng-mal     <BASE>/mk_20210320_en_ml.zip    en-ml/*.en,en-ml/*.ml
Anuvaad-drivespark-20210303-eng-mal     <BASE>/drivespark-en-ml-20210303.zip    en-ml/*.en,en-ml/*.ml
Anuvaad-nativeplanet-20210315-eng-mal   <BASE>/nativeplanet-en-ml-20210315.zip  en-ml/*.en,en-ml/*.ml
Anuvaad-goodreturns-20210320-eng-mal    <BASE>/goodreturns_20210320_en_ml.zip   en-ml/*.en,en-ml/*.ml
Anuvaad-ie_sports-20210320-eng-mal      <BASE>/ie_sports_20210320_en_ml.zip     en-ml/*.en,en-ml/*.ml
Anuvaad-ie_tech-20210320-eng-mal        <BASE>/ie_tech_20210320_en_ml.zip       en-ml/*.en,en-ml/*.ml
Anuvaad-ie_news-20210320-eng-mal        <BASE>/ie_news_20210320_en_ml.zip       en-ml/*.en,en-ml/*.ml
Anuvaad-ie_general-20210320-eng-mal     <BASE>/ie_general_20210320_en_ml.zip    en-ml/*.en,en-ml/*.ml
Anuvaad-ie_entertainment-20210320-eng-mal       <BASE>/ie_entertainment_20210320_en_ml.zip      en-ml/*.en,en-ml/*.ml
Anuvaad-ie_business-20210320-eng-mal    <BASE>/ie_business_20210320_en_ml.zip   en-ml/*.en,en-ml/*.ml
Anuvaad-marketfeed-20210320-eng-mal     <BASE>/marketfeed_20210320_en_ml.zip    en-ml/*.en,en-ml/*.ml
Anuvaad-asianetnews-20210320-eng-mal    <BASE>/asianetnews_20210320_en_ml.zip   en-ml/*.en,en-ml/*.ml
Anuvaad-toi-20210320-eng-mal    <BASE>/toi_20210320_en_ml.zip   en-ml/*.en,en-ml/*.ml
Anuvaad-zee-30042021-eng-mal    <BASE>/zee_30042021_en_ml.zip   en-ml/*.en,en-ml/*.ml
Anuvaad-ik_2021-v1-eng-tel      <BASE>/ik-2021-v1-en-te.zip     en-te/*.en,en-te/*.te
Anuvaad-internal_judicial_2021-v1-eng-tel       <BASE>/internal-judicial-2021-v1-en-te.zip      en-te/*.en,en-te/*.te
Anuvaad-legal_terms_2021-v1-eng-tel     <BASE>/legal-terms-2021-v1-en-te.zip    en-te/*.en,en-te/*.te
Anuvaad-nouns_dict_2021-v1-eng-tel      <BASE>/nouns-dict-2021-v1-en-te.zip     en-te/*.en,en-te/*.te
Anuvaad-pib_2017-2020-eng-tel   <BASE>/pib_2017_2020_en_te.zip  en-te/*.en,en-te/*.te
Anuvaad-pibarchives_2015-2016-eng-tel   <BASE>/pibarchives_2015_2016_en_te.zip  en-te/*.en,en-te/*.te
Anuvaad-wikipedia-20210201-eng-tel      <BASE>/wikipedia-en-te-20210201.zip     en-te/en.txt,en-te/te.txt
Anuvaad-oneindia-20210320-eng-tel       <BASE>/oneindia_20210320_en_te.zip      en-te/*.en,en-te/*.te
Anuvaad-mk-20210320-eng-tel     <BASE>/mk_20210320_en_te.zip    en-te/*.en,en-te/*.te
Anuvaad-drivespark-20210303-eng-tel     <BASE>/drivespark-en-te-20210303.zip    en-te/*.en,en-te/*.te
Anuvaad-nativeplanet-20210315-eng-tel   <BASE>/nativeplanet-en-te-20210315.zip  en-te/*.en,en-te/*.te
Anuvaad-goodreturns-20210320-eng-tel    <BASE>/goodreturns_20210320_en_te.zip   en-te/*.en,en-te/*.te
Anuvaad-sakshi-20210320-eng-tel <BASE>/sakshi_20210320_en_te.zip        en-te/*.en,en-te/*.te
Anuvaad-asianetnews-20210320-eng-tel    <BASE>/asianetnews_20210320_en_te.zip   en-te/*.en,en-te/*.te
Anuvaad-toi-20210320-eng-tel    <BASE>/toi_20210320_en_te.zip   en-te/*.en,en-te/*.te
Anuvaad-zee-30042021-eng-tel    <BASE>/zee_30042021_en_te.zip   en-te/*.en,en-te/*.te
Anuvaad-ik_2021-v1-eng-kan      <BASE>/ik-2021-v1-en-kn.zip     en-kn/*.en,en-kn/*.kn
Anuvaad-legal_terms_2021-v1-eng-kan     <BASE>/legal-terms-2021-v1-en-kn.zip    en-kn/*.en,en-kn/*.kn
Anuvaad-nouns_dict_2021-v1-eng-kan      <BASE>/nouns-dict-2021-v1-en-kn.zip     en-kn/*.en,en-kn/*.kn
Anuvaad-pib_2017-2020-eng-kan   <BASE>/pib_2017_2020_en_kn.zip  en-kn/*.en,en-kn/*.kn
Anuvaad-pibarchives_2015-2016-eng-kan   <BASE>/pibarchives_2015_2016_en_kn.zip  en-kn/*.en,en-kn/*.kn
Anuvaad-oneindia-20210320-eng-kan       <BASE>/oneindia_20210320_en_kn.zip      en-kn/*.en,en-kn/*.kn
Anuvaad-mk-20210320-eng-kan     <BASE>/mk_20210320_en_kn.zip    en-kn/*.en,en-kn/*.kn
Anuvaad-drivespark-20210303-eng-kan     <BASE>/drivespark-en-kn-20210303.zip    en-kn/*.en,en-kn/*.kn
Anuvaad-nativeplanet-20210315-eng-kan   <BASE>/nativeplanet-en-kn-20210315.zip  en-kn/*.en,en-kn/*.kn
Anuvaad-goodreturns-20210320-eng-kan    <BASE>/goodreturns_20210320_en_kn.zip   en-kn/*.en,en-kn/*.kn
Anuvaad-asianetnews-20210320-eng-kan    <BASE>/asianetnews_20210320_en_kn.zip   en-kn/*.en,en-kn/*.kn
Anuvaad-toi-20210320-eng-kan    <BASE>/toi_20210320_en_kn.zip   en-kn/*.en,en-kn/*.kn
Anuvaad-zee-30042021-eng-kan    <BASE>/zee_30042021_en_kn.zip   en-kn/*.en,en-kn/*.kn
Anuvaad-ik_2021-v1-eng-mar      <BASE>/ik-2021-v1-en-mr.zip     en-mr/*.en,en-mr/*.mr
Anuvaad-legal_terms_2021-v1-eng-mar     <BASE>/legal-terms-2021-v1-en-mr.zip    en-mr/*.en,en-mr/*.mr
Anuvaad-nouns_dict_2021-v1-eng-mar      <BASE>/nouns-dict-2021-v1-en-mr.zip     en-mr/*.en,en-mr/*.mr
Anuvaad-pib_2017-2020-eng-mar   <BASE>/pib_2017_2020_en_mr.zip  en-mr/*.en,en-mr/*.mr
Anuvaad-pibarchives_2015-2016-eng-mar   <BASE>/pibarchives_2015_2016_en_mr.zip  en-mr/*.en,en-mr/*.mr
Anuvaad-ie_sports-20210320-eng-mar      <BASE>/ie_sports_20210320_en_mr.zip     en-mr/*.en,en-mr/*.mr
Anuvaad-ie_news-20210320-eng-mar        <BASE>/ie_news_20210320_en_mr.zip       en-mr/*.en,en-mr/*.mr
Anuvaad-ie_lifestyle-20210320-eng-mar   <BASE>/ie_lifestyle_20210320_en_mr.zip  en-mr/*.en,en-mr/*.mr
Anuvaad-ie_entertainment-20210320-eng-mar       <BASE>/ie_entertainment_20210320_en_mr.zip      en-mr/*.en,en-mr/*.mr
Anuvaad-ie_business-20210320-eng-mar    <BASE>/ie_business_20210320_en_mr.zip   en-mr/*.en,en-mr/*.mr
Anuvaad-toi-20210320-eng-mar    <BASE>/toi_20210320_en_mr.zip   en-mr/*.en,en-mr/*.mr
Anuvaad-thewire-20210320-eng-mar        <BASE>/thewire_20210320_en_mr.zip       en-mr/*.en,en-mr/*.mr
Anuvaad-lokmat_entertainment-20210501-eng-mar   <BASE>/lokmat-entertainment_20210501_en_mr.zip  en-mr/*.en,en-mr/*.mr
Anuvaad-lokmat_news-20210501-eng-mar    <BASE>/lokmat-news_20210501_en_mr.zip   en-mr/*.en,en-mr/*.mr
Anuvaad-lokmat_lifestyle-20210501-eng-mar       <BASE>/lokmat-lifestyle_20210501_en_mr.zip      en-mr/*.en,en-mr/*.mr
Anuvaad-lokmat_sports-20210501-eng-mar  <BASE>/lokmat-sports_20210501_en_mr.zip en-mr/*.en,en-mr/*.mr
Anuvaad-lokmat_tech-20210501-eng-mar    <BASE>/lokmat-tech_20210501_en_mr.zip   en-mr/*.en,en-mr/*.mr
Anuvaad-lokmat_financial-20210501-eng-mar       <BASE>/lokmat-financial_20210501_en_mr.zip      en-mr/*.en,en-mr/*.mr
Anuvaad-lokmat_healthcare-20210501-eng-mar      <BASE>/lokmat-healthcare_20210501_en_mr.zip     en-mr/*.en,en-mr/*.mr
Anuvaad-zee-30042021-eng-mar    <BASE>/zee_30042021_en_mr.zip   en-mr/*.en,en-mr/*.mr
Anuvaad-ik_2021-v1-eng-pan      <BASE>/ik-2021-v1-en-pa.zip     en-pa/*.en,en-pa/*.pa
Anuvaad-internal_judicial_2021-v1-eng-pan       <BASE>/internal-judicial-2021-v1-en-pa.zip      en-pa/*.en,en-pa/*.pa
Anuvaad-legal_terms_2021-v1-eng-pan     <BASE>/legal-terms-2021-v1-en-pa.zip    en-pa/*.en,en-pa/*.pa
Anuvaad-nouns_dict_2021-v1-eng-pan      <BASE>/nouns-dict-2021-v1-en-pa.zip     en-pa/*.en,en-pa/*.pa
Anuvaad-pib_2017-2020-eng-pan   <BASE>/pib_2017_2020_en_pa.zip  en-pa/*.en,en-pa/*.pa
Anuvaad-pibarchives_2015-2016-eng-pan   <BASE>/pibarchives_2015_2016_en_pa.zip  en-pa/*.en,en-pa/*.pa
Anuvaad-tribune-20210320-eng-pan        <BASE>/tribune_20210320_en_pa.zip       en-pa/*.en,en-pa/*.pa
Anuvaad-pa_govt-20210320-eng-pan        <BASE>/pa_govt_20210320_en_pa.zip       en-pa/*.en,en-pa/*.pa
Anuvaad-jagran_news-20210320-eng-pan    <BASE>/jagran-news_20210320_en_pa.zip   en-pa/*.en,en-pa/*.pa
Anuvaad-jagran_tech-20210320-eng-pan    <BASE>/jagran-tech_20210320_en_pa.zip   en-pa/*.en,en-pa/*.pa
Anuvaad-jagran_education-20210320-eng-pan       <BASE>/jagran-education_20210320_en_pa.zip      en-pa/*.en,en-pa/*.pa
Anuvaad-jagran_entertainment-20210320-eng-pan   <BASE>/jagran-entertainment_20210320_en_pa.zip  en-pa/*.en,en-pa/*.pa
Anuvaad-jagran_business-20210320-eng-pan        <BASE>/jagran-business_20210320_en_pa.zip       en-pa/*.en,en-pa/*.pa
Anuvaad-jagran_sports-20210320-eng-pan  <BASE>/jagran-sports_20210320_en_pa.zip en-pa/*.en,en-pa/*.pa
Anuvaad-jagran_lifestyle-20210320-eng-pan       <BASE>/jagran-lifestyle_20210320_en_pa.zip      en-pa/*.en,en-pa/*.pa
Anuvaad-ik_2021-v1-eng-guj      <BASE>/ik-2021-v1-en-gu.zip     en-gu/*.en,en-gu/*.gu
Anuvaad-legal_terms_2021-v1-eng-guj     <BASE>/legal-terms-2021-v1-en-gu.zip    en-gu/*.en,en-gu/*.gu
Anuvaad-nouns_dict_2021-v1-eng-guj      <BASE>/nouns-dict-2021-v1-en-gu.zip     en-gu/*.en,en-gu/*.gu
Anuvaad-pib_2017-2020-eng-guj   <BASE>/pib_2017_2020_en_gu.zip  en-gu/*.en,en-gu/*.gu
Anuvaad-pibarchives_2015-2016-eng-guj   <BASE>/pibarchives_2015_2016_en_gu.zip  en-gu/*.en,en-gu/*.gu
Anuvaad-oneindia-20210320-eng-guj       <BASE>/oneindia_20210320_en_gu.zip      en-gu/*.en,en-gu/*.gu
Anuvaad-toi-20210320-eng-guj    <BASE>/toi_20210320_en_gu.zip   en-gu/*.en,en-gu/*.gu
Anuvaad-gu_govt-20210320-eng-guj        <BASE>/gu-govt_20210320_en_gu.zip       en-gu/*.en,en-gu/*.gu
Anuvaad-zee-30042021-eng-guj    <BASE>/zee_30042021_en_gu.zip   en-gu/*.en,en-gu/*.gu
Anuvaad-pib_2017-2020-eng-asm   <BASE>/pib_2017_2020_en_as.zip  en-as/*.en,en-as/*.as
Anuvaad-pibarchives_2015-2016-eng-asm   <BASE>/pibarchives_2015_2016_en_as.zip  en-as/*.en,en-as/*.as
Anuvaad-sentinel-20210320-eng-asm       <BASE>/sentinel_20210320_en_as.zip      en-as/*.en,en-as/*.as
Anuvaad-pib_2017-2020-eng-urd   <BASE>/pib_2017_2020_en_ur.zip  en-ur/*.en,en-ur/*.ur
Anuvaad-pibarchives_2015-2016-eng-urd   <BASE>/pibarchives_2015_2016_en_ur.zip  en-ur/*.en,en-ur/*.ur
Anuvaad-thewire-20210320-eng-urd        <BASE>/thewire_20210320_en_ur.zip       en-ur/*.en,en-ur/*.ur
Anuvaad-pib_2017-2020-eng-ori   <BASE>/pib_2017_2020_en_or.zip  en-or/*.en,en-or/*.or
Anuvaad-pibarchives_2015-2016-eng-ori   <BASE>/pibarchives_2015_2016_en_or.zip  en-or/*.en,en-or/*.or

P.S.
https://github.com/thammegowda/mtdata/blob/master/mtdata/index/anuvaad.py

@thammegowda thammegowda mentioned this issue Feb 28, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@thammegowda