Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate from old site #4

Merged
merged 4,064 commits into from Jul 10, 2020
Merged

Migrate from old site #4

merged 4,064 commits into from Jul 10, 2020

Conversation

garrison
Copy link
Contributor

@garrison garrison commented Jun 13, 2020

Choice and storybook lessons now work. This is pretty rapidly becoming something that could be merged to become a working static site. I am opening this pull request a little bit prematurely, though, just to see what one with 4000+ commits looks like.

I would like to do the following things before merging:

  • get audio/podcast lessons working
  • per-lesson contributor list
  • make a "license info" page (actually, this seems pretty essential given the attribution requirements in CC licenses). This could be rendered either by Hugo or by the browser (using JavaScript).
  • make sure the timestamps on git commits are appropriately timezone aware
  • make sure languageCode in each page's front matter is appropriately used in the new website somehow -- maybe as a lang attribute on content div
  • improve navigation
    • per-language _index.md pages that contain only a title (i.e., the name of the language as a section label)
    • understand page bundles in Hugo
  • improve wikitext scrape by disabling macros on the main site and then scraping the html it generates
  • move lesson media to ipfs using ipfs add -Qr directory a separate git repository. Part of this will involve understanding Ductus's mediacache and how best to scrape it completely
  • investigate storing the URNs in git, similar to how git-svn-id works (decided against this, but added the URN to the log message
  • get the new site working at future.wikiotics.org; move existing site to old.wikiotics.org with a temporary redirect for now (will require access to domain control panel)
  • get taxonomies (in particular, tags from wikiotics.org) working with Hugo
  • the lesson TOML is excessively wordy, so it would be nice to compact this into a form that is actually a joy to work with going forward. This is likely easier to do before merging, but it's not the highest priority.
    • identify a few representative lessons and work them into their ideal TOML structure, tweaking the javascript in the process
    • do a comparison against equivalent yaml before choosing what we like the best (TOML it is)
  • scrape podcasts and restore podcast links
  • map source-language: and target-language: tags to taxonomies

These optional things will likely wait until after merging:

  • breadcrumbs -- see here and here and here
  • it would be nice to get javascript minification working -- right now the site breaks with it enabled, for some reason
  • scrape remaining stubborn audio conversions

Any issues that deal with the quality of the automated scrape should be addressed before merging, as it is easy now to tweak one line of the scrape script and re-run it. Anything that requires manual attention (e.g., a long overdue general cleanup of site content) should wait until after this is merged.

@garrison garrison added the not yet ready to merge Please don't merge this label Jun 13, 2020
garrison added a commit to wikiotics/migration-from-ductus that referenced this pull request Jun 13, 2020
@garrison
Copy link
Contributor Author

@garrison garrison force-pushed the migration branch 6 times, most recently from 3b8f2f7 to f7873ed Compare June 19, 2020 11:18
@garrison
Copy link
Contributor Author

Here's an audio lesson (minus the podcast links, for now): https://wikiotics-pr-4.onrender.com/en/FrenchGreetings-WikiBabel

@garrison garrison force-pushed the migration branch 6 times, most recently from 8154ce7 to 3f90a6c Compare June 21, 2020 10:38
garrison added a commit to wikiotics/migration-from-ductus that referenced this pull request Jun 21, 2020
@garrison garrison force-pushed the migration branch 4 times, most recently from 1912fcd to f4361c2 Compare June 24, 2020 00:13
@garrison
Copy link
Contributor Author

@wikey and @laurentS: let me know if you notice any issues with the scraping at https://wikiotics.org. Here are the issues I am aware of:

  • all lesson media (pictures and audio) are currently served by the old server from https://old.wikiotics.org/mediacache/
  • audio lessons don't have the podcast files linked at the top
  • the license info page is missing

@wikiotics wikiotics deleted a comment from render bot Jun 24, 2020
@wikiotics wikiotics deleted a comment from render bot Jun 24, 2020
@wikiotics wikiotics deleted a comment from render bot Jun 24, 2020
@wikiotics wikiotics deleted a comment from render bot Jun 24, 2020
@wikiotics wikiotics deleted a comment from render bot Jun 24, 2020
@wikiotics wikiotics deleted a comment from render bot Jun 24, 2020
@wikiotics wikiotics deleted a comment from render bot Jun 24, 2020
sophia@wikiotics.org and others added 24 commits July 5, 2020 07:34
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:Jb64o8i1sWsEf2dsc8kWHN4sbuGcYRT_Sa5H7YcmCudeBf2fQ0KdRG7-9hRgGfAI

Parent: urn:sha384:jVZBFDSrGDbH2XUj4y_XQ5sOqXwdvu4bkcNfqzjWMznocTZzrv_XX6ziRliPhOEd
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:URp_DjDq95pfaByETrhocpPJnGb_WaTSAafjrku4sNL5iRrKBVZ9LnK13y68sS9g

Parent: urn:sha384:Jb64o8i1sWsEf2dsc8kWHN4sbuGcYRT_Sa5H7YcmCudeBf2fQ0KdRG7-9hRgGfAI
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:URp_DjDq95pfaByETrhocpPJnGb_WaTSAafjrku4sNL5iRrKBVZ9LnK13y68sS9g

Parent: urn:sha384:Jb64o8i1sWsEf2dsc8kWHN4sbuGcYRT_Sa5H7YcmCudeBf2fQ0KdRG7-9hRgGfAI
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:Q-0r_xaBLmtfQNR2c2t5C2noXMwQYRhE6-XoP4Yj8nnuodoVljAjFw0FnlJ-mrbN
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:PY7mHves_TIfl-1QQGKfe8BQUY9GvK5js1uZ04mhijpTC-iH61vlx5YX9ewOjAD3

Parent: urn:sha384:Q-0r_xaBLmtfQNR2c2t5C2noXMwQYRhE6-XoP4Yj8nnuodoVljAjFw0FnlJ-mrbN
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:TyET45V-fgcm1LsvlJurpGmdBRRENj_kYx3mgKVcwGPtZtMyceYAj6qkWuVvGwh1

Parent: urn:sha384:PY7mHves_TIfl-1QQGKfe8BQUY9GvK5js1uZ04mhijpTC-iH61vlx5YX9ewOjAD3
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:kGQE1TVY61m9XbjTZQ4TL6_VHqWLgvltNuQxzJE2GSBJkt8xkjYqvSPB0f9jd842

Parent: urn:sha384:TyET45V-fgcm1LsvlJurpGmdBRRENj_kYx3mgKVcwGPtZtMyceYAj6qkWuVvGwh1
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:jNFEi4H6rzwg9_zglM6c4U0qEfjxi_acofOZwVX6ZRBehM7TRVC14SDg-QUig4b9
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:_82WcPOiD2jYudr-a_kx5zhp0eQIDGp26EXb1CBKyDX55BY7mQjhvInngNs9rovS

Parent: urn:sha384:jNFEi4H6rzwg9_zglM6c4U0qEfjxi_acofOZwVX6ZRBehM7TRVC14SDg-QUig4b9
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:LDaP5zzJLY5s1QqRZ6BOPLmsCv5FEKlj1_VTkrspTJg-dHgeOOsDrnc2ALoce8Ot
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:mPXdBEKAMbJVEtcONYdNiNhlHXsUR65iiHI39GNFGbTLyYGta5DXW4EENP0qqTou
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:zkwyifaKxGPYn2f59VooDOs7hUaF6kvfeXiyFHjXzUwwIFyQ2DVkvSvbB1SuWT14

Parent: urn:sha384:mPXdBEKAMbJVEtcONYdNiNhlHXsUR65iiHI39GNFGbTLyYGta5DXW4EENP0qqTou
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:0lsNAvRIv67mDZvucs11Q0vhkfJSAfyZ9c_NZpXstxfU7LTn1tdrMk3vn1K3WHoC

Parent: urn:sha384:zkwyifaKxGPYn2f59VooDOs7hUaF6kvfeXiyFHjXzUwwIFyQ2DVkvSvbB1SuWT14
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:IvdHVqrA7adNzVilzq3-o4jqrdQLE_pDLAKapN4wVpQF2Tsexih8H2JG4xFGNI7L

Parent: urn:sha384:0lsNAvRIv67mDZvucs11Q0vhkfJSAfyZ9c_NZpXstxfU7LTn1tdrMk3vn1K3WHoC
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:0OpzcfZb2Z0NBGL2X5BGOjxKoZsNY-0HI1dP5XGEHB0H7PeQhWEXV6rZYZaMqQCT

Parent: urn:sha384:IvdHVqrA7adNzVilzq3-o4jqrdQLE_pDLAKapN4wVpQF2Tsexih8H2JG4xFGNI7L
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:44x8zHF4dc2TO8jNbDwDD-EaphH6X-eM_tF-tAPhvhiEuhiD5f-mZzMpwKpnaTEa

Parent: urn:sha384:0OpzcfZb2Z0NBGL2X5BGOjxKoZsNY-0HI1dP5XGEHB0H7PeQhWEXV6rZYZaMqQCT
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:0MFIst3sHsVm0Hw7M_4wnDtnQCxidICK7YDwIncbmuSSg1pO3NZuQ1UI14sUlPea

Parent: urn:sha384:44x8zHF4dc2TO8jNbDwDD-EaphH6X-eM_tF-tAPhvhiEuhiD5f-mZzMpwKpnaTEa
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:I6RCsMtdDvpODUVf6ZOKeooZY1aRMjOG9sfPWBPtcDCQa5sZb77bI-dz2qf42j8-

Parent: urn:sha384:0MFIst3sHsVm0Hw7M_4wnDtnQCxidICK7YDwIncbmuSSg1pO3NZuQ1UI14sUlPea
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:2pdGA4cl7xpjdkyEp9CRlNXh98Z5YxINKWX6vKC-e_yUZLKU6mKxrrLtAXjUwrIs

Parent: urn:sha384:I6RCsMtdDvpODUVf6ZOKeooZY1aRMjOG9sfPWBPtcDCQa5sZb77bI-dz2qf42j8-
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:hNPgzQ9Z0NHgvmw3OsuWTx6yFao3MZgQFEfvYcpMU-D7IOHCLXDmJ6Qo7CeFPTzM

Parent: urn:sha384:I6RCsMtdDvpODUVf6ZOKeooZY1aRMjOG9sfPWBPtcDCQa5sZb77bI-dz2qf42j8-
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:TMJ-yS7xlv73wp-MVOTodyPkdaVix5tU3zrrLjuF-kF17o9-gdBYxL316YgLs0zl

Parent: urn:sha384:xxRPNLOWCek44bOqgN_8PIivLkycb8AKbkAlzWoRcDiol1C-1o2bQRHDnJ5KR2N3
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:GGwKlFbM2n9z_jr5KbggYuQJ-u4rOg0yWR1bbraREl8U6980DLKsEaa3u4He4ZH7
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:c5KpDgt06RopJ7u9u5Ib2fLWk2oPexAIFcKz3x8bsk4g8uBytPu0tQ1DlQpdeVvO
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:M4gaZsuJkBk2zyyIbDhkJ6c0bEAj0PYqVDeRVr-qAsFMt2nYvZs-m7abjhn5Jn0v

Parent: urn:sha384:92tWw-W2TPl-QvSUoxg5PoaLx9TV_bOA7XHjxUvar-t8TgaPubG3VaoVoeQBKWWZ
Copy link

@wikey wikey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks lovely

@garrison garrison merged commit 6570dd1 into master Jul 10, 2020
@garrison garrison deleted the migration branch July 10, 2020 21:25
garrison pushed a commit that referenced this pull request Jul 10, 2020
Automated import from Ductus
https://github.com/wikiotics/migration-from-ductus

urn:sha384:3oGNEeo1saaX5gsnrSZ58ZKsNv1tqTor3UVDBMZpA33QF-nCtwyRyqU8UHj1-ojY

Parent: urn:sha384:pvlp7APoekGpXWls-r7yiJRXUQZw4dfkdPVZ4t4aqTvW1nC2-4ZpA2DUbaNvyA_W
garrison added a commit that referenced this pull request Jul 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants