Bug: TEMPLATES_DIR cannot be changed

[CatalinFrancu]

Describe the bug

According to the Configuration wiki page, I should be able to change TEMPLATES_DIR to customize the layout. However, I cannot seem to change it.

Editing ArchiveBox.conf manually does nothing: archivebox config --get TEMPLATES_DIR and archivebox version still report the default path, which in my case (Arch Linux) is /usr/lib/python3.9/site-packages/archivebox/templates.

Trying to set it via archivebox config --set TEMPLATES_DIR=test produces the error:

[i] [2021-03-05 12:34:33] ArchiveBox v0.5.4: archivebox config --set TEMPLATES_DIR=test
    > /srv/http/archivebox-data

Traceback (most recent call last):
  File "/usr/bin/archivebox", line 33, in <module>
    sys.exit(load_entry_point('archivebox==0.5.4', 'console_scripts', 'archivebox')())
  File "/usr/lib/python3.9/site-packages/archivebox/cli/__init__.py", line 129, in main
    run_subcommand(
  File "/usr/lib/python3.9/site-packages/archivebox/cli/__init__.py", line 69, in run_subcommand
    module.main(args=subcommand_args, stdin=stdin, pwd=pwd)    # type: ignore
  File "/usr/lib/python3.9/site-packages/archivebox/cli/archivebox_config.py", line 50, in main
    config(
  File "/usr/lib/python3.9/site-packages/archivebox/util.py", line 112, in typechecked_function
    return func(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/archivebox/main.py", line 910, in config
    matching_config = write_config_file(new_config, out_dir=OUTPUT_DIR)
  File "/usr/lib/python3.9/site-packages/archivebox/config.py", line 469, in write_config_file
    section = find_section(key)
  File "/usr/lib/python3.9/site-packages/archivebox/config.py", line 465, in <lambda>
    find_section = lambda key: [name for name, opts in CONFIG_SCHEMA.items() if key in opts][0]
IndexError: list index out of range

This could happen because TEMPLATES_DIR is not defined in CONFIG_SCHEMA (but don't take my word for it).

Thanks for your time and thanks for ArchiveBox!

ArchiveBox version

ArchiveBox v0.5.4
Cpython Linux Linux-5.11.1-arch1-1-x86_64-with-glibc2.33 x86_64 (not in Docker)

[i] Dependency versions:
 √  ARCHIVEBOX_BINARY     v0.5.4          valid     /usr/bin/archivebox                                                         
 √  PYTHON_BINARY         v3.9.2          valid     /usr/bin/python3.9                                                          
 √  DJANGO_BINARY         v3.1.7          valid     /usr/lib/python3.9/site-packages/django/bin/django-admin.py                 
 √  CURL_BINARY           v7.75.0         valid     /usr/bin/curl                                                               
 √  WGET_BINARY           v1.21.1         valid     /usr/bin/wget                                                               
 √  NODE_BINARY           v12.20.1        valid     /usr/bin/node                                                               
 √  SINGLEFILE_BINARY     v0.3.12         valid     /usr/bin/single-file                                                        
 -  READABILITY_BINARY    -               disabled  /usr/bin/readability-extractor                                              
 -  MERCURY_BINARY        -               disabled  /usr/bin/mercury-parser                                                     
 -  GIT_BINARY            -               disabled  /usr/bin/git                                                                
 √  YOUTUBEDL_BINARY      v2021.02.22     valid     /usr/bin/youtube-dl                                                         
 √  CHROME_BINARY         v89.0.4389.72   valid     /usr/bin/chromium                                                           
 √  RIPGREP_BINARY        v12.1.1         valid     /usr/bin/rg                                                                 

[i] Source-code locations:
 √  PACKAGE_DIR           23 files        valid     /usr/lib/python3.9/site-packages/archivebox                                 
 √  TEMPLATES_DIR         3 files         valid     /usr/lib/python3.9/site-packages/archivebox/templates                       

[i] Secrets locations:
 -  CHROME_USER_DATA_DIR  -               disabled                                                                              
 -  COOKIES_FILE          -               disabled                                                                              

[i] Data locations:
 √  OUTPUT_DIR            9 files         valid     /srv/http/archivebox-data                                                   
 √  SOURCES_DIR           8 files         valid     ./sources                                                                   
 √  LOGS_DIR              0 files         valid     ./logs                                                                      
 √  ARCHIVE_DIR           5 files         valid     ./archive                                                                   
 √  CONFIG_FILE           303.0 Bytes     valid     ./ArchiveBox.conf                                                           
 √  SQL_INDEX             212.0 KB        valid     ./index.sqlite3                                                             

[pirate]

Changing that dir used to be supported but is no longer officially supported since we moved to django.

Instead for now I recommend applying a patch to the files inside /usr/lib/python3.9/site-packages/archivebox/templates manually (save your changes somewhere else so that you dont lose them during updates). Future versions may bring back the override ability, but it's not high on our priority list at the moment. PRs welcome.

[CatalinFrancu]

Thank you for the explanation! It might be worth updating the wiki page to reflect this. We will probably use the frame contents only and provide our own header to integrate with the rest of the website. Sadly, I am not proficient enough in Django to be of any help. :-/

[pirate]

Reopening this because we do want to fix this at some point. Here are the steps to fix it for anyone interested in working on this:

1. add the config value back in archivebox/config.py
2. import the value in archivebox/core/settings.py and add it to the list of TEMPLATE_DIRS
3. test to confirm it plays nicely with existing templates (e.g. you should be able to override a single template without needing to copy the whole dir of templates into your custom dir)

[pirate]

Ok I've added this option back in v0.6 e4823cc. It's now called CUSTOM_TEMPLATES_DIR. Here's an example of how to use it:

git clone https://github.com/ArchiveBox/ArchiveBox /tmp/ArchiveBox
cp -r /tmp/ArchiveBox/archivebox/templates ~/archivebox/custom_templates

# make whatever changes you want to the default templates
nano custom_templates/static/admin.css
nano custom_templates/admin/base.html
... etc.

cd ~/archivebox/
archivebox config --set CUSTOM_TEMPLATES_DIR=custom_templates   # location is relative to data dir root by default, you can also pass an absolute path
archivebox server

# served pages should now contain your custom overrides

You can use v0.6 early by installing from the dev branch:

pip install "git+https://github.com/ArchiveBox/ArchiveBox.git@dev"

Or you can wait till it's officially released soon. Feel free to comment back here if you have any trouble and I'll reopen the issue.